Redash

Important Capabilities

Capability	Status	Notes
Table-Level Lineage	✅	Enabled by default

This plugin extracts the following:

• Redash dashboards and queries/visualization
• Redash chart table lineages (disabled by default)

CLI based Ingestion

Install the Plugin

The redash source works out of the box with acryl-datahub.

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: "redash"
  config:
    connect_uri: http://localhost:5000/
    api_key: REDASH_API_KEY

    # Optionals
    # api_page_limit: 1 #default: None, no limit on ingested dashboards and charts API pagination
    # skip_draft: true  #default: true, only ingest published dashboards and charts
    # dashboard_patterns:
    #   deny:
    #     - ^denied dashboard.*
    #   allow:
    #     - .*allowed dashboard.*
    # chart_patterns:
    #   deny:
    #     - ^denied chart.*
    #   allow:
    #     - .*allowed chart.*
    # parse_table_names_from_sql: false

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

Field	Description
api_key string	Redash user API key. Default: REDASH_API_KEY
api_page_limit integer	Limit on number of pages queried for ingesting dashboards and charts API during pagination. Default: 9223372036854775807
connect_uri string	Redash base URL. Default: http://localhost:5000
page_size integer	Limit on number of items to be queried at once. Default: 25
parallelism integer	Parallelism to use while processing. Default: 1
parse_table_names_from_sql boolean	See note below. Default: False
skip_draft boolean	Only ingest published dashboards and charts. Default: True
env string	Environment to use in namespace when constructing URNs. Default: PROD
chart_patterns AllowDenyPattern	regex patterns for charts to filter for ingestion. Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
chart_patterns.ignoreCase boolean	Whether to ignore case sensitivity during pattern matching. Default: True
chart_patterns.allow array	List of regex patterns to include in ingestion Default: ['.*']
chart_patterns.allow.string string
chart_patterns.deny array	List of regex patterns to exclude from ingestion. Default: []
chart_patterns.deny.string string
dashboard_patterns AllowDenyPattern	regex patterns for dashboards to filter for ingestion. Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True}
dashboard_patterns.ignoreCase boolean	Whether to ignore case sensitivity during pattern matching. Default: True
dashboard_patterns.allow array	List of regex patterns to include in ingestion Default: ['.*']
dashboard_patterns.allow.string string
dashboard_patterns.deny array	List of regex patterns to exclude from ingestion. Default: []
dashboard_patterns.deny.string string

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "RedashConfig",
  "type": "object",
  "properties": {
    "connect_uri": {
      "title": "Connect Uri",
      "description": "Redash base URL.",
      "default": "http://localhost:5000",
      "type": "string"
    },
    "api_key": {
      "title": "Api Key",
      "description": "Redash user API key.",
      "default": "REDASH_API_KEY",
      "type": "string"
    },
    "dashboard_patterns": {
      "title": "Dashboard Patterns",
      "description": "regex patterns for dashboards to filter for ingestion.",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    },
    "chart_patterns": {
      "title": "Chart Patterns",
      "description": "regex patterns for charts to filter for ingestion.",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    },
    "skip_draft": {
      "title": "Skip Draft",
      "description": "Only ingest published dashboards and charts.",
      "default": true,
      "type": "boolean"
    },
    "page_size": {
      "title": "Page Size",
      "description": "Limit on number of items to be queried at once.",
      "default": 25,
      "type": "integer"
    },
    "api_page_limit": {
      "title": "Api Page Limit",
      "description": "Limit on number of pages queried for ingesting dashboards and charts API during pagination.",
      "default": 9223372036854775807,
      "type": "integer"
    },
    "parallelism": {
      "title": "Parallelism",
      "description": "Parallelism to use while processing.",
      "default": 1,
      "type": "integer"
    },
    "parse_table_names_from_sql": {
      "title": "Parse Table Names From Sql",
      "description": "See note below.",
      "default": false,
      "type": "boolean"
    },
    "env": {
      "title": "Env",
      "description": "Environment to use in namespace when constructing URNs.",
      "default": "PROD",
      "type": "string"
    }
  },
  "additionalProperties": false,
  "definitions": {
    "AllowDenyPattern": {
      "title": "AllowDenyPattern",
      "description": "A class to store allow deny regexes",
      "type": "object",
      "properties": {
        "allow": {
          "title": "Allow",
          "description": "List of regex patterns to include in ingestion",
          "default": [
            ".*"
          ],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "deny": {
          "title": "Deny",
          "description": "List of regex patterns to exclude from ingestion.",
          "default": [],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "ignoreCase": {
          "title": "Ignorecase",
          "description": "Whether to ignore case sensitivity during pattern matching.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    }
  }
}

Note! The integration can use an SQL parser to try to parse the tables the chart depends on. This parsing is disabled by default, but can be enabled by setting parse_table_names_from_sql: true. The parser is based on the sqlglot package.

Code Coordinates

• Class Name: datahub.ingestion.source.redash.RedashSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Redash, feel free to ping us on our Slack.