Slack

Important Capabilities

Capability	Status	Notes
Detect Deleted Entities	✅	Optionally enabled via stateful_ingestion.remove_stale_metadata

CLI based Ingestion

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

Field	Description
bot_token ✅ string(password)	Bot token for the Slack workspace. Needs users:read, users:read.email, users.profile:read, and team:read scopes.
api_requests_per_min integer	Number of API requests per minute. Low-level config. Do not tweak unless you are facing any issues. Default: 10
channel_min_members integer	Ingest channels with at least this many members. Default: 2
channels_iteration_limit integer	Limit the number of channels to be ingested in a iteration. Low-level config. Do not tweak unless you are facing any issues. Default: 200
enrich_user_metadata boolean	When enabled, will enrich provisioned DataHub users' metadata with information from Slack. Default: True
ingest_public_channels boolean	Whether to ingest public channels. If set to true needs channels:read scope. Default: False
ingest_users boolean	Whether to ingest users. When set to true, will ingest all users in the Slack workspace (as platform resources) to simplify user enrichment after they are provisioned on DataHub. Default: True
should_ingest_archived_channels boolean	Whether to ingest archived channels. Default: False
stateful_ingestion StatefulIngestionConfig	Stateful Ingestion Config
stateful_ingestion.enabled boolean	Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False Default: False

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "SlackSourceConfig",
  "description": "Base configuration class for stateful ingestion for source configs to inherit from.",
  "type": "object",
  "properties": {
    "stateful_ingestion": {
      "title": "Stateful Ingestion",
      "description": "Stateful Ingestion Config",
      "allOf": [
        {
          "$ref": "#/definitions/StatefulIngestionConfig"
        }
      ]
    },
    "bot_token": {
      "title": "Bot Token",
      "description": "Bot token for the Slack workspace. Needs `users:read`, `users:read.email`, `users.profile:read`, and `team:read` scopes.",
      "type": "string",
      "writeOnly": true,
      "format": "password"
    },
    "enrich_user_metadata": {
      "title": "Enrich User Metadata",
      "description": "When enabled, will enrich provisioned DataHub users' metadata with information from Slack.",
      "default": true,
      "type": "boolean"
    },
    "ingest_users": {
      "title": "Ingest Users",
      "description": "Whether to ingest users. When set to true, will ingest all users in the Slack workspace (as platform resources) to simplify user enrichment after they are provisioned on DataHub.",
      "default": true,
      "type": "boolean"
    },
    "api_requests_per_min": {
      "title": "Api Requests Per Min",
      "description": "Number of API requests per minute. Low-level config. Do not tweak unless you are facing any issues.",
      "default": 10,
      "type": "integer"
    },
    "ingest_public_channels": {
      "title": "Ingest Public Channels",
      "description": "Whether to ingest public channels. If set to true needs `channels:read` scope.",
      "default": false,
      "type": "boolean"
    },
    "channels_iteration_limit": {
      "title": "Channels Iteration Limit",
      "description": "Limit the number of channels to be ingested in a iteration. Low-level config. Do not tweak unless you are facing any issues.",
      "default": 200,
      "type": "integer"
    },
    "channel_min_members": {
      "title": "Channel Min Members",
      "description": "Ingest channels with at least this many members.",
      "default": 2,
      "type": "integer"
    },
    "should_ingest_archived_channels": {
      "title": "Should Ingest Archived Channels",
      "description": "Whether to ingest archived channels.",
      "default": false,
      "type": "boolean"
    }
  },
  "required": [
    "bot_token"
  ],
  "definitions": {
    "DynamicTypedStateProviderConfig": {
      "title": "DynamicTypedStateProviderConfig",
      "type": "object",
      "properties": {
        "type": {
          "title": "Type",
          "description": "The type of the state provider to use. For DataHub use `datahub`",
          "type": "string"
        },
        "config": {
          "title": "Config",
          "description": "The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19).",
          "default": {},
          "type": "object"
        }
      },
      "required": [
        "type"
      ],
      "additionalProperties": false
    },
    "StatefulIngestionConfig": {
      "title": "StatefulIngestionConfig",
      "description": "Basic Stateful Ingestion Specific Configuration for any source.",
      "type": "object",
      "properties": {
        "enabled": {
          "title": "Enabled",
          "description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
          "default": false,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    }
  }
}

Code Coordinates

• Class Name: datahub.ingestion.source.slack.slack.SlackSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Slack, feel free to ping us on our Slack.