LDAP

Important Capabilities

Capability	Status	Notes
Detect Deleted Entities	✅	Optionally enabled via stateful_ingestion.remove_stale_metadata

This plugin extracts the following:

• People
• Names, emails, titles, and manager information for each person
• List of groups

CLI based Ingestion

Install the Plugin

pip install 'acryl-datahub[ldap]'

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: "ldap"
  config:
    # Coordinates
    ldap_server: ldap://localhost

    # Credentials
    ldap_user: "cn=admin,dc=example,dc=org"
    ldap_password: "admin"

    # Options
    base_dn: "dc=example,dc=org"

sink:
  # sink configs

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

Field	Description
base_dn ✅ string	LDAP DN.
ldap_password ✅ string	LDAP password.
ldap_server ✅ string	LDAP server URL.
ldap_user ✅ string	LDAP user.
drop_missing_first_last_name boolean	If set to true, any users without first and last names will be dropped. Default: True
filter string	LDAP extractor filter. Default: (objectClass=*)
group_attrs_map object	Default: {}
manager_filter_enabled boolean	Use LDAP extractor filter to search managers. Default: True
manager_pagination_enabled boolean	[deprecated] Use pagination_enabled Default: True
page_size integer	Size of each page to fetch when extracting metadata. Default: 20
pagination_enabled boolean	Use pagination while do search query (enabled by default). Default: True
platform_instance string	The instance of the platform that all assets produced by this recipe belong to
use_email_as_username boolean	Use email for users' usernames instead of username (disabled by default). If enabled, the user and group urn would be having email as the id part of the urn. Default: False
user_attrs_map object	Default: {}
env string	The environment that all assets produced by this connector belong to Default: PROD
attrs_list array	Retrieved attributes list
attrs_list.string string
custom_props_list array	A list of custom attributes to extract from the LDAP provider.
custom_props_list.string string
stateful_ingestion StatefulStaleMetadataRemovalConfig	Base specialized config for Stateful Ingestion with stale metadata removal capability.
stateful_ingestion.enabled boolean	The type of the ingestion state provider registered with datahub. Default: False
stateful_ingestion.remove_stale_metadata boolean	Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled. Default: True

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "LDAPSourceConfig",
  "description": "Config used by the LDAP Source.",
  "type": "object",
  "properties": {
    "env": {
      "title": "Env",
      "description": "The environment that all assets produced by this connector belong to",
      "default": "PROD",
      "type": "string"
    },
    "platform_instance": {
      "title": "Platform Instance",
      "description": "The instance of the platform that all assets produced by this recipe belong to",
      "type": "string"
    },
    "stateful_ingestion": {
      "$ref": "#/definitions/StatefulStaleMetadataRemovalConfig"
    },
    "ldap_server": {
      "title": "Ldap Server",
      "description": "LDAP server URL.",
      "type": "string"
    },
    "ldap_user": {
      "title": "Ldap User",
      "description": "LDAP user.",
      "type": "string"
    },
    "ldap_password": {
      "title": "Ldap Password",
      "description": "LDAP password.",
      "type": "string"
    },
    "base_dn": {
      "title": "Base Dn",
      "description": "LDAP DN.",
      "type": "string"
    },
    "filter": {
      "title": "Filter",
      "description": "LDAP extractor filter.",
      "default": "(objectClass=*)",
      "type": "string"
    },
    "attrs_list": {
      "title": "Attrs List",
      "description": "Retrieved attributes list",
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "custom_props_list": {
      "title": "Custom Props List",
      "description": "A list of custom attributes to extract from the LDAP provider.",
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "drop_missing_first_last_name": {
      "title": "Drop Missing First Last Name",
      "description": "If set to true, any users without first and last names will be dropped.",
      "default": true,
      "type": "boolean"
    },
    "page_size": {
      "title": "Page Size",
      "description": "Size of each page to fetch when extracting metadata.",
      "default": 20,
      "type": "integer"
    },
    "manager_filter_enabled": {
      "title": "Manager Filter Enabled",
      "description": "Use LDAP extractor filter to search managers.",
      "default": true,
      "type": "boolean"
    },
    "manager_pagination_enabled": {
      "title": "Manager Pagination Enabled",
      "description": "[deprecated] Use pagination_enabled ",
      "default": true,
      "type": "boolean"
    },
    "pagination_enabled": {
      "title": "Pagination Enabled",
      "description": "Use pagination while do search query (enabled by default).",
      "default": true,
      "type": "boolean"
    },
    "use_email_as_username": {
      "title": "Use Email As Username",
      "description": "Use email for users' usernames instead of username (disabled by default).             If enabled, the user and group urn would be having email as the id part of the urn.",
      "default": false,
      "type": "boolean"
    },
    "user_attrs_map": {
      "title": "User Attrs Map",
      "default": {},
      "type": "object"
    },
    "group_attrs_map": {
      "title": "Group Attrs Map",
      "default": {},
      "type": "object"
    }
  },
  "required": [
    "ldap_server",
    "ldap_user",
    "ldap_password",
    "base_dn"
  ],
  "additionalProperties": false,
  "definitions": {
    "DynamicTypedStateProviderConfig": {
      "title": "DynamicTypedStateProviderConfig",
      "type": "object",
      "properties": {
        "type": {
          "title": "Type",
          "description": "The type of the state provider to use. For DataHub use `datahub`",
          "type": "string"
        },
        "config": {
          "title": "Config",
          "description": "The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19).",
          "default": {},
          "type": "object"
        }
      },
      "required": [
        "type"
      ],
      "additionalProperties": false
    },
    "StatefulStaleMetadataRemovalConfig": {
      "title": "StatefulStaleMetadataRemovalConfig",
      "description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.",
      "type": "object",
      "properties": {
        "enabled": {
          "title": "Enabled",
          "description": "The type of the ingestion state provider registered with datahub.",
          "default": false,
          "type": "boolean"
        },
        "remove_stale_metadata": {
          "title": "Remove Stale Metadata",
          "description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    }
  }
}

Code Coordinates

• Class Name: datahub.ingestion.source.ldap.LDAPSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for LDAP, feel free to ping us on our Slack.