Microsoft Azure Databricks lineage configuration

To import lineage metadata from Microsoft Azure Databricks, create a connection, data source definition and metadata import job.

Microsoft Azure Databricks is a big data analytics tool that is based on Apache Spark.

Processed metadata

The following Microsoft Azure Databricks metadata is processed and displayed on lineage:

  • Catalogs
  • Schemas
  • Tables and views
  • Functions
  • Notebooks

Limitations

The following limitations apply:

  • Notebook Scala commands are not processed. Depending on the use case, you can use the OpenLineage format as an alternative.
  • Notebook R commands are not processed.
  • Custom Libraries are not processed.
  • Definitions of functions and views from Hive Metastore are not processed.

Prerequisite configuration

Before you import lineage metadata, ensure that the following prerequisites are met:

  • You have network access from the IBM Automatic Data Lineage instance to the Databricks API hosted by Databricks.
  • You provided a Databricks personal access token (PAT) to authenticate the Manta Databricks scanner. You can obtain the personal access token through the Databricks UI. The token is used to authenticate Automatic Data Lineage to the scanned Databricks instance.
  • You have a personal access token associated with a metastore admin account so that you can extract all assets without restrictions.
  • You have a metastore admin role or you have the following privileges to extract individual asset types:
    • Catalogs
      • You have ownership of the catalog or have the USE_CATALOG privilege.
    • Schemas
      • You have ownership of the schema or have the USE_SCHEMA privilege.
      • You have ownership of the parent catalog or have the USE_CATALOG privilege on that catalog.
    • Tables and views
      • You have ownership of the table or view or have the SELECT privilege.
      • You have ownership of the parent catalog or have the USE_CATALOG privilege.
      • You have ownership of the parent schema or have the USE_SCHEMA privilege.
    • Functions
      • You have ownership of the function or have the EXECUTE privilege.
      • You have ownership of the parent catalog or have the USE_CATALOG privilege.
      • You have ownership of the parent schema or have the USE_SCHEMA privilege.
    • Notebooks
      • You have ownership of the notebook or have at least READ access.
  • You have the SELECT privilege on all Hive metastore assets. For example, schemas and tables to ensure full extraction. Hive metastore assets are also accessed using the personal access token.

Creating a metadata import asset

Data source connection

To connect to the data source from which you want to import lineage metadata, you need to select a data source definition and a connection. You can create them before you start creating the metadata import, or you can create them when you create and configure the metadata import asset.

Data source definition

Select Microsoft Azure Databricks as the data source type.

Connection

For connection details, see Microsoft Azure Databricks connection.

Connection mode

You can connect to Microsoft Azure Databricks by using one of the following connection modes:

Include and exclude lists

You can include or exclude assets up to the schema level. Provide catalogs and schemas in the format catalog/schema. Each part is evaluated as a regular expression. Assets which are added later in the data source will also be included or excluded if they match the conditions specified in the lists. Example values:

  • myCatalog/: all schemas in myCatalog,
  • myCatalog/.*: all schemas in myCatalog,
  • myCatalog3/mySchema1: mySchema1 from myCatalog3,
  • myCatalog4/mySchema[1-5]: any schema in my myCatalog4 with a name that starts with mySchema and ends with a digit between 1 and 5

External inputs

If you use external Microsoft Azure Databricks dll archives, you can add them in a .zip file as an external input. You can organize the structure of the .zip file as the dll folder with subfolders or archives that represent the workspace structure. The .zip file can have the following structure:

<dll>
    <catalog_name_folder>
      <schema_name_folder>
        <tables>
          <table_name.sql>
        <views>
          <view_name.sql>

Advanced import options

Display table lineage
Generate edges between tables for which the column-level lineage information was not found.
Use lineage tracking API
Determine how table and column lineage data is retrieved from Databricks. When set to true, the Lineage Tracking API is used, which can impact performance when scanning large volumes of data. When set to false, the SQL Statements API is used instead (for example, system.access.table_lineage and related tables).

Learn more