Creating mappings for OpenLineage events

Create a mapping to display lineage for a custom or unsupported technology.

Overview

To display lineage for custom or unsupported technologies, complete these tasks:

  1. Create data source definitions.
  2. Create custom technology.
  3. Create a mapping rule.
  4. Run a metadata import or ingest the events by using APIs.

Prerequisites

Permissions

The following permissions and roles are required.

Task Permission or role
Create a data source definition • Platform asset catalog role: Admin or Editor
• IBM Cloud IAM role: CloudPak Data Source Creator or CloudPak Data Source Administrator
Create a connection to a data source • Access to connections to data sources
• Project role: Admin or Editor
Create a custom technology • Manage data lineage permission
Create an OpenLineage mapping • Manage data lineage permission
Import lineage metadata • Manage data lineage permission
• Project role: Admin or Editor
View lineage • Manage data lineage or Access data lineage permission

Data source definitions

Create a data source definition for each endpoint that is referenced in the OpenLineage event.

For more information, see Creating a data source definition.

Create a custom technology

If the supported technologies do not match the structure of your technology, create a custom technology. Complete these steps:

  1. Go to Data > Data lineage > Map lineage > Map OpenLineage > Custom technologies, and click Create technology.
  2. Enter a meaningful name that easily identifies the technology.
  3. Enter a unique branch name.
  4. Select the technology type that is most appropriate for your technology.
  5. Define the number of assets in the data structure hierarchy, and edit asset type names if needed.
  6. Optionally, define which asset type occurs recursively in the hierarchy.

For more information, see Technology type.

Create a mapping

To create a mapping for an OpenLineage event, complete these steps:

  1. Go to Data > Data lineage > Map lineage > Map OpenLineage > Active mappings, and click Create mapping.
  2. Select the type of namespace, Dataset namespace or Job namespace. See Rule scope.
  3. Select the namespace matching method, Namespace prefix or Namespace exact value. See Namespace matching method and namespace value.
  4. Provide the prefix or exact value. If you provide prefix, include the delimiter :// or . in the value. Click Next.
  5. Select the technology type, a predefined or a custom one. If needed, create a new custom technology.
  6. Specify the data source definition assignment method. Select Assign automatically and proceed, or Select manually and select a data source definition from the list. See Data source definitions.
  7. Review the summary of your mapping and click Create.

Create a metadata import

When your mapping is ready, it is processed when an OpenLineage event is imported. The event must be in the .json format, and the .json file must be compressed in a .zip file.

To import the OpenLineage event, complete these steps:

  1. In the project, go to the Assets tab, and click New asset > Import metadata for data assets.
  2. Specify a name and select the Import lineage metadata goal.
  3. Select the data source definition that matches your event data source.
  4. In the Add inputs from file section, click Add, and select the .zip file with the .json file inside.
  5. Optionally, edit the remaining metadata import options and save and run your job.

When the metadata import job is completed, see the lineage in Data > Data lineage.

For more information, see Creating metadata imports.

Ingest events by using APIs

Alternatively to importing metadata, you can ingest events by using APIs. See Ingesting OpenLineage events.

Learn more