OpenLineage integration

Integrate with OpenLineage to capture, ingest, and process lineage that is produced by systems that automatically emit OpenLineage events, and custom lineage that is documented by using OpenLineage payloads.

OpenLineage across IBM watsonx platform

IBM uses OpenLineage as the open standard for exchanging data lineage information across watsonx platform. OpenLineage is an open, community‑driven specification. It replaces proprietary lineage formats with a common model. This approach allows lineage metadata to be exchanged consistently across products and with external systems that emit OpenLineage events. It provides a unified view of lineage for both structured and unstructured data.

Review how different watsonx products work together with OpenLineage:

watsonx.data: Acts as an OpenLineage producer. These events capture runtime lineage for structured analytics workloads. They describe source datasets, executed jobs, and generated outputs. See Integrating with IBM Manta Data Lineage.
watsonx.data integration: Acts as an OpenLineage producer for ingestion and transformation workflows. This includes pipelines that process unstructured data. Emitted events describe data movement and transformation at execution time. See Lineage for unstructured data.
watsonx.data intelligence: Acts as the OpenLineage consumer. It imports and processes OpenLineage events generated by IBM products, third‑party tools, and custom lineage mappings that include custom lineage for proprietary and legacy systems. It correlates these events with lineage collected by automated scanners. The result is an end‑to‑end lineage view.

This producer–consumer model represents lineage across the data lifecycle. It covers ingestion, transformation, analytics, and downstream consumption. The model is extensible to external tools and custom or legacy systems. Lineage metadata remains portable and interoperable across platforms.

Overview of OpenLineage integration in watsonx.data intelligence

By integrating with OpenLineage, you can ingest data from systems and custom technologies that are not supported, but are used in your data flow. As a result, you have a complete, end-to-end view of your data on lineage. For example, you can integrate with OpenLineage in the following cases:

Your data flows through data sources that are supported, and those data sources that are not currently supported for lineage import. By importing events, your data flow becomes complete and is easier to follow. For example, you have a database as a source and a custom ETL tool that is built in-house and that moves data to Amazon S3. By importing an event and creating a mapping for it, you can correctly connect source and target assets.
You use a technology that is not supported for lineage import, for example Azure Data Factory. With OpenLineage events, you can add its activities into your lineage.

Types of events that are processed

The following OpenLineage event types are processed:

Job events. They represent a static, design-time lineage. Job events are ingested, stored, and processed. Job-level inputs and outputs are retrieved, along with column-level lineage, if the column lineage facet is present.
Run events. They represent a runtime lineage. Run events are ingested, stored, and processed. They describe the state changes of the execution of a job, start, complete, and failure. Job-level inputs and outputs are retrieved, along with column-level lineage, if the column lineage facet is present.

The OpenLineage payloads are provided in the JSON format.

Ingestion methods

You can ingest OpenLineage events by using the following methods:

HTTP API ingestion. The OpenLineage events can be received directly by using authenticated HTTPS endpoints.
A .zip file upload. You can upload a .zip file that contains .json files with events. You can upload the file in the product user interface, or by using API.

Mapping rules

To correctly process data that is referenced in OpenLineage events, create mapping rules. Each rule defines how vendor-specific payloads are mapped to asset types on lineage. The mappings are based on the namespace, so three mappings are needed for one event. Default mappings for known technologies are already present in the product. Create new mappings for custom technologies. When the datasets that are referenced in mappings are mapped to known data sources by using data source definitions, OpenLineage edges are connected with data that is discovered by default connectors. As a result, an integrated, end-to-end lineage graph is created, capturing design-time, runtime, and automatically discovered artifacts.

For more information about mappings, see Mapping OpenLineage events.

Ingesting OpenLineage events

The OpenLineage events must be provided in the JSON format, and conform to the OpenLineage specification. For details, see OpenLineage Integrations.

Ingesting payloads by using HTTP API

With HTTP API method, you can ingest a single event or a batch (array) of events. You can ingest only run events with this method. To ingest job events, upload a .zip file with the metadata import job.

Prerequisites
To use the HTTP API method, you must have the Lineage Administrator role or a custom role with the Manage data lineage permission.

Additionally, a user API key is required. You can manage them in your account profile in Profile and settings > User API key. For details, see Managing the user API key.

API endpoint to ingest a single event
The following API endpoint is used for ingestion of a single event:

HTTP method: POST
HTTP URL: https://<HOST>/gov_lineage/v2/lineage_events/openlineage
Payload: valid OpenLineage event JSON payload

To emit events to this API, use either OpenLineage producers, or client libraries, for example Java. Go to the Configuration section of a client library for details, including authentication configuration, for example JWT Token Provider.

API endpoint to ingest events in batch
The following API endpoint is used for ingestion of OpenLineage events in batch:

HTTP method: POST
HTTP URL: https://<HOST>/gov_lineage/v2/lineage_events/openlineage/batch
Payload: an array of valid OpenLineage event JSON payloads

Uploading .zip file

You can compress the .json file that contains OpenLineage event to a .zip file, and upload the .zip file during metadata import. The .zip file can contain one or more events. Compress the events directly to the .zip file, so that it has the following structure:

openlineage-events.zip
├─ job_run_1.json
└─ job_run_2.json

You can upload the .zip file by using the product user interface, or by using APIs.

In the product user interface, upload the .zip file as an external input during the metadata import. See Creating metadata imports.
- You can include or exclude assets by using job namespaces in OpenLineage events. The whole input is evaluated as a regular expression. Example values:
  - myPrestoApp1Namespace: all events with job namespace myPrestoApp1Namespace.
  - mySparkApp[1-5]Namespace: all events with job namespace that starts with mySparkApp1Namespace and ends with a digit between 1 and 5.
To upload the .zip file by using APIs, follow these steps:
1. Create an asset for the .zip file. See Create an asset.
2. Upload the .zip file as the attachment to a project where you want to import metadata. See Create an attachment.
3. Create a data source definition of type OpenLineage.
4. Create a metadata import asset, select the data source definition that you created, and reference the attached .zip file as an input file. In the file_list parameter, provide the id of the .zip file. See Create a Metadata Import asset.
5. Create a job for the metadata import asset. See Create a new job.
6. Run the metadata import job. See Start a run for a job.

Monitoring OpenLineage events

Use the Processed OpenLineage events dashboard to monitor events that are received by the system through the API and track their processing status. To open the dashboard, go to Data > Data lineage > Map lineage > Map OpenLineage. From this view, you can review event volume and see whether individual events were processed successfully, failed, rejected, or are still pending.

To review the status of the events that are imported in a .zip file with a metadata import job, check the status of the particular metadata import job.

Use this centralized view of OpenLineage events to:

Verify events that were submitted through the API.
Troubleshoot ingestion issues by identifying failed, rejected, or pending events.
Monitor the overall health of the processing pipeline by reviewing trends over time.

The details table shows OpenLineage events that are grouped by a job run. The event identifier maps to a single job run, which can include multiple events. The processing status reflects the cumulative processing result for that job run. To view error messages for the failed or rejected events, expand the event identifier and click View for the selected event.

Design lineage and runtime lineage

You can import both design and runtime lineage. Learn about the differences, about lineage versions for both types, and how data can be reimported.

Design lineage

Design lineage represents the planned or intended flow of data through your system as defined during the design phase, before the actual processing. It shows a static structure of your data, for example of ETL jobs, database stored procedures, SQL code. Design lineage helps you understand the intended structure and relationships in your data ecosystem.

The transormation definitions, and thus design information, change over time. To keep track of these changes, versions of lineage are created and saved. See Previous versions of the data lineage.

Design lineage can also be reimported when needed, to replace an old version with the new version.

Runtime lineage

Runtime lineage captures the actual flow of data as it moves through your system during execution. It represents what actually happened to your data during processing. It shows dynamic data, such as the jobs that were executed. Runtime lineage helps you verify that data is flowing as expected and troubleshoot when it isn't.

The concept of lineage versioning is not that relevant for the runtime lineage. However, you can still see when the runtime lineage occurred. When you view lineage history, runtime lineage is available for the last 30 days from the selected time instance in the lineage history. The value that is compared to that interval is the eventTime value from the COMPLETE event of the job run that states when the job execution finished.

It is not possible to reimport the runtime lineage. When the new runtime lineage is processed, the old one is not affected. If you want to replace the runtime lineage, delete the existing version and import a new one.

Limitations

For runtime lineage events, the runId value in the event payload must be a valid UUID.
Only jobs that finished in last 30 days are shown by default. If historical lineage is used, the 30 day window is relative to the selected time in the past. The information about when the job execution finished is taken from the eventTime parameter of the COMPLETE event of the job run.

Exporting lineage to OpenLineage format

You can export your lineage to the OpenLineage format and use it in any tools that support this standard. For details, see Exporting data lineage to OpenLineage format.

Learn more

OpenLineage documentation