OpenLineage integration for off-platform assets

The process of integrating Data Observability with your system consists of the following steps:

Generate an authentication token. The script provided in this document uses username/password to generate the token but any Cloud Pak for Data authentication method can be used. In the following script, the username and the password are your Cloud Pak for Data account credentials.
```
curl -k -X POST \
  {environment-url}/icp4d-api/v1/authorize \
  -H 'Content-Type: application/json' \
  -d '{
    "username": "{username}",
    "password": "{password}"
  }'
```
Log in to watsonx.data integration and copy the project ID where you want to observe job runs. You can find the project ID in the Manage tab of the project. If you don't have a project, create one and copy its ID. You must have an existing project to receive the job metadata.
Write a code to connect your system with Data Observability.
Run the code to pull the data from your system.
Convert the received data to the request payload that conforms to the Data Observability schema.
In step 2, you generated a token. Use this token to authenticate the request by including it in the request sent to the API endpoint.
- Use the token as a header:
"Authorization": f"Bearer {token}"
- The API endpoint uses the following format for both single and batch event submission:
https://{watsonx_hostname}/data-obs/v1/lineage
Run the code periodically to send your data.

OpenLineage API

For a list of events that you can send with OpenLineage, see OpenLineage documentation.

Payload

Make sure that you add a facet that is called IBMworkspace inside your job facets. For IBMworkspace provide one of the following fields:

project_id: For development and collaboration environments
space_id: For production deployment environments

In this way, you specify where your lineage events are stored and displayed in watsonx.data integration. For example, if you have job A that needs to appear in Project A and job B that needs to appear in Project B, you specify different project_id values in the IBMworkspace facet for each job's events. As a result, you can control which events are assigned to what project, even when all events come from the same source system.

Payload example

Use this example as a reference to build and send your own OpenLineage event payload to the Data Observability API.

curl -k -X POST \
"https://{url}/data-obs/v1/lineage" \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $OPENLINEAGE_TOKEN" \
-d '{
  "eventType": "COMPLETE",
  "eventTime": "2026-07-16T07:24:00Z",
  "inputs": [],
  "job": {
    "facets": {
      "IBMworkspace": {
        "_producer": "https://some.producer.com/version/1.0",
        "_schemaURL": "https://openlineage.io/spec/facets/1-0-0/IBMWorkspaceFacet.json",
        "project_id": "project_id"
      },
      "jobType": {
        "integration": "my-engine",
        "processingType": "batch"
      },
      "externalLink": {
        "_producer": "https://some.producer.com/version/1.0",
        "_schemaURL": "https://openlineage.io/spec/facets/1-0-0/ExternalJobLinkFacet.json",
        "url": "https://link.to.your.job.com"
      }
    },
    "namespace": "your-job-namespace",
    "name": "your-job-name"
  },
  "outputs": [],
  "run": {
    "facets": {
      "tags": {
        "_producer": "https://some.producer.com/version/1.0",
        "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/BaseFacet",
        "runName": "test_run_name"
      },
      "metrics": [
        {
          "metricName": "rows_processed",
          "timestamp": "2026-07-16T07:24:00Z",
          "metricValue": 4763.99,
          "_producer": "https://some.producer.com/version/1.0",
          "_schemaURL": "https://raw.githubusercontent.com/OpenLineageOpenLineage/main/spec/OpenLineage.json#/definitions/BaseFacet"
        },
        {
          "metricName": "execution_time_ms",
          "timestamp": "2026-07-16T07:24:00Z",
          "metricValue": 2024.9,
          "_producer": "https://some.producer.com/version/1.0",
          "_schemaURL": "https://raw.githubusercontent.com/OpenLineageOpenLineage/main/spec/OpenLineage.json#/definitions/BaseFacet"
        }
      ],
      "externalLink": {
        "_producer": "https://some.producer.com/version/1.0",
        "_schemaURL": "https://openlineage.io/spec/facets/1-0-0/ExternalRunLinkFacet.json",
        "url": "https://link.to.your.run.com"
      }
    },
    "runId": "10dd62e6-f4f0-4b72-94c3-be43fa90f3b3"
  },
  "producer": "https://custom.api",
  "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
}'