Integrating with IBM Manta Data Lineage

Integrating Manta with watsonx.data allows you to capture and publish jobs, runs, and dataset events from Spark and Presto through the Manta UI, providing full visibility into data flows and transformations.

watsonx.data on IBM Software Hub

watsonx.data on IBM Software Hub

Before you begin

  • Create a data source definition and an OpenLineage connection.
  • Create an OpenLineage metadata import in your project and run it.
  • Software Hub endpoint provided must be external URL and it’s SSL certificate must be installed the cluster. To generate the self-signed SSL certificate follow 
    export STORE_ENDPOINT=<endpoint>:<port number>
echo QUIT | openssl s_client -showcerts -connect $STORE_ENDPOINT | awk '/-----BEGIN CERTIFICATE-----/ {p=1}; p; /-----END CERTIFICATE-----/ {p=0}' | awk '{printf "%s\\n", $0}'
    For information, see Creating a secret to store shared custom certificates.
    1. Run the following command to login to the Software Hub cluster:
      oc login a<endpoint:port> -u kubeadmin -p <password> --insecure-skip-tls-verify=true
    2. Import the certificate from the manta configured cluster:
      export STORE_ENDPOINT=<endpoint>:<port number>
export CERT=$(echo QUIT | openssl s_client -showcerts -connect $STORE_ENDPOINT | awk '/-----BEGIN CERTIFICATE-----/ {p=1}; p; /-----END CERTIFICATE-----/ {p=0}' | awk '{printf "%s\\n", $0}')
    3. Run the following command to patch watsonx.data service instance with the new certificate:
      oc patch wxd/lakehouse --type=merge -n ${PROJECT_CPD_INST_OPERANDS} -p "{ \"spec\": {
          \"update_ca_certs\": true,
          \"extra_ca_certs_secret\": \"$CERT\"
      } }"
For more information, see Preparing data for IBM Manta Data Lineage.

Procedure

  1. Log in to the watsonx.data console.
  2. From the navigation menu, go to Configurations > IBM Manta Data Lineage.
  3. Under Details, choose the connection type:
    • Native endpoint
    • External endpoint
  4. Depending on the selected connection type, enter the following details:
    Field Description
    Lineage ingestion endpoint (visible only for external endpoint) Enter the Software Hub host endpoint URL where the data lineage service is activated.
    Username Enter your username for the host endpoint.
    API key Enter the API key. For information about generating API key, see Platform API keys.
  5. Click Save to save the details.
    You can edit the saved details by clicking Edit.
  6. Click Enable to enable Manta Data Lineage.
    Note:
    • Data lineage takes effect for new jobs that are started after enabling it. Previous and ongoing jobs will not display data lineage.
    • Currently, it supports viewing lineages for CREATE TABLE AS (CTAS) and INSERT INTO SELECT operations in Manta Data Lineage.

What to do next