TLS in IBM Automatic Data Lineage

TLS is typically used for the following in the Automatic Data Lineage context.

Automatic Data Lineage defines the following truststores.

• Connectors truststore of Manta Flow CLI

  • Certificates can be imported via Admin UI, Configuration tab, CLI section, Common Config page.

  • Certificates can be imported via the Orchestration API endpoint [POST] /public/configurator/v1/configurations/CLI/Common/Common Config/keystores/manta.cli.mantaConnectors.settings/import.

• System truststore of Manta Flow CLI

  • Certificates can be imported via Admin UI, Configuration tab, CLI section, Common Config page.

  • Certificates can be imported via the Orchestration API endpoint [POST] /public/configurator/v1/configurations/CLI/Common/Common Config/keystores/manta.cli.systemTruststore.settings/import.

• System truststore of Manta Flow Server

  • Certificates can be imported via Admin UI, Configuration tab, Server section, System properties page.

  • Certificates can be imported via the Orchestration API endpoint [POST] /public/configurator/v1/configurations/FLOW_SERVER/Common/System properties/keystores/manta.server.systemTruststore.settings/import.

The truststores do not exist by default. If the UI is used for the certificate import, the truststore first has to be created using the Create button. When the API is used, the truststore is automatically created if it doesn’t exist; there is no need to explicitly create it up front.

Protecting Automatic Data Lineage UIs and APIs

HTTPS may be turned on for all applications. By default HTTPS is turned off. When HTTPS is turned on, the server user interface and APIs start working only through HTTPS.

For more information about HTTPS configuration, see:

• Configuring HTTPS

Protecting Communication between Automatic Data Lineage Services

This requires the UIs and APIs of all Automatic Data Lineage services to be TLS protected. (See the previous section for more details.) Once that is done, we need to ensure that a TLS handshake can be successfully established between all services that communicate with each other. This means uploading the certificate(s) used to protect the services to the following truststores.

• System truststore of Manta Flow CLI

• System truststore of Manta Flow Server

Protecting Communication between Automatic Data Lineage and External Identity Providers

There are three different cases.

• Automatic Data Lineage integrates with Keycloak (which is part of standard Automatic Data Lineage deployment), and Keycloak integrates with the external identity provider via OIDC/SAML/LDAP.

  • All configuration is done in Keycloak in this case. The details are described in User Management.

• Automatic Data Lineage does not integrate with Keycloak but instead integrates with the external identity provider directly via the Gateway authentication mechanism.

• Automatic Data Lineage does not integrate with Keycloak but instead integrates with the external identity provider directly via the OIDC authentication mechanism.

Protecting Communication between Automatic Data Lineage (Scanners) and the Scanned Source Systems

Scanners that have explicit TLS support implemented by Automatic Data Lineage utilize the connectors truststore of Manta Flow CLI. The source system certificate has to be imported there. The scanners supporting connector truststore are:

• PostgreSQL scanner

• DB2 scanner

• MS SQL scanner

• Hive scanner

• Teradata scanner

• Kafka scanner

• SSIS scanner

• StreamSets scanner

• OBIEE scanner

• ODI scanner

• Oracle scanner

• Power BI scanner — only in LOCAL extraction mode

• SSRS scanner

• Qlik Sense scanner

• Tableau scanner

Some JDBC-based scanners do not have explicit TLS support, but they support TLS via parameters passed in the JDBC connection string in their connection configuration. See the documentation for those scanners for more details. The scanners supporting this method are:

• Netezza scanner

• SAP Hana scanner

Cloud-based technologies do not need any certificate handling, as their certificates are always signed by public certification authorities, and are thus trusted by default. An example of such is the Snowflake scanner.

Protecting Communication between Automatic Data Lineage (Exporters) and the Third-Party Solutions That Automatic Data Lineage Integrates With

This section covers integration with data catalogs and data governance tools (e.g., Alation, Collibra, Informatica EDC, IBM IGC) and integration with vaults (e.g., CyberArk). Automatic Data Lineage communicates with these systems’ APIs via HTTPS.

For CyberArk integration, the CyberArk public key for signing has to be imported to the:

• Manta Flow CLI system truststore

• Manta Flow Server system truststore

For other integrations, importing the respective public key for signing to the Manta Flow CLI connectors truststore is sufficient.