GitHubContribute in GitHub: Edit online

Manta Flow Configuration Framework

The purpose of this page is to explain in general how configurations are handled in Manta Flow and Manta Admin UI products.

Product Default and User-Set Configurations

IBM Manta Data Lineage works with the concept of product default and user-set configuration.

The (product) default configuration is bundled as part of the product binaries, it is not supposed to be changed either during the installation of Manta Data Lineage or after it, and it is used during runtime whenever there is no user-set configuration that would override it. Please note that the product default configuration is overwritten during upgrades without any warnings. Each Manta Data Lineage component — Manta Flow Server, Manta Flow CLI, and Manta Admin UI — has its own default configuration. Most configurations (or configurable features) have a default configuration, but there are also pieces of configuration that don’t have one as no reasonable defaults can be provided (such as dictionary mappings or export mappings).

Whenever users make configuration changes to Manta Data Lineage components through Admin UI or API, these changes are stored as user-set configurations. This means that the configuration is stored separately from the default configuration and that (in most cases) only the values that are actually changed are stored (the rest of the values are still loaded from the default configuration).

This concept allows both seamless upgrades of Manta Data Lineage (as the user-set configuration is preserved) and automatic propagation of new configuration defaults introduced in new versions of Manta Data Lineage (as the product default configuration is simply overwritten).

Configuration Service

Manta Data Lineage comes with a separate service that handles the configuration, called Manta Configuration Service. The service honors the principles described above and encapsulates them. Customers are not expected to work directly with this service, but rather to use Manta Admin UI or Manta Orchestration API to retrieve or update the configuration.

Here are some basic instructions for operating Manta Configuration Service.

Configuration Service comes with startup, shutdown, and check-status scripts (the last one only for Linux) that are to be used for the respective operations. The scripts are located in <manta_home>/configurationservice/bin/.

All persistent files (internal configuration, logs, data — configuration of other Manta Data Lineage services) are by default stored in <manta_home>/configurationservice/manta-configuration-service-dir.

The internal configuration of Configuration Service is very minimalistic. It consists of:

The logs are by default located in <manta_home>/configurationservice/manta-configuration-service-dir/logs.

The following API endpoints can be used to verify whether Configuration Service is running and ready to accept requests (in the same order).

If needed, an endpoint exposing runtime metrics can be enabled. This can be done by putting the property management.endpoints.web.exposure.include=health,prometheus into <manta_home>/configurationservice/manta-configuration-service-dir/conf/application.properties or by passing it to Configuration Service as a system property during startup. If the endpoint is enabled, it is exposed at the URL http://localhost:8083/actuator/prometheus.

To back up / restore all configurations that Configuration Service manages (including its internal configuration), back up / restore the <manta_home>/configurationservice/manta-configuration-service-dir directory. Configuration Service has to be shut down while the backup/restoration is performed.

How to Pick Up Connection Configurations from a File Instead of Configuration Service for Scanners That Have Not Been Integrated with Admin UI Yet

Manta Data Lineage will fetch connections in Configuration Service by default. While this is desirable for the majority of scanners, this approach is not applicable to scanners that have not been integrated with Manta Admin UI yet (bytecode scanner). For these scanners, it is still necessary to pick up connection configurations from the filesystem. To make this possible, adjust the file mantaflow/cli/scenarios/manta-dataflow-cli/etc/masterScenarioTemplate.xml and uncomment the line <constructor-arg name="inputFile" value="${master.dir.properties}" />. Please note that this change will apply to all connections of all scanners. All scanners will search the connections in filesystem.

<bean id="inputReader" class="eu.profinit.manta.platform.automation.ConnectionReader">
      <!--
      <constructor-arg name="inputFile" value="${master.dir.properties}" />
      -->
      <constructor-arg name="inputFile">
          <null />
      </constructor-arg>