Manta Flow Architecture

Manta Flow Components

Manta Flow is a data lineage platform consisting of three major components.

• Manta Flow CLI - Java command-line application that extracts all scripts from source databases and repositories, analyzes them, sends all gathered metadata to the Manta Flow Server, and optionally, processes and uploads the generated export to a target metadata database

• Manta Flow Server — JavaServer application that stores all gathered metadata inside its metadata repository, transforms it to a form suitable for import to a target metadata database, and provides it to its visualization or third-party applications through API

• Manta Admin UI (runs on the Manta Flow Service Utility application server) — JavaServer application that provides a graphical and programming interface for installation, configuration, updating, and overall maintenance of Manta Flow. The application consists of several modules:

  • Manta Installer - a native application used to install Manta applications (Manta Flow, Manta Admin UI). Windows and Unix distributions of the application are available. Once executed, an installation wizard opens. The installation wizard allows the Manta administrator to do a fresh installation of Manta or, if an older version of Manta is already installed on the operation system, it can begin the update of this Manta instance.

  • Manta Configurator - a module of the Manta Admin UI web application. It allows the Manta administrator to:

    • Configure connections to source systems that should be analyzed by IBM Automatic Data Lineage lineage scanners (Manta Flow CLI)
    • Configure the general behavior of Automatic Data Lineage scanners (Manta Flow CLI)
    • Configure the general behavior of the Automatic Data Lineage repository (Manta Flow Server)
    • Configure the general behavior of Automatic Data Lineage visualization (Manta Flow Server)
    • Configure the authentication and authorization of Manta Flow Server and its graphical (data lineage visualization) and programming Manta Repository API, export/import features) APIs For more details of these functionalities see IBM Automatic Data Lineage Configuration

  • Manta Process Manager - a module of the Manta Admin UI web application. It allows the Manta administrator to:

    • Define Manta Workflows — definitions of a process that may contain the following phases - Metadata extraction from source systems for data lineage analysis; Data lineage analysis (analysis of metadata extracted during the extraction phase, manually provided metadata and/or scripts, and custom metadata); Export of data lineage available in the Automatic Data Lineage repository to output files and/or third-party applications
      • Execute (and terminate) Manta Workflows
      • Monitor Manta Workflow execution For more details of these functionalities see Manta Process Manager.

  • Manta Log Viewer - a module of the Manta Admin UI web application. It allows the Manta administrator to:

    • Browse and filter log entries created by Manta Workflow executions, Manta Viewer actions, Repository API requests
    • Create Manta diagnostic packages — bundles of Manta logs that can be investigated by Manta support if an issue occurs
      For more details of these functionalities see Manta Log Viewer.

  • Manta Orchestration API - REST API programming interface. It allows the consumer to:

    • Configure connections to source systems that should be analyzed by Automatic Data Lineage scanners (the functionality equivalent to the functionality of Manta Configurator, accessible through the graphical user interface)
    • Define Manta Workflows (the functionality equivalent to the functionality of Manta Process Manager, accessible through the graphical user interface)
    • Execute (and terminate) Manta Workflows (the functionality equivalent to the functionality of Manta Process Manager, accessible through the graphical user interface)
    • Fetch the status of running Manta Workflows (the functionality equivalent to the functionality of Manta Process Manager, accessible through the graphical user interface)
    • Acquire outputs of finished Manta Workflows

The installation and update process are described in Manta Admin UI

• Manta Configuration Service — stores all Manta configuration. Runs as a SpringBoot application and is backed by Apache Tomcat web server, see Manta Flow Configuration Framework

• Manta Launcher - orchestrates startup and shutdown of all Manta Flow components. Runs as a SpringBoot application and backed by Apache Tomcat web server, see Manta Launcher

• Keycloak — Identity access management for IBM Automatic Data Lineage, see Manta Identity Management: Keycloak

Manta Flow High-Level Architecture

The following diagram shows the high-level dependencies between the preceding list of Manta Flow components.

Manta Flow CLI and Manta Flow Server Detailed Architecture

The following diagram illustrates Manta Flow's architecture, components, and interactions with third-party resources.

Manta Flow Deployment Diagram: Installer

All components are deployed to the same physical or virtual server. Manta UI is accessed through a web-based interface, so there is no need to install anything on the end-user machine or on source systems such as ETL and database servers.

Manta Flow Security Architecture

The following diagram illustrates a high-level security architecture. This diagram doesn’t consider internal communication between Automatic Data Lineage services and components that are only used internally.