Manta Flow IBM IGC User Documentation

This document introduces how Manta Flow for IBM IGC analyzes dataflows in SQL scripts and ETL, analytical, and reporting tools and how it is visualized in IBM Information Governance Catalog.

Structure of the Metadata Created

Manta Flow for IBM IGC analyzes SQL scripts (procedures, view definitions, macros, ad-hoc scripts, and so on.) and ETL, analytical, and reporting tools. It then imports metadata about all interesting SQL statements (basically, that means every SQL query which directly transfers data—inserts, updates, deletes, etc.), ETL transformations, and analytical models and reports. The export is based on the latest revision in the IBM Manta Data Lineage repository. Manta Flow creates new assets for each transformation described. These new assets have hierarchical parents, so they are well arranged and easy to find. These assets are also connected to the database or file columns whose data has been transferred.

The Manta Meta Model structure is as follows:

Browse Assets

Manta Flow for IBM IGC imports new assets in the standard way, so it is possible to use IGC features just as for other metadata. There are two basic ways to find a specific element.

Figure 1 - browse assets in hierarchical view

Figure 1: Browse assets in the hierarchical view

Figure 2 - list of types in the Manta Meta Model

Figure 2: List of types in the Manta meta model

Figure 3 - assets list of specific type

Figure 3: Asset list of a specific type

Data Lineage

Default data lineage is at the table level, as shown in Figure 4. Manta Flow also aggregates lineage on the input level, and therefore, the whole lineage is more clear. The blue assets were created by the Metadata Asset Manager while loading the MS SQL resource. The yellow assets were created by Manta Flow IGC. Manta Flow IGC also created all the relationships shown in Figure 4.

Figure 4 - default lineage at table level

Figure 4: Default lineage at table level

IGC can show lineage at the column level. In that case, the data lineage looks like that in Figure 5.

Figure 5 - data lineage at column level

Figure 5: Data lineage at column level

It is possible to inspect individual statements for a specific transformation unit by expanding them, as shown in Figure 6.

Figure 5 - data lineage through statements

Figure 6: Data lineage through statements