Manta Flow IBM IGC User Documentation
This document introduces how Manta Flow for IBM IGC analyzes dataflows in SQL scripts and ETL, analytical, and reporting tools and how it is visualized in IBM Information Governance Catalog.
Structure of the Metadata Created
Manta Flow for IBM IGC analyzes SQL scripts (procedures, view definitions, macros, ad-hoc scripts, and so on.) and ETL, analytical, and reporting tools. It then imports metadata about all interesting SQL statements (basically, that means every SQL query which directly transfers data—inserts, updates, deletes, etc.), ETL transformations, and analytical models and reports. The export is based on the latest revision in the IBM Manta Data Lineage repository. Manta Flow creates new assets for each transformation described. These new assets have hierarchical parents, so they are well arranged and easy to find. These assets are also connected to the database or file columns whose data has been transferred.
The Manta Meta Model structure is as follows:
- Technology
- Such as MS SQL / Teradata / Oracle / Netezza / DB2 / Postgre / IPC / Cognos / Excel / ODI / SSIS / SSAS / SSRS / Talend
- Resource
- Identifier of the system analyzed
- Group
- Hierarchy based on a specific dialect
- Child types—Database, Schema, Package, Directory, Repository, Workflow, Server, Project, Report, Layout
- Input
- Asset representing a transformation unit
- Child types—Procedure, Function, Trigger, CreateView, Script, Session, Task, Job, ReportItem, Cube, AnalyticalModel, DataModel, Page
- Operation
- Asset representing one specific operation
- Child types—Statement, Transformation, DataSeries, Dimension, Set, AnalyticalTable, KPI, Visualization, Property
- Leaf
- Asset representing the transformation of one value
- The flows are only on this level
- Child types—Column, Port, Attribute, Measure, CalculatedMember, Level
Browse Assets
Manta Flow for IBM IGC imports new assets in the standard way, so it is possible to use IGC features just as for other metadata. There are two basic ways to find a specific element.
- Hierarchical view—Figure 1
- Browse by type—Figure 2 and Figure 3
Figure 1: Browse assets in the hierarchical view
Figure 2: List of types in the Manta meta model
Figure 3: Asset list of a specific type
Data Lineage
Default data lineage is at the table level, as shown in Figure 4. Manta Flow also aggregates lineage on the input level, and therefore, the whole lineage is more clear. The blue assets were created by the Metadata Asset Manager while loading the MS SQL resource. The yellow assets were created by Manta Flow IGC. Manta Flow IGC also created all the relationships shown in Figure 4.
Figure 4: Default lineage at table level
IGC can show lineage at the column level. In that case, the data lineage looks like that in Figure 5.
Figure 5: Data lineage at column level
It is possible to inspect individual statements for a specific transformation unit by expanding them, as shown in Figure 6.
Figure 6: Data lineage through statements