Manta Flow Alation: End-User Documentation

Definition of Exported Entities

Manta Flow analyzes SQL scripts (procedures, view definitions, macros, ad-hoc scripts, etc.) as well as ETL, analytical, and reporting tools. Then, it pushes metadata about every interesting SQL statement (basically, that means every SQL query which directly transfers data—inserts, updates, deletes, etc.), all ETL transformations, and all analytical models and reports to Alation. The export is based on the latest revision in the IBM Automatic Data Lineage repository. Manta Flow creates new dataflow objects for each transformation described. These dataflow objects are connected to the database columns whose data has been transferred.

The documentation below utilizes screenshots from Automatic Data Lineage to illustrate which entities are and are not exported to Alation. The ones that are not exported are tagged with a red X. Please note that the lineage through a non-exported object is exported if the non-exported object is between two exported assets. See the first screenshot for an example.

Entities Exported by Connector Type

• Database connectors (Db2, MS SQL, Netezza, Oracle, PostgreSQL, Teradata)

  • Transformation assets (such as procedures, functions, scripts, and triggers) are exported.

    • If there is a sequence of transformations (e.g., a procedure calling a function that calls another function), only the last transformation applied is exported (in this case, the procedure) and the lineage through the functions is contracted without listing the specific intermediate transformation object.

      

  • The lineage between the transformation assets and assets provided by Alation (such as tables) is exported.

• Data integration connectors (DataStage, IPC, ODI, OpenLineage, SSIS, Talend)

  • The lineage between the data integration objects and database/reporting/analytical objects is exported.

    • If there is a sequence of transformations, only the last transformation applied is exported and the lineage through the functions is contracted without listing the specific intermediate transformation object.

      

• Analytical connectors (SSAS)

  • Both the analytical assets and the lineage between them are exported as well as the source/target assets.

• Reporting connectors (Cognos, Excel, OBIEE, Power BI, Qlik Sense, SAP BO, SSRS, StreamSets, Tableau)

  • Both the reporting assets and the lineage between them are exported as well as the source/target assets.

    

• Open Manta Extensions connectors for Custom Database (follow the structure described in Open Manta Extensions: Template Models).

  • Both the custom database assets (table, column) and the lineage between them are exported.

Entities That Are Not Exported

• Files - up to and including R42.11.

• Tables (as they should be available in Alation via native scanners). This information is not applicable to Custom Database.

• Views (as they should be available in Alation via native scanners); however, create view scripts are exported as transformations (dataflow objects), so the transformation details will be available in Alation. This information is not applicable to Custom Database.

• Pseudo-columns and lineage into pseudo-columns; an example of a pseudo-column is RowNumber

  

• Deduced tables/views and lineage to/from deduced tables/views (as they are not available in Alation; all tables/views should be scanned from the database and be standard objects)

  

• Indirect (filter) lineage

  

• Standalone insert, delete, and truncate statements

  

• Master selects (e.g., select used as a procedure output)

Exceptions

• (MS SQL) Lineage to volatile and global temporary tables is not exported.

• (Oracle) External system cursors (created in external systems such as a Java application) are exported.

  • Both the cursor itself and the upstream lineage are exported.

  • The cursor is only exported if there is no downstream lineage from it.

    

• (PostgreSQL) Lineage to volatile and global temporary tables is not exported.

• (Teradata) Lineage to volatile tables is not exported.

• (Teradata) Downstream lineage from tables/views created in scripts is not exported.

Export Model

The diagram below illustrates a general domain model of Alation—the classes and their associations that Automatic Data Lineage uses in Alation.

The lineage for (database) synonyms is exported as well. Not all synonyms are ingested by Alation. (Synonyms are supported for MS SQL and Oracle at the time of writing, and even for these technologies, the ingestion of synonyms has to be explicitly turned on.) In some cases, this might lead to the creation of temporary objects in Alation.

Browse Assets

To find an Alation source or BI server containing assets uploaded by Automatic Data Lineage, use the left menu on Alation’s main screen.

At the moment, Automatic Data Lineage uses/creates:

• One source in Alation per each Automatic Data Lineage database server node. The source has to be created manually in Alation before the export is executed. For more information, see Manta Flow Alation: Alation Configuration. This Alation source contains the data assets loaded by Alation (typically database data assets).

• One BI server in Alation per each Automatic Data Lineage analytical/reporting tool server node. The BI server can be created manually before the export is executed, or it can be executed automatically by Automatic Data Lineage during the export. For more information, see Manta Flow Alation: Alation Configuration. This Alation BI server contains the analytical/reporting assets loaded by Automatic Data Lineage.

• One VDS server in Alation per each Automatic Data Lineage custom database server node. The VDS server can be created manually before the export is executed, or it can be executed automatically by Automatic Data Lineage during the export. For more information, see Manta Flow Alation: Alation Configuration. This Alation VDS server contains the custom database assets loaded by Automatic Data Lineage.

• Additionally, Automatic Data Lineage creates dataflow objects in Alation that represent the data lineage for all sorts of source systems. But the dataflow objects are not tied to a specific source or BI server in Alation.

In order to find the assets (such as tables or reports), use the search feature on Alation’s main screen. If the searched asset is a reporting asset, make sure that BI server objects are included in the search.

Dataflow Objects and Lineage Diagrams

Lineage diagrams can be rendered for a particular table or report.

1. Lineage diagrams are available under the Lineage tab on the asset overview screen.

   

2. After clicking on the Lineage tab, the lineage diagram is shown. By default, Alation will hide the columns of the tables and reports, but they can be shown by expanding the tables/reports. The lineage for a specific column can be highlighted by clicking on the column name.

   

3. Additional metadata uploaded by Automatic Data Lineage can be found in the lineage view, mainly in the detail of dataflow objects. Clicking on a specific dataflow object (the gray circle), displays its detail containing the content (the source code in the case of database transformations) and the description (containing hyperlinks to Automatic Data Lineage Viewer, transformation logic for individual columns of the transformation, and the export date). When you view data lineage diagrams in Alation, you can click a dataflow object to see more information, such as transformation logic for that object, the date it was exported, and a link to view it in Automatic Data Lineage. The link is available only for dataflow objects. For other objects like data assets, BI data sources, or BI reports, the link is not exported.

   

4. Dataflow objects can also be found directly using the search feature on Alation’s main screen.

   

Export Statistics

The export log file contains information about the file sizes for Alation. This can be used as an indication that many assets have been exported and will be uploaded to Alation.

2020-09-25 14:33:10.326 [pool-2-thread-1] INFO  alation_test_db2.properties:0 - eu.profinit.manta.dataflow.repository.exporter.client.ExportReader Export finished with the status 200.
2020-09-25 14:33:10.395 [pool-2-thread-1] INFO  alation_test_db2.properties:0 - eu.profinit.manta.dataflow.repository.exporter.client.FileLogger Decompressed file bi_server_DB2_alation_test_db2_1.json with size 438 bytes.
2020-09-25 14:33:10.403 [pool-2-thread-1] INFO  alation_test_db2.properties:0 - eu.profinit.manta.dataflow.repository.exporter.client.FileLogger Decompressed file dataflow_DB2_alation_test_db2_1.json with size 47717 bytes.
2020-09-25 14:33:10.406 [pool-2-thread-1] INFO  alation_test_db2.properties:0 - eu.profinit.manta.dataflow.repository.exporter.client.FileLogger Decompressed file lineage_DB2_alation_test_db2_1.json with size 1547 bytes.