GitHubContribute in GitHub: Edit online

How to Rename Nodes in the IBM Manta Data Lineage Repository (aka How to Apply Aliases)

Goal

The physical names of some objects in the Manta Data Lineage repository may not be ideal for end users who may not be familiar with them. Manta Data Lineage provides the option to give specific objects user-defined names using Manta Alias Display User-Supplied Aliases. The most common use of Manta Alias is to change the names of physical servers. Users are usually familiar with the names of databases, schemas, tables, and columns.

This article illustrates two specific examples of how to import Manta Aliases for server names—WIN-RRR5IO49B55 under MS SQL and mantatestdev144814 under Tableau.

No alt text provided

This can be adjusted for use with other objects as well.

We would like to achieve something like the following.

No alt text provided

Note that the top level of the repository (all the technologies such as Big Query, Oracle, MS SQL, Tableau, etc.) cannot be renamed using the alias feature. If you would like to rename top-level objects, use Open Manta Perspectives instead.

Instructions

Manta Alias import is handled through Open Manta Extensions. The high-level process is:

  1. Create a connection for Open Manta Extensions to import aliases in Admin UI.
  2. Create Open Manta Extensions CSV files to import the aliases.
  3. Load them into the Manta Data Lineage repository.
  4. Review the results in the Manta Data Lineage UI and log files.
  5. Fix any issues by repeating steps 1–5.

Creating a Connection for Alias Import

  1. Create a new Open Manta Extensions connection in Admin UI → Connections → Add Connection, type in Open MANTA Extension.
  2. Enter a Connection Name and Connection ID: for example, MANTAAlias.

Creating Extensions CSV Files

Use Open Manta Extensions: Usage to import the MANTA_ALIAS attribute. According to the Open Manta Extensions format, the objects that we are referencing need to be defined as well.

Create the following four files the in mantaflow/cli/input/import/MANTAAlias/ folder, or upload them later via Process Manager (manually) or Orchestration API.

  1. Create the file node_attribute.csv with contents as follows. You will need to adjust this based on the technologies applicable to your case. The node_id represents a fully qualified path to the node within Manta Data Lineage repository.

    node_attribute.csv

    "node_id","attribute_name","attribute_value"
"/MSSQL/WIN-RRR5IO49B55","MANTA_ALIAS","IDW"
"/Tableau/mantatestdev144814","MANTA_ALIAS","Main Tableau"

  2. If you are using Manta Data Lineage R39 or older, you will also need the following files.

    1. Create a layer.csv file with contents as follows.

      layer.csv

      "layer_id","layer_name","layer_type"
"Physical","Physical","Physical"

    2. Create the file resource.csv with contents similar to those that follow. You will need to adjust this based on the technologies applicable to your case.

      resource.csv

      "resource_id","resource_name","resource_type","resource_description","layer_id"
"MSSQL","MSSQL","MSSQL","","Physical"
"Tableau","Tableau","Tableau","","Physical"

    3. Create the file node.csv with contents as follows. You will need to adjust this based on the technologies applicable to your case.

      node.csv

      "node_id","parent_id","node_name","node_type","resource_id"
"/MSSQL/WIN-RRR5IO49B55",,"WIN-RRR5IO49B55","Server","MSSQL"
"/Tableau/mantatestdev144814",,"mantatestdev144814","Tableau Site","Tableau"
Make sure that the file is delimited by commas and that there are no extra spaces around the commas or at the ends of lines. Such spaces usually cause issues during import.

A complete package based on the example above is available for download here:

aliases.zip

Loading Aliases into Manta Data Lineage

The files created in the previous steps are automatically ingested during the lineage analysis run that is triggered in Process Manager in Admin UI, Orchestration API, or Orchestration API scripts.

However, for a quick test, it is often more convenient to only run the import manually, as follows, or by creating the following workflow in Process Manager.

  1. New Minor Revision Scenario—to open a new minor revision and add the custom lineage to the last existing revision
  2. Import Dataflow Scenario—to ingest layer.csv, resource.csv, node.csv, and node_attribute.csv
  3. Commit Revision Scenario—to persist data into the repository; if you see any errors coming from the previous steps, you can also use the Rollback Revision Scenario to revert back and start over again
  4. Review the results in Manta Flow Server UI and also the Extensions import logs.

If you are not happy with the results and need to repeat the above steps, simply run the Delete Revision Scenario to remove the newly added minor revision and start over again.

Review Logs

Any errors reported during lineage import will help you identify any issues with the input file format. The logs are located under the LogViewer tab in Admin UI or directly on the filesystem under mantaflow/cli/logs/.