Manta Flow Integration Configuration

startNodeEvaluator

Evaluator to determine if the node is a target node of the flow on the column level. For example, in the case of databases, the target node type is typically Column or StructureItem.

False

reportLeafEvaluator (Collibra only)

Evaluator to determine if the node is a leaf of a report. For example, report leaves are nodes of the types Value, Data Point, Tooltip, Label, Data Point, Table Field, and Pivot Table Field.

False

locationInHierarchyNodeEvaluator

Evaluator to determine if there is a location leaf in the node’s hierarchy. This is by default used for Oracle dialect where ColumnFlow nodes can be non-atomic; that is, the ColumnFlow node is an ancestor of the transformation leaf, not the leaf itself.

False

startNodeOptionalEvaluator

Evaluator to determine whether the target* node is optional for the given transformation node. If this evaluator determines that the target node is required and the transformation does not have a target node, the transformation’s flow will not be exported.

By default, target nodes are typically not required for transformation nodes like Delete, TruncateTable, or Query.

False

linkedScriptNodeEvaluator

Evaluator to determine if it is a script node to be connected via an enum link.

False

printEnumeratedLinksEvaluator

Evaluator to determine which nodes are exported as “outer” nodes; that is, nodes usually imported by third-party tools through their native interface.

True

innerNodeEvaluator

Evaluator to determine which nodes are exported as “inner” nodes; that is, nodes provided exclusively by Manta Data Lineage, in addition to the ones imported by third-party tools through their native interface.

True

transformationClassificationEvaluator

Evaluator to determine whether this dialects supports transformation classification export; that is, export of the flag noting whether the transformation node transforms data or not.

False

TruePredicate

Returns True

NotPredicate

vertexPredicate: Vertex predicate

Returns the opposite of the vertexPredicate

OrPredicate

children: List of vertex predicates

When at least one child returns true, returns True

InChildHierarchyTypePredicate

typeSet: Set of allowed types

Returns True if and only if all the descendants of the node are of the allowed types

InHierarchyPropertyPredicate

vertexTypeSet: The set of vertex types on which the given property is searched

propertyKey: Key of the property

propertyValueSet: A set of values that a property must have

Returns True if somewhere in the hierarchy of the node is a node of a given type with a given property having a given value

IsLeafPredicate

Returns True if the node has no children

NamePredicate

nameSet: Set of allowed names

Returns True if the name of the node is in the nameSet

ParentTypePredicate

typeSet: Set of allowed types

Returns True if the parent of the node is a type from the typeSet

SinkPredicate

labels: Labels of edges( DIRECT, FILTER)

Returns True if the node is sink (has no edges of the specified labels)

TableSourcePredicate

typeSet: Set of allowed types

Returns True if the node is a column in the table and, additionally, this table has a source ( OBJECT_SOURCE_TYPE attribute) located in the given set of allowable values

VertexAttributePredicate

attributeKey: Key of the attribute

allowedValues: Set of allowed values

Returns True if the node has an attribute with the given key and a value from the set of allowed values

VertexTypePredicate

valueSet: Set of allowed values

Returns True if the type of the vertex is from the allowed set

typeMapping

A JSON object with internal lineage node types as keys and names for export as values

Mapping of internal lineage node types into names for export

Example:

"typeMapping": {
      "Server": "SERVER",
      "Database": "DATABASE",
      "Insert": "STATEMENT",
      "Update": "STATEMENT"
      ...
}

An empty object

detailedLineagePolicy

NEVER | ALWAYS | SOURCE_TO_TARGET

It is only taken into account if detailed lineage is enabled in the global settings of Manta Data Lineage to IEDC.

ALWAYS: Summary lineage will be exported between the source and the target. All flow links will be exported as detailed lineage.

• Typical use case when more than one transformation can be exported between the source and the target (e.g., ETLs)

NEVER: Detailed lineage will never be exported. All flow links will be exported as standard lineage.

• Typical use case when there is no way to specify detailed lineage in relation to summary lineage (e.g., analytical and reporting tools)

SOURCE_TO_TARGET: Summary lineage will be exported between the source and target. Flow links between the source and target will be exported as detailed lineage. Other flow links between will be exported as detailed lineage.

• Typical use case when just one transformation is exported between the source and target (e.g., database transformations)

NEVER

inputLevelSourceSupportNodeTypes

An object with the following parameters
selfSourceTypes: Set of types
parentSourceTypes: Map of types

It has the same structure as sourceSupportNodeTypes, but it has different semantics.

There’s an abstract layer of the type “input” (from IEDC terminology) where nodes of types such as procedures, triggers, scripts, and ETL transformations belong.

If this feature is enabled (see the exportInputLevelSourceCode configuration property below) and if for the given node there is no source code to be exported based on the sourceSupportNodeTypes configuration, then inputLevelSourceSupportNodeTypes is applied.

Example:

"sourceSupportNodeTypes": {
  "selfSourceTypes": [
    "PLSQL Insert",
    "PLSQL Update"
  ],
  "parentSourceTypes": {
    "PLSQL MergeInsert": "PLSQL Merge",
    "PLSQL MergeUpdate": "PLSQL Merge"
  }
},
"inputLevelSourceSupportNodeTypes": {
  "selfSourceTypes": [
    "Procedure"
  ]
}

In this case, for the PLSQL Insert and PLSQL Update nodes, PLSQL Insert and PLSQL Update, respectively, are exported. For PLSQL MergeInsert and PLSQL MergeUpdate, the whole PLSQL Merge statement is exported.

And for a node of the type Procedure, the whole procedure source code with all procedure statements is exported.

Enable input-level source code export:

exportInputLevelSourceCode: true|false in the client configuration defines whether the source code is exported at the input level (defined in inputLevelSourceSupportNodeTypes) or whether only sourceSupportNodeTypes is applied.

An object with the following:

selfSourceTypes: An empty collection

parentSourceTypes: An empty collection

nameResolverMapping

An object representing the map with the node type as the key and a name resolver configuration object with the following properties

type: Type of name resolver — NamespaceIncludingResolver or NameOfResource

If the resolver type is NamespaceIncludingResolver, then the following properties can be configured:

ancestorsTypesFilter — types of nodes allowed to be part of a namespace, if empty

There are two types of name resolvers:

NameOfResource — Per-connection export node name resolver getting its name from the resource name

NamespaceIncludingResolver — Per-connection export node name resolver composing the result name from the node name and its namespace. The namespace contains names of its ancestors that are of an allowed type.

Namespace including resolver:

We can configure it so that some nodes will not be exported. But the concept of namespaces exists so we don’t have to lose all the information from the excluded nodes.

This configuration property defines which ancestors will be included in the node namespace. There are two strategies:

FIRST_ANY_ANCESTOR — The namespace will contain the closest ancestor that meets the specified condition (e.g., the node type condition). If there is no such ancestor, no ancestor information will be added as a node namespace.

ALL_DIRECT_ANCESTOR — The namespace will contain all the closest ancestors that meet the condition (the exporter goes up in the ancestor hierarchy and adds the ancestors to the node namespace until the first node that does not meet the condition is found).

An empty mapping object

If a name resolver mapping is configured with the resolver type NamespaceIncludingResolver, then ancestorsTypesFilter is empty and namespaceBuildingStrategy is set to ALL_DIRECT_ANCESTORS by default for this mapping

allowedLocationDialects

An array of allowed dialect names (case insensitive)

The locations (transformations) in any dialect can have source or target nodes in another dialect (e.g., ETL transformations can have any database as a source or target).

For the given dialect, this configuration defines a list of dialects that nodes of this dialect can be exported for as a source or target of the transformation. The purpose of this configuration is to limit the lineage export.

If the dialect has this configuration property configured as an empty array (usually for databases), the dialect can be exported as a target or source of the locations (transformations) of all dialects.

Example:

For SSAS, allowedLocationDialects is configured to ["SSAS", "SSRS", "Excel"].

For MS SQL, allowedLocationDialects is configured as an empty array.

• If the export exports a node in SSRS dialect and its source is in SSAS dialect, this SSAS source is exported only if SSAS contains SSRS in its list of allowedLocationDialect or if the list is empty. In this case, the first condition is met, so the source would be exported.

• If the export exports a node in SSRS dialect and its target is in MS SQL dialect, this MS SQL target is exported only if MS SQL contains SSRS in its list of allowedLocationDialect or if the list is empty. The latter condition is met, so the MS SQL source would be exported.

Note: This logic is applied in both directions — to both incoming and outgoing edges.

An empty array — allowing the dialect to be exported as a source or target of the locations for all dialects

assetEntryTypeDetails

JSON object that maps Manta Data Lineage internal model nodes into Collibra types

• typeKey — Key of the type in Collibra; that is, collibra.table, collibra.file

• category — Domain of the given object according to Manta Flow Collibra Export Model; possible values are TECHNOLOGY, PHYSICAL, MAPPING, REPORTS, or LOGICAL

• parentRelations — Possible child-parent relation types; left for top-level assets

  • key — Key of the type in Collibra

  • referencedAssetRole — SOURCE or TARGET depending on the edge orientation

attributeMapping

JSON object with an array that declares how attributes from Manta Data Lineage are mapped into Collibra attribute types

• collibraKey — Key of the type in Collibra

• mantaName — Manta attribute name; it’s possible to name more of them on one line using the pipe sign, as in the example DESCRIPTION|COMMENT_STRING

• nodeType — Manta asset type

• parentType — Possible parent types; optional, left for any parent type

• valueConverter — Converts attributes into suitable values for Collibra; optional, left for no conversion

  • type — Type of the converter

  • Additional parameters if needed such as valueMap

Value Converters

• IntegerFilter — Filters non-integer values. If the Manta attribute is not an integer value, it is ignored or will be replaced by implicit value.

• ValueMapping — Maps Manta-attributed values on Collibra attribute values. It’s necessary to provide valueMap, as shown in the example.

{
  "collibraKey": "collibra.server.description",  
  "mantaName": "DESCRIPTION|COMMENT_STRING",  
  "nodeType": "HOST"
}

or

{
        "collibraKey": "collibra.column.nullability",
        "mantaName": "COLUMN_NULLABILITY",
        "nodeType": "COLUMN",
        "parentType": "TABLE|VIEW",
        "valueConverter": {
          "type": "ValueMapping",
          "valueMap": {
            "true": "True",
            "nullable": "True",
            "false": "False",
            "not null": "False"
          }
        }
  }