Configuring Resources

If it is necessary to adjust how elements from existing resources are presented or to introduce a new custom resource, there are several configuration files to edit.

Level of Detail

To change the levels of detail on which elements of selected types that are contained in resources of selected types are displayed, edit the Manta Admin UI page Configurations > Server > Visualization > Levels that has the following XML-based structure.

<levels>
    <technology name="Resource Type">
        <map type="Element Type Top" level="top"/>
        <map type="Element Type Middle" level="middle"/>
        <map type="Element Type Bottom" level="bottom" atomic="true"/>
    </technology>
</levels>

Resources of the selected types that are specified thorugh the technology element with a name attribute are nested in the levels element. Note the name attribute refers to a type of resource (for example, Oracle scripts) rather than its name (for example, Oracle PL/SQL).

Mappings between element types (for example, database, table, column) and their levels (top, middle, and bottom) specified through a map element with type and level attributes are nested in the technology element. It is necessary to follow this rule: if an element of the first type should contain an element of the second type, the second type must not be mapped to a higher level (top > middle > bottom) than the first type. For example, a column must not have a higher level than a table.

The mapped levels are used to decide which elements are displayed to users in the data flow graph visualization for the selected level of detail in the visualization parameters.

Note that at least the highest element in the hierarchy is always shown regardless of its mapped level and selected level of detail.

There is an extra optional attribute for the map element that is only used for the lowest elements. If it is true that elements of the selected type usually do not contain other elements (for example, column), it is possible to add the atomic="true" attribute. Then, the visualization of these elements is more compact than that of the others. If there is an exceptional element containing other elements but with this attribute set, it is visualized like the other ones.

Data flow Look

The configuration-element-type look is split into two groups.

Behavior Settings

The configuration is stored in the JSON structure on the Manta Admin UI page Configurations > Server > Visualization > Common Configuration, and it is based on the resource type and element type. For each pair, users can configure the following properties.

You can also define default configurations to use for undefined resource types. For this feature, use the keyword Default.

Example of a configuration:

{
   "Default":{
      "Directory":{
         "iconPositive":"/img/icons/folder_icon_v2_positive.png",
         "iconNegative":"/img/icons/folder_icon_v2_negative.png",
         "colorSchemaName":"group"
      },
      "File":{
         "iconPositive":"/img/icons/file_icon_positive.png",
         "iconNegative":"/img/icons/file_icon_negative.png",
         "colorSchemaName":"object"
      }
   },
   "Resource Type":{
      "Element Type":{
         "iconPositive":"/img/icons/sample_icon_positive.png",
         "iconNegative":"/img/icons/sample_icon_negative.png",
         "colorSchemaName":"group|object|column",
         "attributes":{
            "ATTRIBUTE_1":"Attribute 1",
            "ATTRIBUTE_2":"Attribute 2",
            "ATTRIBUTE_3":"Attribute 3"
         }
      }
   }
}

Color Settings

The color settings are divided into layers. The layers have names and priorities to specify the order in which they are resolved. Then, users can dynamically activate or deactivate them in the visualization while exploring the lineage. The layer definitions are available on the Manta Admin UI page Configurations > Server > Skinning > Layers Configuration.

The base structure of the JSON-based configuration is:

[
    {
      ...layer defintion...
    },
    ...,
    {
      ...layer defintion...
    }
]

Identification Possibilities

There are more ways of identifying configurations than what are in the behavior settings.

Scope

Type attribute

Specification

Use case

To each element

"type":"any"

The default behavior for each element

By resource type

"type":"resource"

"value":"<resource_type>"

Standard coloring by resource type, which is used by default

By element type

"type":"type"

"value":"<element_type>"

Special configuration for interesting elements; for example, the distinction between tables and views

By attribute value

"type":"attribute"

"value":"<attribute_key>"

"attrValue":"<attribute_value>"

Detailed granulation, for example, by column data type; the property attrValue can be omitted, in which case, a color layer is applied if the attribute has any value

Resolving Algorithm

The algorithm to find which definition should be used is:

  1. List the layer definitions in order by priority.

  2. For each layer starting from the highest priority:

    1. Try to find the By Attribute Value definition that is fulfilled .

    2. Try to find the By Element Type definition that is fulfilled.

    3. Try to find the By Resource Type definition that is fulfilled.

    4. Apply the To Each Element definition if it exists.

    5. Exit the algorithm if any configuration is found.

Configuration Description

The configuration of a layer contains:

An example of a layer configuration without the colorSchemas section:

File structure Explanation
{
"name": "By resource", The name shown in the visualization
"checked": true, This look definition is used by default
"priority": 0, Lowest priority (used with multiple color definitions)
"definitions": [ The list of definitions
{
"identification": {
"type": "resource", Identification by resource type
"value": "Teradata" Resource type that this definition applies to
},
"colorSchema": {
"group": { Color schema definition for the schema named “group”
"bgcolor" : "#dddfe1", Background color; format #RRGGBB (red, green, blue in HEX)
"color" : "#393b3d", Text color
"borderColor" : "#9fa0a1", Border color
"positive" : true True if the color schema is positive (dark text on a light background)
},
"object": { Color schema definition for the schema named “object”
"bgcolor" : "#0b9211",
"color" : "#ffffff",
"borderWidth" : 0, Border width in pixels
"positive" : false
},
"column": { Color schema definition for the schema named “column”
"bgcolor" : "#ffffff",
"color" : "#616365",
"borderWidth" : 0,
"positive" : true
}
}
}
]
}

An example of a layer configuration with the colorSchemas section:

File structure Explanation
{
"name": "By category", The name shown in the visualization
"checked": false, This look definition is not used by default
"priority": 1, Priority (used with multiple color definitions)
"withResource": true, Resource nodes should be visible
"colorSchemas": { List of color schemas divided into categories
"storage_asset": { "Storage assets" category
"group": { Color schema definition for the schema named “group”
"bgcolor": "#d4a4dd", Background color; format #RRGGBB (red, green, blue in HEX)
"color": "#393b3d", Text color
"borderColor": "#3f0e40", Border color
"positive": true True if the color schema is positive (dark text on a light background)
},
"object": { Color schema definition for the schema named “object”
"bgcolor": "#751c85",
"color": "#ffffff",
"borderWidth": 0,
"positive": false
},
"column": { Color schema definition for the schema named “column”
"bgcolor": "#ffffff",
"color": "#616365",
"borderWidth": 0, Border width in pixels
"positive": true
},
}
},
{
"definitions": [ The list of definitions
"identification": {
"type": "resource", Identification by resource type
"value": "Teradata" Resource type this definition applies to
},
"colorSchemaRef": "storage_asset" Reference to the "storage_asset" color schema
}
]
}

Vertical Filters

Vertical filters specify which element types in element hierarchies are not (partly or fully) shown in the data flow graph visualization. For example, it is not necessary to visualize all directories on a path to a script file. It is enough to visualize only the lowest one (immediately before the file).
To change the vertical filters, edit the Manta Admin UI page Configurations > Server > Visualization > Vertical Filters that has the following XML-based structure.

<filters>
  <technology name="Resource Type">
    <element name="Element Type Top" mode="withoutTop"/>
    <element name="Element Type Bottom" mode="withoutBottom"/>
    <element name="Element Type None"/>
  </technology>
</filters>

Resources of the selected types specified through the technology element with a name attribute are nested in the filters element. The name attribute refers to a type of resource (for example, Oracle scripts) rather than its name (for example, Oracle PL/SQL).

A list of element types to filter, which is specified through the element called element with name and optional mode attributes, is nested in the technology element. Note that the name attribute refers to the element type. Choose from three modes of filtering.

Horizontal Filters

Manta Flow has such types of filters.

Both of them can be grouped into the horizontal filter groups that are shown on the catalog page. The horizontal filter group configuration is available on the Manta Admin UI page Configurations > Server > Visualization > Horizontal Filter Groups with the following JSON-based structure.

horizontalFilterGroups.json
[
    {
        "id": 1,
        "order": 0,
        "name": "Everything",
        "filterIds": []
    },
    {
        "id": 2,
        "order": 1,
        "name": "Databases & files",
        "filterIds": ["IFPC", "MSSQL DDL", "MSSQL TSQL", "Oracle DDL", "Oracle PLSQL", "Teradata BTEQ", "Teradata DDL", "Teradata TPT"]
    }
]

The configuration is an array of groups. Every group has an:

Horizontal filters are defined on the Manta Admin UI page Configurations > Server > Visualization > Horizontal Filters with the following JSON-based structure.

[
  {
    "id": "HideInnerTrans",
    "name": "Hide inner transformations",
    "type": "CUSTOM",
    "order": 1,
    "layer": "Physical",
    "logicalSolver": [
      [
        {
          "truthValue": "true",
          "predicate": {
            "type": "hasParent",
            "predicateUnits": [
              {
                "iterateOverAll": "false",
                "logicalSolver": [
                  [
                    {
                      "truthValue": "true",
                      "unit": {
                        "type": "type",
                        "impactedResources": [],
                        "disabledTypes": [
                          "IFPC Sorter",
                          "IFPC Rank",
                          "IFPC Router"
                        ]
                      }
                    }
                  ],
                  [
                    {
                      "truthValue": "true",
                      "unit": {
                        "type": "attribute",
                        "impactedResources": [],
                        "impactedTypes": [],
                        "key": "TRANSFORMATION_TAG",
                        "disabledValues": [
                          "NON_TRANSFORMING"
                        ]
                      }
                    }
                  ]
                ]
              }
            ]
          }
        }
      ]
    ]
  },
  {
      "id": "pathFilter_Teradata",
      "name": "Path filter, only display /Teradata/db/table1 and /Teradata/db/table2/t2c2",
      "type": "CUSTOM",
      "order": 7,
      "logicalSolver": [
        [
          {
            "truthValue": "false",
            "unit": {
              "type": "pathPrefix",
              "pathPrefix": "/Teradata/db/table1"
            }
          },
          {
            "truthValue": "false",
            "unit": {
              "type": "pathPrefix",
              "pathPrefix": "/Teradata/db/table2/t2c2"
            }
          }
        ]
      ]
    }
]

The configuration is an array of filters. Every filter has an:

Logical Solver

JSON schema of the solver:

[
  {
  "$title": "JSON schema for filter in Manta",
  "type": "object",
  "required": [
    "id",
    "name",
    "type",
    "order",
    "logicalSolver"
  ],
  "properties": {
    "id": {
      "description": "identification used in the rest of the application including the horizontal filter group",
      "type": "string"
    },
    "name": {
      "description": "name shown on the dataflow page",
      "type": "string"
    },
    "type": {
      "description": "for user defined filters it should always be CUSTOM",
      "type": "string",
      "const": "CUSTOM"
    },
    "order": {
      "description": "number by which filters are sorted",
      "type": "integer"
    },
    "layer": {
      "description": "name of the layer the filter is valid for; if not set, the filter is valid for all layers",
      "type": "string"
    },
    "logicalSolver": {
      "$ref": "#/definitions/logicalSolver"
    }
  },
  "definitions": {
    "logicalSolver": {
      "description": "array of minterms, if one of them returns true,the node is filtered",
      "type": "array",
      "items": {
        "$ref": "#/definitions/minterm"
      }
    },
    "minterm": {
      "description": "array of literals, if all of them returns true,the node is filtered",
      "type": "array",
      "items": {
        "$ref": "#/definitions/literal"
      }
    },
    "literal": {
      "description": "literal represents a logical part of the filter, if predicate/unit returns the same value as the truth value, the node is filtered",
      "type": "object",
      "required": [
        "truthValue"
      ],
      "properties": {
        "truthValue": {
          "description": "expected return value of the predicate/unit when node should be filtered",
          "type": "string",
          "enum": [
            "true",
            "false"
          ]
        },
        "predicate": {
          "$ref": "#/definitions/predicate"
        },
        "unit": {
          "$ref": "#/definitions/filterUnit"
        }
      },
      "oneOf": [
        {
          "required": [
            "predicate"
          ]
        },
        {
          "required": [
            "unit"
          ]
        }
      ]
    },
    "predicate": {
      "description": "predicate solves higher logic of filter accross multiple vertices",
      "type": "object",
      "required": [
        "type",
        "predicateUnits"
      ],
      "properties": {
        "type": {
          "description": "type of the predicate",
          "type": "string",
          "enum": [
            "hasParent"
          ]
        }
      },
      "if": {
        "properties": {
          "type": {
            "const": "hasParent"
          }
        }
      },
      "then": {
        "properties": {
          "predicateUnits": {
            "type": "array",
            "items": {
              "$ref": "#/definitions/predicateUnits/hasParent"
            }
          }
        }
      }
    },
    "predicateUnits": {
      "hasParent": {
        "description": "hasParent predicate unit iterates over nodes parents. If one of the nodes satisfies the filter,its parent is sent to next hasParent predicate unit. If all predicateUnits are satisfied, the node is filtered",
        "type": "object",
        "required": [
          "iterateOverAll",
          "logicalSolver"
        ],
        "properties": {
          "iterateOverAll": {
            "description": "if set to false, iterates over parents of the node. When node that satisifies inner logic is found, pass its parent to next hasParent predicate unit. If set to true, iterates over all parents of the node. If inner logic for node and all parents is satisfied, pass the node to next hasParent predicate unit.",
            "type": "string",
            "enum": [
              "true",
              "false"
            ]
          },
          "logicalSolver": {
            "$ref": "#/definitions/logicalSolver"
          }
        }
      }
    },
    "filterUnit": {
      "description": "units solve logic of filter on single nodes",
      "type": "object",
      "required": [
        "type"
      ],
      "properties": {
        "type": {
          "description": "type of the filter unit to be used",
          "type": "string",
          "enum": [
            "attribute",
            "type",
            "resource",
            "name"
          ]
        }
      }
    }
  }
}
]

The logical solver contains complex filter logic consisting of minterms and literals.

A minterm is represented as an array in the logical solver. If the logic in at least one of the minterms is fulfilled, the vertex is filtered.

One minterm contains one or more literals. If the logic in all the literals is fulfilled, the minterm is fulfilled.

Each literal contains a truth value and a predicate or filter unit. The truth value is the expected value of the predicate for the node to be filtered. It is used to negate the statements.

For example, “filter all the columns that are NOT in table1“.

Predicates solve filter logic over multiple vertices.

Logical Solver Examples

Filter all nodes that have the name “Example” AND type “Column”
"logicalSolver": [
  [
    {
      "truthValue": "true",
      "predicate": {
        "type": "hasParent",
        "predicateUnits": [
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example"
                  }
                },
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "type",
                    "disabledTypes": ["Column"]
                  }
                }
              ]
            ]
          }
        ]
      }
    }
  ]
]
Filter all nodes that have the name “Example“ and do not have the type “Column”
"logicalSolver": [
  [
    {
      "truthValue": "true",
      "predicate": {
        "type": "hasParent",
        "predicateUnits": [
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example"
                  }
                },
                {
                  "truthValue": "false",
                  "unit": {
                    "type": "type",
                    "disabledTypes": ["Column"]
                  }
                }
              ]
            ]
          }
        ]
      }
    }
  ]
]
Filter all nodes that have the name “Example“ or the type “Table”
"logicalSolver": [
  [
    {
      "truthValue": "true",
      "predicate": {
        "type": "hasParent",
        "predicateUnits": [
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example"
                  }
                }
              ],
              [
                {
                  "truthValue": "false",
                  "unit": {
                    "type": "type",
                    "disabledTypes": ["Column"]
                  }
                }
              ]
            ]
          }
        ]
      }
    }
  ]
]
Filter everything but columns
"logicalSolver": [
  [
    {
      "truthValue": "true",
      "predicate": {
        "type": "hasParent",
        "predicateUnits": [
          {
            "iterateOverAll": "true",
            "logicalSolver": [
              [
                {
                  "truthValue": "false",
                  "unit": {
                    "type": "type",
                    "disabledTypes": ["Column"]
                  }
                }
              ]
            ]
          }
        ]
      }
    }
  ]
]
Filter all nodes that have the name “Example“ and the type “View” OR “Table”
"logicalSolver": [
  [
    {
      "truthValue": "true",
      "predicate": {
        "type": "hasParent",
        "predicateUnits": [
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example"
                  }
                },
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "type",
                    "disabledTypes": ["Table"]
                  }
                }
              ],
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example"
                  }
                },
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "type",
                    "disabledTypes": ["View"]
                  }
                }
              ]
            ]
          }
        ]
      }
    }
  ]
]
Filter all nodes that have the name “Example“ and the type “Column” and one of their ancestors has the type “Table”
"logicalSolver": [
  [
    {
      "truthValue": "true",
      "predicate": {
        "type": "hasParent",
        "predicateUnits": [
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example"
                  }
                },
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "type",
                    "disabledTypes": ["Column"]
                  }
                }
              ]
            ]
          },
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "type",
                    "disabledTypes": ["Table"]
                  }
                }
              ]
            ]
          }
        ]
      }
    }
  ]
]
Filter all nodes that have the name “Example“ AND one of their ancestors has the name “Example2“ AND one of theirs has the name “Example3“
"logicalSolver": [
  [
    {
      "truthValue": "true",
      "predicate": {
        "type": "hasParent",
        "predicateUnits": [
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example"
                  }
                }
              ]
            ]
          },
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example2"
                  }
                }
              ]
            ]
          },
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example3"
                  }
                }
              ]
            ]
          }
        ]
      }
    }
  ]
]
Filter all nodes that have the name “Example” and NONE of their ancestors have the type “Table”
"logicalSolver": [
  [
    {
      "truthValue": "true",
      "predicate": {
        "type": "hasParent",
        "predicateUnits": [
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example"
                  }
                }
              ]
            ]
          },
          {
            "iterateOverAll": "true",
            "logicalSolver": [
              [
                {
                  "truthValue": "false",
                  "unit": {
                    "type": "type",
                    "disabledTypes": ["Table"]
                  }
                }
              ]
            ]
          }
        ]
      }
    }
  ]
]
Filter all nodes that have the name “Example” and NONE of their ancestors have the type “Table” OR that have the name “Example2” and NONE of their ancestors have the type “View”
"logicalSolver": [
  [
    {
      "truthValue": "true",
      "predicate": {
        "type": "hasParent",
        "predicateUnits": [
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example"
                  }
                }
              ]
            ]
          },
          {
            "iterateOverAll": "true",
            "logicalSolver": [
              [
                {
                  "truthValue": "false",
                  "unit": {
                    "type": "type",
                    "disabledTypes": ["Table"]
                  }
                }
              ]
            ]
          }
        ]
      }
    }
  ],
  [
    {
      "truthValue": "true",
      "predicate": {
        "type": "hasParent",
        "predicateUnits": [
          {
            "iterateOverAll": "false",
            "logicalSolver": [
              [
                {
                  "truthValue": "true",
                  "unit": {
                    "type": "name",
                    "regexp": "Example2"
                  }
                }
              ]
            ]
          },
          {
            "iterateOverAll": "true",
            "logicalSolver": [
              [
                {
                  "truthValue": "false",
                  "unit": {
                    "type": "type",
                    "disabledTypes": ["View"]
                  }
                }
              ]
            ]
          }
        ]
      }
    }
  ]
]
Filter nodes that are from other perspectives
"logicalSolver": [
      [
        {
          "truthValue": "true",
          "predicate": {
            "type": "hasParent",
            "predicateUnits": [
              {
                "iterateOverAll": "false",
                "logicalSolver": [
                  [
                    {
                      "truthValue": "true",
                      "unit": {
                        "type": "layer"
                      }
                    }
                  ]
                ]
              }
            ]
          }
        }
      ]
    ]
Filter all nodes beyond /Teradata/db/table1 and /Teradata/db/table2/t2c2 path
{
    "id": "pathFilter_Teradata",
    "name": "Path filter, only display /Teradata/db/table1 and /Teradata/db/table2/t2c2",
    "type": "CUSTOM",
    "order": 7,
    "logicalSolver": [
      [
        {
          "truthValue": "false",
          "unit": {
            "type": "pathPrefix",
            "pathPrefix": "/Teradata/db/table1"
          }
        },
        {
          "truthValue": "false",
          "unit": {
            "type": "pathPrefix",
            "pathPrefix": "/Teradata/db/table2/t2c2"
          }
        }
      ]
    ]
  }

Horizontal filter for Lineage Listing

Horizontal filters can be used to limit the results to certain types or paths. This is especially relevant for source/target analysis, where we can use horizontal filters to specify a certain path, where we usually search for sources or targets (like Oracle/DWH/L2 for sources or reports, or PowerBI/Server/Workspace/Report to specify targets of some DWH tables).

Horizontal filters can be configured in the Admin UI. Configuration->Server->Visualization->Horizontal filters. Then, edit the JSON configuration to use path prefixes.

A picture that sows the confiugration of the horizontal filter group

Example of a horizontal filter that uses path prefix:

{
  "id": "pathFilter1",
  "name": "Path filter PathFilterTest, only display /Teradata/db/table1",
  "type": "CUSTOM",
  "order": 7,
  "logicalSolver": [
    [
      {
        "truthValue": "false",
        "unit": {
          "type": "pathPrefix",
          "pathPrefix": "/Teradata/db/table1"
        }
      }
    ]
  ]
}

Horizontal Filter group can be configured as on the following page in the Admin UI. Configuration->Server->Visualization->Horizontal filter Groups. To add new horizontal filter to group, it’s necessary to use filter ID. In our case ID = pathFilter1

A picture that sows the confiugration of the horizontal filter group

The Manta Flow server must be notified of configuration changes by calling <MANTA_SERVER_URL>/api/refresh. A second option is to restart the server.