Using Open Manta Conceptual Overlays

Import File Formats

Metadata import supports the following types of import files.

File type	File name mask	Note
CSV	*.csv	With configurable encoding and a delimiter
Excel sheet	.xls, .xlsx	-

The first row of the input file is a header row. The column names in this header row must match those that are used in the metadata configuration (including case).

The default format of the import, as described in the metadata configuration, expects a CSV or XLS file with the following columns.

Column	Description	Mandatory / optional	Example value
Application	The name of the application where the asset is used (visible in the visualization as the object label)	Mandatory	DWH
Layer	The name of the part of the system where the asset is used (visible in the visualization as the object label)	Mandatory	Core
Asset	The name of the asset (visible in the visualization as the object label)	Mandatory	Party
Asset description	The description of the asset (visible in the visualization as the object attribute)	Mandatory	Core party
Attribute	The name of the asset’s attribute (visible in the visualization as the object label)	Mandatory	Country ID
Attribute description	The description of the attribute (visible in the visualization as the object attribute)	Mandatory	Customer's country ID
Column Resource	The physical resource of the associated column	Mandatory	MSSQL
Column	The path to the physical column	Mandatory	WIN-RRR5IO49B55/manta/dwh/party/COUNTRY_KEY
Transformation	The unique identification of the transformation that computes this attribute; Manta matches this name against all transformation objects in the physical layer that participate in the data flow for this entity	Optional	IMPORT_CRM
Calculation	The business name of the transformation (visible in the visualization as the object label)	Optional	CRM customer import
Calculation description	The description of the calculation (visible in the visualization as the object attribute)	Optional	Country Key or 0

Data Flow Generation

To generate lineage over conceptual objects, Open Manta Interpolation needs to be configured and executed. This configuration does not depend on the conceptual overlay configuration.

For more information, see Open Manta Interpolation.

Metadata Configuration

This is an advanced configuration file that you don't need to change for standard metadata import.

You can configure the structure of the imported metadata by using the metadata definition file. By default, this file is called metadataDefinition.json. The file has the following default content.

You can provide a specific metadata definition file for each conceptual overlay connection. By default, the specific definition file is expected in the input folder. For more information, see Configuration Properties. If the specific definition file is not found, the default file applies.

Example Metadata Definition File

{
  "$schema": "schema.json",
  "resources": [
    {
      "layer": {
        "name": "Business",
        "type": "Business"
      },
      "name": {
        "valueType": "FIXED",
        "value": "Business Glossary"
      },
      "type": {
        "valueType": "FIXED",
        "value": "Business Glossary"
      },
      "description": {
        "valueType": "FIXED",
        "value": "Business Glossary"
      },
      "nodes": [
        {
          "name": {
            "valueType": "FIELD",
            "fieldName": "Application"
          },
          "type": {
            "valueType": "FIXED",
            "value": "Application"
          },
          "children": [
            {
              "name": {
                "valueType": "FIELD",
                "fieldName": "Layer"
              },
              "type": {
                "valueType": "FIXED",
                "value": "Layer"
              },
              "children": [
                {
                  "name": {
                    "valueType": "FIELD",
                    "fieldName": "Asset"
                  },
                  "type": {
                    "valueType": "FIXED",
                    "value": "Asset"
                  },
                  "attributes": [
                    {
                      "name": "Description",
                      "value": {
                        "valueType": "FIELD",
                        "fieldName": "Asset description"
                      }
                    }
                  ],
                  "children": [
                    {
                      "name": {
                        "valueType": "FIELD",
                        "fieldName": "Attribute"
                      },
                      "type": {
                        "valueType": "FIXED",
                        "value": "Attribute"
                      },
                      "attributes": [
                        {
                          "name": "Description",
                          "value": {
                            "valueType": "FIELD",
                            "fieldName": "Attribute description"
                          }
                        },
                        {
                          "name": "mapsTo",
                          "value": {
                            "valueType": "QUALIFIED_NAME",
                            "resourceColumn": "Column Resource",
                            "nodePathColumn": "Column"
                          }
                        },
                        {
                          "name": "NODE_TRANSFORMATION",
                          "value": {
                            "valueType": "LIST",
                            "items": [
                              {
                                "valueType": "RESOURCE",
                                "name": {
                                  "valueType": "FIXED",
                                  "value": "Business Transformation"
                                },
                                "type": {
                                  "valueType": "FIXED",
                                  "value": "Business Transformation"
                                },
                                "description": {
                                  "valueType": "FIXED",
                                  "value": "Business Transformation"
                                }
                              },
                              {
                                "valueType": "PATH",
                                "fieldName": "Transformation"
                              },
                              {
                                "valueType": "FIELD",
                                "fieldName": "Calculation"
                              },
                              {
                                "valueType": "MAP",
                                "items": [
                                  "Calculation Description"
                                ]
                              }
                            ]
                          }
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

This file describes how imported metadata files are transformed into metadata objects. This transformation is made sequentially for each line that is imported. The input of each line transformation is the line data and the metadata definition file content, while the output is the appropriate metadata objects. The result of the whole process is a merger of all line transformation metadata objects.

The default content of the file can be modified if you abide by the the rules for metadata definition file content.

Rules for Metadata Definition File Content

Because the Metadata definition file is a JSON format, the easiest way to describe its content rules is to call it a combination of a pseudo-JSON language and regular grammar.

The following list describes the expressions that are used.

/* (A) */ - Occurrence of (A) just once, where (A) is defined sooner or later.
/* (A)? */ - Optional occurrence of (A).
/* (A)* */ - Optional occurrence of (A) any number of times.
/* (A1)|(A2)|...|(An) */ - Occurrence of (A₁), (A₂), ..., or (A_n).

Content of the Metadata Definition File

On the highest level, the meta metadata consists of a list of resource definitions.

Metadata Definition File

{
  "resources": [
    /* (resource definition)* */
  ]
}

Resource Definitions

Resource definitions describe the way a line read from the imported fileis transformed into a resource. Resource definition properties include the following items:

Node definition property item	Description
`layer`	The layer the resource belongs to. The layer has these attributes: - `name` — Name of the layer - `type` — Type of the layer
`name`	A way to get the resource name from the imported line.
`type`	A way to get the resource type from the imported line.
`description`	A way to get the resource description from the imported line.
`nodes`	A way to get the resource nodes from the imported line.

(resource definition)

   {
     "layer": {
       "name": "/* layer name : non-blank string */",
       "type": "/* layer name : non-blank string */"
     },
     "name": /* (value definition) */,
     "type": /* (value definition) */,
     "description": /* (value definition) */,
     "nodes": [
       /* (node definition)* */
     ]
   }

Node Definitions

Node definitions describe the way a line read from the imported file is transformed into a node. Node definition properties include the following items:

Node definition property item	Description
`name`	A way to get the node name from the imported line.
`hierarchy`	Can be used instead of the `name`. A way to get a hierarchy of nodes with the same `type` from the imported line.
`type`	A way to get the node type from the imported line.
`children` (optional)	A way to get the node children from the imported line. If not set, it means that the node has no children.
`attributes` (optional)	A way to get the node attributes from the imported line. If not set, it means that the node has no attributes.

(node definition)

   {
     "name": /* (value definition) */,
     "type": /* (value definition) */,
     /* (child nodes)? */,
     /* (attributes)? */
   }

(child nodes)

   "children" : [
     /* (node definition)* */
   ]

(attributes)

   "attributes" : [
     /* (attribute definition)* */
   ]

Attribute Definitions

Attribute definitions describe the way a line read from the imported file is transformed into an attribute. Attribute definition constructor arguments include (in this order):

Node definition property item	Description
`name`	The attribute name.
`value`	A way to get the attribute value from the imported line.

(attribute definition)

   {
     "name": "/* attribute name : non-blank string */",
     "value": /* (value definition) */
   }

Value Definitions

Value definitions describe the way a line read from the imported file is transformed into a value. Types of value definitions include:

Value definition type	Description
`fixed value definition`	The value is fixed, independent of the line read.
`field value definition`	The value stored in the given field of the line read.
`qualified name value definition`	The value is an element-qualified name, stored in the given fields of the line read; the constructor arguments are (in this order): - `resource field` — A field containing the element resource name. - `type field` — Optional. A field containing the element type name. If not set, it means that the element might be of any type. - `path field` — A field containing a path to the element. - `separator` — Optional. A character used as a separator in the path field. If not set, the slash character (`/`) is used by default.

(value definition)

   /* (fixed value definition) | (field value definition) | (qualified name value definition) */

(fixed value definition)

   {
     "valueType": "FIXED",
     "value": "/* value : string */"
   }

(field value definition)

    {
      "valueType": "FIELD",
      "fieldName": "/* name of field containing value : non-empty string */"
    }

(qualified name value definition)

   {
     "valueType": "QUALIFIED_NAME",
     "resourceColumn": "/* name of field containing resource name : non-empty string */",
     /* (type field)? */,
     "nodePathColumn": "/* name of field containing path to element : non-empty string */",
     /* (path separator)? */
   }

(type field)

   "typeColumn": /* name of field containing element type : non-empty string */

(path separator)

   "delimiter": /* element path separator : char */

Conceptual Overlays Connection

To perform conceptual overlays, you need a connection configuration. For more information, see Open Manta Interpolation Configuration.

When you have a connection configuration, run the Conceptual Overlays / Import Metadata Scenario. You might need to introduce a minor revision.