Open Manta Designer - Administration Manual for R42.2

This manual for OMD administration describes several topics related to the operation and setup of Open Manta Designer, a visual modeling tool for designing custom data lineages.

In case of questions or problems, contact IBM Manta Data Lineage support.

Installation folder structure

In Manta INSTALLATION_ROOT, you can find the following folders and files:

OMD start & stop scripts

The OMD startup and shutdown scripts can be found in $OMD_installationfolder/bin subfolder.

Similarly to other Manta Data Lineage services, once the Open Manta Designer is enabled, its status can be seen directly in Manta Launcher from where it can be accessed as well.

No alt text provided

OMD users and roles in Keycloak

oidc.properties in the OMD_installationfolder/manta-openmantadesigner-dir/conf/subfolder contains the URI of the Keycloak for authentication and client Secret that can be obtained in Keycloak administration.

By default, a local Keycloak from Manta Data Lineage installation is being used. However, you can use a remote Keycloak as well. If so, make sure to set up proper user roles for OMD.
# Keycloak address
manta.oidc.issuerUri=http://localhost:9090/auth/realms/manta

# OAuth's client ID
manta.oidc.clientId=omd-client

# Special for FE - Used in FE authentication
manta.oidc.publicClientId=omd-fe-client

# Secret from Keycloak
manta.oidc.clientSecret= PASTE YOUR clientSecret HERE

# All the necessary information we need
manta.oidc.scope=roles,profile,openid

# Web token's Json path, where are the roles stored
manta.oidc.roleClaim=realm_access.roles

If you want to add a new OMD user, create a new user account in Keycloak and assign proper roles.

OMD uses several user roles for its operation, which are automatically created in Keycloak when Manta is being installed. Anyone who wants to work with OMD must have at least one of the following roles:

application.properties file

In the application.properties file, you can define some properties influencing connection timeouts between OMD and the Manta Flow server and size of Open Manta Extension CSV files being imported into OMD.

manta.openmantadesigner.flowserver.service-request.read-timeout-seconds=120
manta.openmantadesigner.flowserver.service-request.connect-timeout-seconds=30
spring.servlet.multipart.max-file-size=10MB
spring.servlet.multipart.max-request-size=10MB

artemis.properties file

In this file, all the relevant properties which are used for logging messages from OMD into Artemis are defined. Mostly these logs cover warnings and error which might occur during import of Nodes from Manta Flow Viewer (or CSV/JSON), especially if relevant Resource Templates are not found for Resources being imported or in case some Node Type definitions are missing in the Resource Templates.

The messages being sent from OMD into Artemis can be then accessed in Manta Admin UI / Log Viewer.

manta.artemis.keystore-password=PASTE YOUR keystore password HERE
manta.artemis.keystore-path= #full path to .../manta-openmantadesigner-dir/conf/artemiskeystore
manta.artemis.mtls-enabled=true
manta.artemis.truststore-password=PASTE YOUR truststore password HERE
manta.artemis.truststore-path= #full path to .../manta-openmantadesigner-dir/conf/artemistruststore
manta.artemis.port=61616 #artemis port number

Configuration setting YAML

OMD uses several configuration settings that influence its behaviour. These are stored in settings.yml which is bundled directly in OMD java archive (.jar). However, these settings can be overwritten by creating your own custom settings.yml and storing it in the $OMD_installationfolder/manta-openmantadesigner-dir/conf/ subfolder.

If you create your own settings.yml, it is necessary to restart OMD in order to apply the changes. In case you encounter some unexpected errors immediately after your custom settings.yml was applied, just remove it from the manta-openmantadesigner-dir/conf/ folder, restart OMD and it will again use the default one bundled in java archive.

The setting.yml file contains the following sections:

settings.yml

connections:
  icon: repository
  name:
    pattern: '^[^*&!@#$)(%^_~+?|":}{\s](.*\S)?$'
    min-length: 1
    max-length: 128

layers:
  types:
    -
      name: Physical
      default-name: Physical
      icon: layer

    -
      name: Logical
      default-name: Logical
      icon: expression

    -
      name: Conceptual
      default-name: Conceptual
      icon: balance_data_distribution

    -
      name: Perspective
      default-name: Perspective
      icon: view

    -
      name: Application Level
      default-name: Application Level
      icon: view

  name:
    pattern: '^[^*&!@#$)(%^_~+?|":}{\s](.*\S)?$'
    min-length: 1
    max-length: 128

resources:
  defaultType: Oracle
  name:
    pattern: '^[^*&!@#$)(%^_~+?|":}{\s](.*\S)?$'
    min-length: 1
    max-length: 128
  icon: server

nodes:
  name:
    pattern: '^[^*&!@#$)(%^_~+?|":}{\s](.*\S)?$'
    min-length: 1
    max-length: 128
  icon: asset

canvases:
  name:
    pattern: '^.*$'
    min-length: 1
    max-length: 128

csv:
  export:
    folder: 'input/import/'

How to manage Resource Templates

Resource Template (RT) is a specific metadata configuration for given technology supported by OMD. By introducing new Resource Template, OMD can easily support new technologies without need to modify OMD code. It has a form of YAML file, which can be easily prepared and verified according to defined YAML schema.

With OMD/Manta release, a default set of Resource Templates is bundled along with the application in .jar file. However, users can extend this basic set of RTs or even overwrite default Resource Templates for particular technology.

There are two ways how to load new Resource Template into OMD, either by using OMD API or via storing YAML file in a particular subfolder.

Using Resource Templates API endpoints

This has an advantage of YAML validation against defined YAML schema. In case the YAML structure is not valid, the Resource Template is not uploaded into OMD and thus ensure the application remains stable.

Available endpoints:

Storing new Resource Templates in resource_templates subfolder

This is another way how to add custom Resource Templates.

The subfolder can be found in {MANTA_installation_folder}/openmantadesigner/manta-openmantadesigner-dir/conf/resource_templates/.

For each Resource Template, there can be .yml file defined. It's name must be the same as Resource Type and it can even overload default Resource Templates. For example, there can be Oracle.yml, which overloads default Oracle Resource Template bundled with OMD).

If you want to load your custom Resource Template into OMD, just copy the .YML file into resource templates subfolder and reload OMD by simply reloading the page in browser. From now on the new Resource Type is available in OMD.

Although being straightforward, this approach has one drawback - the custom YAML files are not validated.

In case OMD cannot start after Resource Template has been copied into this subfolder, it is probable the YAML file is not valid. Simply remove the YAML from the subfolder and restart OMD again either via Manta Launcher or startup / shutdow scripts.

Validation of Node hierarchy imported from Manta Flow Server or CSV

Each time some Nodes are being imported into OMD either from Manta Flow Server or via importing whole Connection from Open Manta Extensions (CSV files), the imported data are being validated against the available Resource Template. If there are found some discrepancies between imported data and RT, these are reported to the user both via OMD front-end (in Import dialog) and into Admin UI/Log Viewer.

Following situations can occur:

Resource Template structure in R42.2

A Resource Template has following structure.

resource level

nodes level

Referencing other Resource Templates

-> notation for referencing Resource Types can be used in case some Resource contains sub-hierarchy of Nodes from another Resource. For example, "Oracle scripts" sub-hierarchy can be found within Oracle Resource (and similarly in all other DBMS technologies and Scripts).

In the example, within the Schema node, a CreateViewScript and its sub-hierarchy from Oracle scripts can be created:

- type: Schema
    icon: database_schema
    name: a_schema
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 0
    supportedStartEdges: [PERSPECTIVE, THIRDLEVEL_MACROEDGE]
    supportedEndEdges: [THIRDLEVEL_MACROEDGE]
    children:
      - Table
      - View
      - Procedure
      - Function
      - Synonym
      - PLSQL Synonym
      - PLSQL Sequence
      - PLSQL Package
      - Oracle scripts->CreateViewScript

Referencing Nodes groups

In case there are more Nodes and their sub-hierarchies from various Resources available, these can be references using @ notation. All Node Types which belong to the specific Node group can be a Child nodes of the Node Type which is referencing this group.

For example PLSQL Block is a Node Type from Oracle. It belongs to DBMSScripts node group along with some other nodes from e.g. MSSQL, DB2, PostgreSQL, and so on.

type: PLSQL Block
  icon: plsql_block
  name: a_plsqlBlock
  groups:
  - '@DBMSScripts'
  childNodeImportabilityLimit: 999
  canBeDroppedToCanvas: false
  children:
  - PLSQL Block
  - PLSQL Merge
  - PLSQL Return
  - PLSQL VarAssignment
  - PLSQL Insert

Then, this Node group is referenced from within SSIS Resource Template, SqlTask node in particular. It means that under SqlTask node, whatever Node from DBMScripts group can be created:

- type: SqlTask
  icon: block
  name: a_sqlTask
  childNodeImportabilityLimit: 999
  canBeDroppedToCanvas: false
  children:
  - '@DBMSScripts'
The mechanism of node groups is sometimes used even within one Resource Template, especially if there are repetitions and recursion in Node Hierarchy. This helps to keep Resource Template definition shorter and more manageable.

Defining default Node attributes and their default values

For each Node Type in RT, it can be defined if there are some default Node Attributes which are created along with a Node of such Node Type.

The following is an example of three Node Attributes (DATA_TYPE, DESCRIPTION, and DATA_QUALITY) that are created along with each Column Node.

For each node attribute, the following fields can be defined:

- type: Column
  icon: column
  name: a_column
  childNodeImportabilityLimit: 999
  canBeDroppedToCanvas: false
  childNodeVisibilityLimit: 50
  supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
  supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
  children:
    - StructureItem
    - PLSQL Variable
  attrs:                          # List of default node attributes
    - name: DATA_TYPE              # Attribute name
      options:                    # Attribute type (optional) is enum
      - INT                       # The first is the default type
      - VARCHAR
      - DOUBLE
      strictOptions: false       # The listed options are only recommended
                                 # User can define any type he likes
    - name: DESCRIPTION          # Attribute name
      type: string               # Attribute type
      defaultValue: 'this is description' # Attribute default value
    - name: QUALITY              # Attribute name
      type: float                # Attribute type
      defaultValue: 0            # Attribute default value

Macroedges

New edge type was introduced SECONDLEVEL_MACROEDGE. When such edge is created between 2nd level nodes (Tables, Views, and so on), the Edge Wizard dialog is automatically opened in OMD where user can easily create many individual edges just by providing input parameters for matching algorithm.

Fallback nodes

In some cases, the Node Hierarchy defined in RT may contain fallback nodes, which are used only in case extraction and analysis if a given Resource fails. Although such nodes can be imported into OMD, it should not be possible to created them manually in OMD. Such Node Types can have the fallbackNode: true attribute and OMD does not allow for them to be created manually via UI. You can see IFPC Resource Template as an example.

A sample of R42 Resource Template for Oracle

The only major change in OMD R42.2. relates to Application Level lineages support. For all high level nodes in all Resource Templates (for Physical Resources), two edge types Application Direct and Application Filter are defined, see sample Oracle RT below. Similarly, all Physical Resources can be created in Application Level layer as well.
resource:
  type: Oracle
  name: Oracle
  icon: database_source
  layers:
    - Physical
    - Application Level   #this Resource can be created in "Application Level" layer too
  children:
    - Database
    - PLSQL Package

nodes:
  - type: Database
    icon: dtb
    name: a_database
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 0
    supportedStartEdges: [PERSPECTIVE, APPLICATION_DIRECT, APPLICATION_FILTER] #new support of Aplication Level lineages in R42.2
    supportedEndEdges: [APPLICATION_DIRECT, APPLICATION_FILTER] #new support of Aplication Level lineages in R42.2
    children:
      - Schema
      - PLSQL Package

  - type: Schema
    icon: database_schema
    name: a_schema
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 0
    supportedStartEdges: [PERSPECTIVE, THIRDLEVEL_MACROEDGE]
    supportedEndEdges: [THIRDLEVEL_MACROEDGE]
    children:
      - Table
      - View
      - Procedure
      - Function
      - Synonym
      - Trigger
      - PLSQL Synonym
      - PLSQL Sequence
      - PLSQL Package
      - Oracle scripts->CreateViewScript

  - type: Table
    icon: table
    name: a_table
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 30
    supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
    supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
    children:
      - Column

  - type: View
    icon: view
    name: a_view
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 30
    supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
    supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
    children:
      - Column

  - type: Procedure
    icon: procedure_v1
    name: a_procedure
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 30
    supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
    supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
    children:
      - Parameter
      - Oracle scripts->PLSQL Block

  - type: Function
    icon: function
    name: a_function
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 30
    supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
    supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
    children:
      - Parameter
      - ReturnValue
      - Oracle scripts->PLSQL Block

  - type: Synonym
    name: a_Synonym
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 30
    supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
    supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
    children:
      - Column
      - Pseudocolumn

  - type: Trigger
    name: a_Trigger
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 30
    supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
    supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
    children:
      - Oracle scripts->PLSQL Block

  - type: PLSQL Synonym
    icon: transaction_control
    name: a_plsqlSynonym
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 30
    supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
    supportedEndEdges: [SECONDLEVEL_MACROEDGE]
    children:
      - Column

  - type: PLSQL Sequence
    icon: sequence
    name: a_plsqlSequence
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: false
    supportedStartEdges: [PERSPECTIVE]
    childNodeVisibilityLimit: 0
    children:
      - Oracle scripts->Pseudocolumn

  - type: Column
    icon: column
    name: a_column
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: false
    childNodeVisibilityLimit: 0
    supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
    supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
    children:
      - StructureItem
      - PLSQL Variable
      - Pseudocolumn

  - type: Pseudocolumn
    name: a_pseudocolumn
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: false
    childNodeVisibilityLimit: 0
    supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
    supportedEndEdges: [DIRECT, FILTER, MAPS_TO]

  - type: StructureItem
    name: a_structureItem
    icon: parameter
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: false
    childNodeVisibilityLimit: 30
    supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
    supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
    children:
      - StructureItem

  - type: Parameter
    icon: parameter
    name: a_parameter
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: false
    childNodeVisibilityLimit: 10
    supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
    supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
    children:
      - Oracle scripts->PLSQL CursorRow

  - type: ReturnValue
    icon: output
    name: a_returnvalue
    canBeDroppedToCanvas: false
    supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
    supportedEndEdges: [DIRECT, FILTER, MAPS_TO]

  - type: PLSQL Package
    name: a_plsqlPackage
    icon: package_v1
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 0
    supportedStartEdges: [PERSPECTIVE, APPLICATION_DIRECT, APPLICATION_FILTER]  #APPLICATION_DIRECT and APPLICATION_FILTER are defined here for Application Level Lineages
    supportedEndEdges: [APPLICATION_DIRECT, APPLICATION_FILTER]         #APPLICATION_DIRECT and APPLICATION_FILTER are defined here for Application Level Lineages
    children:
      - PLSQL Variable
      - Function
      - Procedure
      - Oracle scripts->PLSQL DeclareCursor
      - Oracle scripts->PLSQL Cursor

  - type: PLSQL Variable
    icon: parameter
    name: a_PLSQLVariable
    childNodeImportabilityLimit: 99
    canBeDroppedToCanvas: false
    childNodeVisibilityLimit: 30
    supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
    supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
    children:
      - PLSQL RecordField

  - type: PLSQL RecordField
    icon: parameter
    name: a_PLSQLRecordField
    canBeDroppedToCanvas: false
    supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
    supportedEndEdges: [DIRECT, FILTER, MAPS_TO]