Open Manta Designer - Resource Templates Manual

Resource Template (RT) is a specific metadata configuration for given technology supported by OMD. By introducing new Resource Template, OMD can easily support new technologies without need to modify OMD code. It has a form of YAML file, which can be easily prepared and verified according to defined YAML schema.

With OMD/Manta release, a default set of Resource Templates is bundled along with the application in .jar file. However, users can extend this basic set of RTs or even overwrite default Resource Templates for particular technology.

There are two ways how to load new Resource Template into OMD, either by using OMD API or via storing YAML file in a particular subfolder.

Using Resource Templates API endpoints

This has an advantage of YAML validation against defined YAML schema. In case the YAML structure is not valid, the Resource Template is not uploaded into OMD and thus ensure the application remains stable.

Available endpoints:

GET {{BASE_OMD_URL}}/private/v1/resource-templates - Returns all RTs in JSON format.
POST {{BASE_OMD_URL}}/private/v1/resource-templates - Upload Resource Template YAML in request body.
DELETE {{BASE_OMD_URL}}/private/v1/resource-templates/{Resource_Type} - Remove particular Resource Template (found by Resource Type).
- If there is a default Resource Template for the one being removed, the default one is preserved and can be used in OMD.
- If default Resource Template is tried to be removed ( it has not been overwritten by some new version), it is not removed at all and thus can be used. This is to prevent OMD from missing some Resource Templates which are used in other Manta services/applications.
- IFPCf user removes custom Resource Template (and there is no other default Resource Template for this technology), it is removed and unless user provides another RT, such Resource cannot be created in OMD anymore.
- If user tries to remove Resource Template which does not exist, nothing happens at all.

Storing new Resource Templates in `resource_templates` subfolder

This is another way how to add custom Resource Templates.

The subfolder can be found in {MANTA_installation_folder}/openmantadesigner/manta-openmantadesigner-dir/conf/resource_templates/.

For each Resource Template, there can be .yml file defined. It's name must be the same as Resource Type and it can even overload default Resource Templates. For example, there can be Oracle.yml, which overloads default Oracle Resource Template bundled with OMD.

If you want to load your custom Resource Template into OMD, just copy the .YML file into resource templates subfolder and reload OMD by simply reloading the page in browser. From now on the new Resource Type is available in OMD.

Although being straightforward, this approach has one drawback - the custom YAML files are not validated.

In case OMD cannot start after Resource Template has been copied into this subfolder, it is probable the YAML file is not valid. Simply remove the YAML from the subfolder and restart OMD again either via Manta Launcher or startup / shutdow scripts.

Validation of Node hierarchy imported from Manta Flow Server or CSV

Each time some Nodes are being imported into OMD either from Manta Flow Server or via importing whole Connection from Open Manta Extensions (CSV files), the imported data are being validated against the available Resource Template. If there are found some discrepancies between imported data and RT, these are reported to the user both via OMD front-end (in Import dialog) and into Admin UI/Log Viewer.

Following situations can occur:

No Resource Template is found for imported technology In this case, OMD imports all nodes without doing any other validations. However, user cannot create any child nodes in the Connection because there is no RT associated with the given technology. Similarly, all edges are imported as well, but user cannot create new edges.
Node Type definition is missing in existing Resource Template In this case, OMD found related Resource Template based on Resource Type comparison (be aware that it is case sensitive), but the Node Type definition for the Node being imported could not be found in Resource Template.
Inconsistency found between imported Edges and Edge Types expected based on Resource Template For each Node Type defined in particular Resource Template, there are definitions about available Edge Types for given Node Type. If the imported Edges does not conform to these definitions, it is reported both to OMD front-end and Admin UI / Log Viewer.

Resource Template structure in R42

In R42, a couple of changes were made to RT structure.

Supported Edges redefined

supportedEdges attribute was replaced by two attributes supportedStartEdges and supportedEndEdges . The first enumerates available Edge Types to be started at the Node of the given Node Type, the latter the Edges which can end at the given Node Type. This can be used for scenarios. For example:

PERSPECTIVE edges start at physical assets and end at Container node in Perspective Resource.
MAPS_TO edges start from assets in Logical/Conceptual layers (such as PowerDesigner, ERwin, ER/Studio) and end at physical assets.

Referencing other Resource Templates

-> notation for referencing Resource Types can be used in case some Resource contains sub-hierarchy of Nodes from another Resource. For example, "Oracle scripts" sub-hierarchy can be found within Oracle Resource (and similarly in all other DBMS technologies and Scripts).

In the example below, within the Schema node, a CreateViewScript and its sub-hierarchy from Oracle scripts can be created:

- type: Schema
    icon: database_schema
    name: a_schema
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 0
    supportedStartEdges: [PERSPECTIVE, THIRDLEVEL_MACROEDGE]
    supportedEndEdges: [THIRDLEVEL_MACROEDGE]
    children:
      - Table
      - View
      - Procedure
      - Function
      - Synonym
      - PLSQL Synonym
      - PLSQL Sequence
      - PLSQL Package
      - Oracle scripts->CreateViewScript

Referencing Nodes groups

In case there are more Nodes and their sub-hierarchies from various Resources available, these can be references using @ notation. All Node Types which belong to the specific Node group can be a Child nodes of the Node Type which is referencing this group.

For example PLSQL Block is a Node Type from Oracle. It belongs to DBMSScripts node group along with some other nodes from e.g. MSSQL, DB2, PostgreSQL, and so on.

type: PLSQL Block
  icon: plsql_block
  name: a_plsqlBlock
  groups:
  - '@DBMSScripts'
  childNodeImportabilityLimit: 999
  canBeDroppedToCanvas: false
  children:
  - PLSQL Block
  - PLSQL Merge
  - PLSQL Return
  - PLSQL VarAssignment
  - PLSQL Insert

Then, this Node group is referenced from within SSIS Resource Template, SqlTask node in particular. It means that under SqlTask node, whatever Node from DBMScripts group can be created:

- type: SqlTask
  icon: block
  name: a_sqlTask
  childNodeImportabilityLimit: 999
  canBeDroppedToCanvas: false
  children:
  - '@DBMSScripts'

The mechanism of node groups is sometimes used even within one Resource Template, especially if there are repetitions and recursion in Node Hierarchy. This helps to keep Resource Template definition shorter and more manageable.

Defining default Node attributes and their default values

For each Node Type in RT, it can be defined if there are some default Node Attributes which are created along with a Node of such Node Type.

The following is an example of three Node Attributes (DATA_TYPE, DESCRIPTION, and DATA_QUALITY) that are created along with each Column Node.

For each node attribute, the following fields can be defined:

name [string] - Name of the node attributes (can be arbitrary string, not only UPPER CASE).
type [string] - Data type of the node attribute (it is not used now in OMD R42, all attributes are handled as strings).
defaultValue [string] - Node attribute default value.
options [list of strings] - In case the node attribute is a string selected from enumeration of predefined values.
- strictOptions [boolean] - in case the list of values is fixed or user can add new values via OMD UI.

- type: Column
  icon: column
  name: a_column
  childNodeImportabilityLimit: 999
  canBeDroppedToCanvas: false
  childNodeVisibilityLimit: 50
  supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
  supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
  children:
    - StructureItem
    - PLSQL Variable
  attrs:                          # List of default node attributes
    - name: DATA_TYPE              # Attribute name
      options:                    # Attribute type (optional) is enum
      - INT                       # The first is the default type
      - VARCHAR
      - DOUBLE
      strictOptions: false       # The listed options are only recommended
                                 # User can define any type he likes
    - name: DESCRIPTION          # Attribute name
      type: string               # Attribute type
      defaultValue: 'this is description' # Attribute default value
    - name: QUALITY              # Attribute name
      type: float                # Attribute type
      defaultValue: 0            # Attribute default value

Macroedges

New edge type was introduced SECONDLEVEL_MACROEDGE. When such edge is created between 2nd level nodes (Tables, Views, and so on), the Edge Wizard dialog is automatically opened in OMD where user can easily create many individual edges just by providing input parameters for matching algorithm.

Fallback nodes

In some cases, the Node Hierarchy defined in RT may contain fallback nodes, which are used only in case extraction&analysis of given Resource fails. Although such nodes can be imported into OMD, it should not be possible to created them manually in OMD. Such Node Types can have attribute fallbackNode: true and OMD will not allow their manual creation via UI (you can see IFPC Resource Template as an example).

A sample of R42 Resource Template for Oracle

resource:
  type: Oracle
  name: Oracle
  icon: database_source
  layers:
    - Physical
  children:
    - Database
    - PLSQL Package

nodes:
  - type: Database
    icon: dtb
    name: a_database
    supportedStartEdges: [PERSPECTIVE]
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: false
    children:
      - Schema
      - PLSQL Package

  - type: Schema
    icon: database_schema
    name: a_schema
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: true
    childNodeVisibilityLimit: 0
    supportedStartEdges: [PERSPECTIVE, THIRDLEVEL_MACROEDGE]
    supportedEndEdges: [THIRDLEVEL_MACROEDGE]
    children:
      - Table
      - View
      - Procedure
      - Function
      - Synonym
      - PLSQL Synonym
      - PLSQL Sequence
      - PLSQL Package
      - Oracle scripts->CreateViewScript

- type: Table
  icon: table
  name: a_table
  childNodeImportabilityLimit: 999
  canBeDroppedToCanvas: true
  childNodeVisibilityLimit: 30
  supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
  supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
  children:
    - Column

- type: Column
    icon: column
    name: a_column
    childNodeImportabilityLimit: 999
    canBeDroppedToCanvas: false
    childNodeVisibilityLimit: 30
    supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
    supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
    children:
      - StructureItem
      - PLSQL Variable
    attrs:                        # List of default node attributes if any (this is just an example)
    - name: DATATYPE              # Attribute name
      options:                    # Attribute type (optional) is enum
      - INT                       # The first is the default type
      - VARCHAR
      - DOUBLE
      strictOptions: false       # The listed options are only recommended
                                 # User can define any type he likes
    - name: description          # Attribute name
      type: string               # Attribute type
      defaultValue: 'this is description' # Attribute default value
    - name: quality              # Attribute name
      type: float                # Attribute type
      defaultValue: 0            # Attribute default value

...