Open Manta Designer - Administration Manual for R42.2
This manual for OMD administration describes several topics related to the operation and setup of Open Manta Designer, a visual modeling tool for designing custom data lineages.
In case of questions or problems, contact IBM Manta Data Lineage support.
Installation folder structure
In Manta INSTALLATION_ROOT
, you can find the following folders and files:
-
/openmantadesigner/
-
manta-open-manta-designer-VERSION-app.jar
-
legal_notices_js.csv
-
/manta-openmantadesigner-dir/
-
/conf/
-
oidc.properties
-
application.properties
-
artemis.properties
-
resource_templates/
.yml file for each technology
-
-
/data/
-
/log/
-
console.log
-
logging.log
-
README.txt
-
-
-
/bin/
-
startup.bat
(for Windows) -
shutdown.bat
(for Windows) -
startup.sh
(for Linux) -
shutdown.sh
(for Linux)
-
-
OMD start & stop scripts
$OMD_installationfolder/bin
subfolder.
Similarly to other Manta Data Lineage services, once the Open Manta Designer is enabled, its status can be seen directly in Manta Launcher from where it can be accessed as well.
OMD users and roles in Keycloak
oidc.properties
in the OMD_installationfolder/manta-openmantadesigner-dir/conf/
subfolder contains the URI of the Keycloak for authentication and client Secret that can be obtained in Keycloak administration.
# Keycloak address
manta.oidc.issuerUri=http://localhost:9090/auth/realms/manta
# OAuth's client ID
manta.oidc.clientId=omd-client
# Special for FE - Used in FE authentication
manta.oidc.publicClientId=omd-fe-client
# Secret from Keycloak
manta.oidc.clientSecret= PASTE YOUR clientSecret HERE
# All the necessary information we need
manta.oidc.scope=roles,profile,openid
# Web token's Json path, where are the roles stored
manta.oidc.roleClaim=realm_access.roles
If you want to add a new OMD user, create a new user account in Keycloak and assign proper roles.
OMD uses several user roles for its operation, which are automatically created in Keycloak when Manta is being installed. Anyone who wants to work with OMD must have at least one of the following roles:
OPEN_MANTA_UI_READ
- Role enabling read-only access to OMD. No Create/Update or Delete operations are allowed for this role.OPEN_MANTA_UI_WRITE
- Role enabling the creation of new / modification of existing Connections in OMD including import of existing metadata either from the Manta Flow repository or from Open Manta Extension CSV files. Users with this role can access only Connections created by themselves. A user should have bothOPEN_MANTA_UI_READ
andOPEN_MANTA_UI_WRITE
roles for full access to his/her Connections.OPEN_MANTA_UI_SUPER_USER
- role enabling full access to all Connections of all users working with OMD within the particular Manta installation.
application.properties file
In the application.properties
file, you can define some properties influencing connection timeouts between OMD and the Manta Flow server and size of Open Manta Extension CSV files being imported into OMD.
manta.openmantadesigner.flowserver.service-request.read-timeout-seconds=120
manta.openmantadesigner.flowserver.service-request.connect-timeout-seconds=30
spring.servlet.multipart.max-file-size=10MB
spring.servlet.multipart.max-request-size=10MB
artemis.properties file
In this file, all the relevant properties which are used for logging messages from OMD into Artemis are defined. Mostly these logs cover warnings and error which might occur during import of Nodes from Manta Flow Viewer (or CSV/JSON), especially if relevant Resource Templates are not found for Resources being imported or in case some Node Type definitions are missing in the Resource Templates.
The messages being sent from OMD into Artemis can be then accessed in Manta Admin UI / Log Viewer.
manta.artemis.keystore-password=PASTE YOUR keystore password HERE
manta.artemis.keystore-path= #full path to .../manta-openmantadesigner-dir/conf/artemiskeystore
manta.artemis.mtls-enabled=true
manta.artemis.truststore-password=PASTE YOUR truststore password HERE
manta.artemis.truststore-path= #full path to .../manta-openmantadesigner-dir/conf/artemistruststore
manta.artemis.port=61616 #artemis port number
Configuration setting YAML
OMD uses several configuration settings that influence its behaviour. These are stored in settings.yml
which is bundled directly in OMD java archive (.jar). However, these settings can be overwritten by creating your own custom settings.yml
and storing it in the $OMD_installationfolder/manta-openmantadesigner-dir/conf/
subfolder.
settings.yml
, it is necessary to restart OMD in order to apply the changes. In case you encounter some unexpected errors immediately after your custom settings.yml
was applied, just remove it from the manta-openmantadesigner-dir/conf/
folder, restart OMD and it will again use the default one bundled in java archive.
The setting.yml
file contains the following sections:
-
connections
icon
- Default icon for the connection which is used in the root level of the Open Manta Designer repository tree. The name of the icon must match an icon name from the manta-fe-library.name
- Validation patterns for Connection name, regEx, and minimum/maximum Connection name length
-
layers
- List of all Layer Types (e.g. Physical, Logical, …) that are offered to the user in the combo box when a new layer is created in OMD. For each Layer type, specify the following properties:name
- layer type name.default-name
- Default layer name.icon
- Select an icon that appears at the appropriate repository tree level. The name of the icon must match an icon name from the manta-fe-library.name
- Common validation pattern for Layer Names entered by the user. Its RegEx and min/max name length.
-
resources
defaultType
- Default resource type preselected in the New resource modal dialog. It has to match with some existing Resource types supported by OMD.name
- Validation pattern for Resource names entered by the user.icon
- Default icon for Resource in OMD Repository Tree (it is used in case no other icon is specified in the appropriate Resource template).
-
nodes
name
- Validation pattern for Node names entered by the user.icon
- Default icon for Node in OMD Repository Tree (it is used in case no other icon is specified in the appropriate Resource template).
-
canvases
name
- Validation pattern for Canvas names.
-
csv
input/import/
- The hierarchy of subfolders to be used within ZIP archive with “Open Manta Extension” CSV files.
settings.yml
connections:
icon: repository
name:
pattern: '^[^*&!@#$)(%^_~+?|":}{\s](.*\S)?$'
min-length: 1
max-length: 128
layers:
types:
-
name: Physical
default-name: Physical
icon: layer
-
name: Logical
default-name: Logical
icon: expression
-
name: Conceptual
default-name: Conceptual
icon: balance_data_distribution
-
name: Perspective
default-name: Perspective
icon: view
-
name: Application Level
default-name: Application Level
icon: view
name:
pattern: '^[^*&!@#$)(%^_~+?|":}{\s](.*\S)?$'
min-length: 1
max-length: 128
resources:
defaultType: Oracle
name:
pattern: '^[^*&!@#$)(%^_~+?|":}{\s](.*\S)?$'
min-length: 1
max-length: 128
icon: server
nodes:
name:
pattern: '^[^*&!@#$)(%^_~+?|":}{\s](.*\S)?$'
min-length: 1
max-length: 128
icon: asset
canvases:
name:
pattern: '^.*$'
min-length: 1
max-length: 128
csv:
export:
folder: 'input/import/'
How to manage Resource Templates
Resource Template (RT) is a specific metadata configuration for given technology supported by OMD. By introducing new Resource Template, OMD can easily support new technologies without need to modify OMD code. It has a form of YAML file, which can be easily prepared and verified according to defined YAML schema.
There are two ways how to load new Resource Template into OMD, either by using OMD API or via storing YAML file in a particular subfolder.
Using Resource Templates API endpoints
This has an advantage of YAML validation against defined YAML schema. In case the YAML structure is not valid, the Resource Template is not uploaded into OMD and thus ensure the application remains stable.
Available endpoints:
GET {{BASE_OMD_URL}}/private/v1/resource-templates
- Returns all RTs in the JSON format.POST {{BASE_OMD_URL}}/private/v1/resource-templates
- Upload Resource Template YAML in request body.-
DELETE {{BASE_OMD_URL}}/private/v1/resource-templates/{Resource_Type}
- Remove particular Resource Template (found by Resource Type).- If there is a default Resource Template for the one being removed, the default one is preserved and can be used in OMD.
- If default Resource Template is tried to be removed (i.e. it has not been overwritten by some new version), it is not removed at all and thus can be used. This is to prevent OMD from missing some Resource Templates which are used in other Manta services/applications.
- If user removes custom Resource Template (and there is no other default Resource Template for this technology), it is removed and unless user provides another RT, such Resource cannot be created in OMD anymore.
- If user tries to remove Resource Template which does not exist, nothing happens at all.
Storing new Resource Templates in resource_templates
subfolder
This is another way how to add custom Resource Templates.
The subfolder can be found in {MANTA_installation_folder}/openmantadesigner/manta-openmantadesigner-dir/conf/resource_templates/
.
For each Resource Template, there can be .yml
file defined. It's name must be the same as Resource Type and it can even overload default Resource Templates. For example, there can be Oracle.yml
, which overloads default
Oracle Resource Template bundled with OMD).
Although being straightforward, this approach has one drawback - the custom YAML files are not validated.
startup
/ shutdow
scripts.
Validation of Node hierarchy imported from Manta Flow Server or CSV
Each time some Nodes are being imported into OMD either from Manta Flow Server or via importing whole Connection from Open Manta Extensions (CSV files), the imported data are being validated against the available Resource Template. If there are found some discrepancies between imported data and RT, these are reported to the user both via OMD front-end (in Import dialog) and into Admin UI/Log Viewer.
Following situations can occur:
-
No Resource Template is found for imported technology In this case, OMD imports all nodes without doing any other validations. However, user cannot create any child nodes in the Connection because there is no RT associated with the given technology. Similarly, all edges are imported as well, but user cannot create new edges.
-
Node Type definition is missing in existing Resource Template In this case, OMD found related Resource Template based on Resource Type comparison (be aware that it is case sensitive), but the Node Type definition for the Node being imported could not be found in Resource Template.
-
Inconsistency found between imported Edges and Edge Types expected based on Resource Template For each Node Type defined in particular Resource Template, there are definitions about available Edge Types for given Node Type. If the imported Edges does not conform to these definitions, it is reported both to OMD front-end and Admin UI / Log Viewer.
Resource Template structure in R42.2
A Resource Template has following structure.
resource level
type
[string] - Resource Type - it should match to the Resource Types used by Manta in case of technologies which can be automatically extracted. Otherwise user can use whatever Resource Type needed. It will be used in Resource Type selection combo box when new Resource is being created in OMD.name
[string] - Default Resource Name for the new Resource being created in OMD. Usually it is identical to Resource Type.icon
[string] - A name of an icon which will be used for representing this particular Resource. The icon is used in OMD Repository Tree and at Canvas.layers
[string] - List of layer types in which the Resource can be created (for example, Physical, Logical, and so on). It must have values from those defined insettings.yml
. This attribute is optional and if omitted, the Resource can be created in all Layers Typeschildren
[list of nodes] - The top-most level Node Types within this Resource. From this Node(s), the subsequent hierarchy is derived.nodes
- Particular Node Type definitions within the Resource. These definitions form whole Node Types hierarchies which is used for validations when new Node is being created in OMD.
nodes level
type
[string] - Node Type - it should match to the Node Types used by Manta Viewer in case of technologies which can be automatically extracted. Otherwise user can use whatever Node Types needed. It is used in Node Type selection combo box when new Node is being created in OMD.name
[string] - Default Node Name for a new Node of this Type.icon
[string] - A name of an icon which will be used for representing this particular Node. The icon is used in OMD Repository Tree and at Canvas.childNodeImportabilityLimit
[integer]- Limit for importing Child Nodes from Flow Repository - If some Node of this Type is being imported from Flow Repository, this Limit influences whether the Child Nodes will be imported along with this Node (if number of actual Child nodes \<=childNodeImportabilityLimit
) or not.canBeDroppedToCanvas
[boolean] - If a Node of this type can be placed to OMD Canvas by drag&drop from Repository Tree.childNodeVisibilityLimit
[integer] - Limit for displaying Child Nodes if a Node of this Type is placed to Canvas. It makes sense only if previous attribute isTrue
, otherwise it has no effect. If the Node being placed to Canvas has less Child Nodes than this Limit, all of them are displayed at Canvas after placement. Otherwise no Child Nodes are shown by default.supportedStartEdges
[enum] - List of Edge Types which can start from this Node TypesupportedEndEdges
[enum] - List of Edge Types which can terminate to this Node Typechildren
[list of nodes] - available Child Node Types for the given Node Type. If more Types are available, the first is the default Node used in Create New Node dialog. Each Node Type from this list must have its corresponding definition in the YAML.
Referencing other Resource Templates
-> notation for referencing Resource Types
can be used in case some Resource contains sub-hierarchy of Nodes from another Resource. For example, "Oracle scripts" sub-hierarchy can be found within Oracle Resource (and similarly
in all other DBMS technologies and Scripts).
In the example, within the Schema
node, a CreateViewScript
and its sub-hierarchy from Oracle scripts
can be created:
- type: Schema
icon: database_schema
name: a_schema
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: true
childNodeVisibilityLimit: 0
supportedStartEdges: [PERSPECTIVE, THIRDLEVEL_MACROEDGE]
supportedEndEdges: [THIRDLEVEL_MACROEDGE]
children:
- Table
- View
- Procedure
- Function
- Synonym
- PLSQL Synonym
- PLSQL Sequence
- PLSQL Package
- Oracle scripts->CreateViewScript
Referencing Nodes groups
In case there are more Nodes and their sub-hierarchies from various Resources available, these can be references using @ notation
. All Node Types which belong to the specific Node group can be a Child nodes of the Node Type which is
referencing this group.
For example PLSQL Block
is a Node Type from Oracle. It belongs to DBMSScripts
node group along with some other nodes from e.g. MSSQL, DB2, PostgreSQL, and so on.
type: PLSQL Block
icon: plsql_block
name: a_plsqlBlock
groups:
- '@DBMSScripts'
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: false
children:
- PLSQL Block
- PLSQL Merge
- PLSQL Return
- PLSQL VarAssignment
- PLSQL Insert
Then, this Node group is referenced from within SSIS Resource Template, SqlTask
node in particular. It means that under SqlTask
node, whatever Node from DBMScripts
group can be created:
- type: SqlTask
icon: block
name: a_sqlTask
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: false
children:
- '@DBMSScripts'
Defining default Node attributes and their default values
For each Node Type in RT, it can be defined if there are some default Node Attributes which are created along with a Node of such Node Type.
The following is an example of three Node Attributes (DATA_TYPE
, DESCRIPTION
, and DATA_QUALITY
) that are created along with each Column Node.
For each node attribute, the following fields can be defined:
name
[string] - Name of the node attributes (can be arbitrary string, not only UPPER CASE).type
[string] - data type of the node attribute (it is not used now in OMD R42, all attributes are handled as strings).defaultValue
[string] - Node attribute default value.-
options
[list of strings] - In case the node attribute is a string selected from enumeration of predefined values.strictOptions
[boolean] - In case the list of values is fixed or user can add new values via OMD UI.
- type: Column
icon: column
name: a_column
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: false
childNodeVisibilityLimit: 50
supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
children:
- StructureItem
- PLSQL Variable
attrs: # List of default node attributes
- name: DATA_TYPE # Attribute name
options: # Attribute type (optional) is enum
- INT # The first is the default type
- VARCHAR
- DOUBLE
strictOptions: false # The listed options are only recommended
# User can define any type he likes
- name: DESCRIPTION # Attribute name
type: string # Attribute type
defaultValue: 'this is description' # Attribute default value
- name: QUALITY # Attribute name
type: float # Attribute type
defaultValue: 0 # Attribute default value
Macroedges
New edge type was introduced SECONDLEVEL_MACROEDGE
. When such edge is created between 2nd level nodes (Tables, Views, and so on), the Edge Wizard dialog is automatically opened in OMD where user can easily create many individual edges
just by providing input parameters for matching algorithm.
Fallback nodes
In some cases, the Node Hierarchy defined in RT may contain fallback nodes, which are used only in case extraction and analysis if a given Resource fails. Although such nodes can be imported into OMD, it should not be possible to created them manually
in OMD. Such Node Types can have the fallbackNode: true
attribute and OMD does not allow for them to be created manually via UI. You can see IFPC Resource Template as an example.
A sample of R42 Resource Template for Oracle
resource:
type: Oracle
name: Oracle
icon: database_source
layers:
- Physical
- Application Level #this Resource can be created in "Application Level" layer too
children:
- Database
- PLSQL Package
nodes:
- type: Database
icon: dtb
name: a_database
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: true
childNodeVisibilityLimit: 0
supportedStartEdges: [PERSPECTIVE, APPLICATION_DIRECT, APPLICATION_FILTER] #new support of Aplication Level lineages in R42.2
supportedEndEdges: [APPLICATION_DIRECT, APPLICATION_FILTER] #new support of Aplication Level lineages in R42.2
children:
- Schema
- PLSQL Package
- type: Schema
icon: database_schema
name: a_schema
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: true
childNodeVisibilityLimit: 0
supportedStartEdges: [PERSPECTIVE, THIRDLEVEL_MACROEDGE]
supportedEndEdges: [THIRDLEVEL_MACROEDGE]
children:
- Table
- View
- Procedure
- Function
- Synonym
- Trigger
- PLSQL Synonym
- PLSQL Sequence
- PLSQL Package
- Oracle scripts->CreateViewScript
- type: Table
icon: table
name: a_table
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: true
childNodeVisibilityLimit: 30
supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
children:
- Column
- type: View
icon: view
name: a_view
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: true
childNodeVisibilityLimit: 30
supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
children:
- Column
- type: Procedure
icon: procedure_v1
name: a_procedure
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: true
childNodeVisibilityLimit: 30
supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
children:
- Parameter
- Oracle scripts->PLSQL Block
- type: Function
icon: function
name: a_function
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: true
childNodeVisibilityLimit: 30
supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
children:
- Parameter
- ReturnValue
- Oracle scripts->PLSQL Block
- type: Synonym
name: a_Synonym
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: true
childNodeVisibilityLimit: 30
supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
children:
- Column
- Pseudocolumn
- type: Trigger
name: a_Trigger
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: true
childNodeVisibilityLimit: 30
supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
supportedEndEdges: [SECONDLEVEL_MACROEDGE, MAPS_TO]
children:
- Oracle scripts->PLSQL Block
- type: PLSQL Synonym
icon: transaction_control
name: a_plsqlSynonym
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: true
childNodeVisibilityLimit: 30
supportedStartEdges: [PERSPECTIVE, SECONDLEVEL_MACROEDGE]
supportedEndEdges: [SECONDLEVEL_MACROEDGE]
children:
- Column
- type: PLSQL Sequence
icon: sequence
name: a_plsqlSequence
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: false
supportedStartEdges: [PERSPECTIVE]
childNodeVisibilityLimit: 0
children:
- Oracle scripts->Pseudocolumn
- type: Column
icon: column
name: a_column
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: false
childNodeVisibilityLimit: 0
supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
children:
- StructureItem
- PLSQL Variable
- Pseudocolumn
- type: Pseudocolumn
name: a_pseudocolumn
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: false
childNodeVisibilityLimit: 0
supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
- type: StructureItem
name: a_structureItem
icon: parameter
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: false
childNodeVisibilityLimit: 30
supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
children:
- StructureItem
- type: Parameter
icon: parameter
name: a_parameter
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: false
childNodeVisibilityLimit: 10
supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
children:
- Oracle scripts->PLSQL CursorRow
- type: ReturnValue
icon: output
name: a_returnvalue
canBeDroppedToCanvas: false
supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
- type: PLSQL Package
name: a_plsqlPackage
icon: package_v1
childNodeImportabilityLimit: 999
canBeDroppedToCanvas: true
childNodeVisibilityLimit: 0
supportedStartEdges: [PERSPECTIVE, APPLICATION_DIRECT, APPLICATION_FILTER] #APPLICATION_DIRECT and APPLICATION_FILTER are defined here for Application Level Lineages
supportedEndEdges: [APPLICATION_DIRECT, APPLICATION_FILTER] #APPLICATION_DIRECT and APPLICATION_FILTER are defined here for Application Level Lineages
children:
- PLSQL Variable
- Function
- Procedure
- Oracle scripts->PLSQL DeclareCursor
- Oracle scripts->PLSQL Cursor
- type: PLSQL Variable
icon: parameter
name: a_PLSQLVariable
childNodeImportabilityLimit: 99
canBeDroppedToCanvas: false
childNodeVisibilityLimit: 30
supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
supportedEndEdges: [DIRECT, FILTER, MAPS_TO]
children:
- PLSQL RecordField
- type: PLSQL RecordField
icon: parameter
name: a_PLSQLRecordField
canBeDroppedToCanvas: false
supportedStartEdges: [DIRECT, FILTER, PERSPECTIVE]
supportedEndEdges: [DIRECT, FILTER, MAPS_TO]