Contribute in GitHub:

Manta Orchestration API Scripts

It is possible to use Manta Orchestration API with the help of prepared scripts stored in <INSTALATION_DIR>/serviceutility/manta-admin-ui-dir/bin. All the tools needed to run the script have been added, so there is no need to install any additional tools. To run a script in Windows, you need CMD or Powershell. To run scripts in Linux, you need bash.

There are three generic scripts that can be used to execute, terminate, and check the status of a workflow or scenario.

There are also standard scripts for running individual scenarios (e.g., odiExtractorScenario) or entire blocks of an individual phase (e.g., _run_extract).

Authentication

The scripts load the file mantaflow/serviceutility/manta-admin-ui-dir/conf/oidc-api.properties (prior to R40 <INSTALATION_DIR>/serviceutility/WEB-INF/bin/oidc.properties). Refer to API Token-Based Authentication (as of R40)for information about authentication.

HTTPS Configuration

By default, Orchestration API scripts are set up for HTTP configuration. When HTTPS is configured (and especially when it is enforced, i.e., pure HTTP is no longer available), the following changes are required.

Review the oidc.properties file to verify that the manta.oidc.issuerUri value is pointing to the correct server and port number (that have likely changed for HTTPS) and leveraging the correct communication protocol (https).
Provide the path to the certificate that is used to encrypt the HTTPS connection for Service Utility.
1. For R40.1 and newer, use the -c|--certificate parameter to point to the correct Service Utility certificate. This change can be done directly in _execute.[bat|sh] (but will need to be reapplied after each upgrade) or, even better, in the scripts that you are using to trigger the built-in ones.
2. For versions prior to R40.1, add the --cacert /path/to/cert.crt arguments to the CURL_PATH variable in the _execute.sh file. Note that this change has to be reapplied after every upgrade of IBM Manta Data Lineage until the abovementioned -c option can be used on R40.1 or newer.

Script Documentation

Script for Executing a Workflow or a Scenario

Execute a workflow or scenario via:

Linux: <DIR>/_execute.sh
Windows: <DIR>\_execute.bat

Parameter	Description	Example	Mandatory	Notes
-h, --help	Show script help		no
-c, --certificate	Path to the certificate (public key) used by Manta Service Utility HTTPS. The certificate must be in PEM format. The hostname in the certificate should match the hostname used to access Manta Data Lineage. (Available as of R40.1)	-c /home/path/to/mantaserviceutility/https_certificate.pem	no; needed only when HTTPS is used by Manta Service Utility
-s, --scenario	Scenario name to execute; the name of the scenario is case sensitive	-s collibraTerminateExportScenario	yes; only one of -s -w can be used
-w, --workflow	Workflow name to execute; the name of the workflow is case sensitive If the workflow contains a space in the name, the entire name must be enclosed in quotes	-w customWorkflowName -w "Run Extraction"	yes; only one of -s -w can be used
--wait	If the script should wait for the workflow to complete (the default value is true)	--wait false	no	2
--host	The Manta Admin UI base URL which the command will be executed against; enter the URL without the trailing slash, like in the example The parameter can be replaced with the environment variable `ADMIN_UI_BASE_URL=http://localhost:8181/manta-admin-gui` The parameter can be replaced with the environment variable `MANTA_ADMIN_HOST=http://localhost:8181/manta-admin-gui`	`--host http://localhost:8181/manta-admin-gui`	yes	3
--inputs-file	Path to a zip file containing additional inputs for the workflow. (The zip file will be sent to Process Manager.) The path must refer to a file that is reachable from where the script is executed. A zip file is expected, containing an Input folder with the same hierarchy as when inputs are provided directly via filesystem.	--inputs-file additional-inputs.zip	no; only one of --inputs-file --inputs-path can be used
--inputs-path	Path to a shared folder containing additional inputs for the workflow. (The path will be sent to Process Manager.) Path to a shared folder containing additional inputs for the workflow. The folder is considered to be on the machine where Manta Data Lineage is installed. See Preparing Scanner Inputs.	--inputs-path /input/path	no; only one of --inputs-file --inputs-path can be used

Example Output

./_execute.sh --workflow Run --username User --password secretPass --host http://localhost:8181/manta-admin-gui 
Execute workflow Run: START
Execute workflow 'Run': status=PENDING  executionId='159'
Workflow state can be check with command
   ./_status.sh -u $USERNAME -p $PASSWORD --host http://localhost:8181/manta-admin-gui -e 159

Workflow can be terminated with command
   ./_terminate.sh -u $USERNAME -p $PASSWORD --host http://localhost:8181/manta-admin-gui -e 159

Execute workflow 'Run': status=RUNNING  executionId='159' stepFinished=8/24 
Execute workflow 'Run': status=DONE  executionId='159' stepFinished=24/24 
Last server response
{"executionId":"159","workflowDefinitionName":"Run","status":"DONE","executedBy":"User","addedToWaitingQueue":"2022-02-17T08:16:54.841Z","executionStart":"2022-02-17T08:16:56.234Z","executionEnd":"2022-02-17T08:17:01.095Z","workflowExecutionStepsNumber":24,"workflowExecutionFinishedSteps":24,"outputExpected":false,"outputSize":-1,"workflowExecutionSteps":[]}

Standard Scripts for Running Individual Scenarios

The first script executes a workflow or scenario using a name.

This second part is to refer to the prepared script that already contains the scenario name.

Example:

./_execute.sh -s odiExtractorScenario -u User -p pass --host http://localhost:8181/manta-admin-gui
./odiExtractorScenario.sh  -u User -p pass --host http://localhost:8181/manta-admin-gui

Example for odiExtractorScenarioExecute :

Linux: <DIR>/odiExtractorScenario.sh
Windows: <DIR>\odiExtractorScenario.bat

Parameter	Description	Example	Mandatory	Notes
-h, --help	Show script help		no
-c, --certificate	Path to the certificate (public key) used by Manta Service Utility HTTPS. The certificate must be in PEM format. The hostname in the certificate should match the hostname used to access Manta Data Lineage. (Available as of R40.1)	-c /home/path/to/mantaserviceutility/https_certificate.pem	no; needed only when HTTPS is used by Manta Service Utility
--wait	If the script should wait for the workflow to complete (the default value is true)	--wait false	no	2
--host	The Manta Admin UI base URL which the command will be executed against; enter the URL without the trailing slash, like in the example The parameter can be replaced with the environment variable `ADMIN_UI_BASE_URL=http://localhost:8181/manta-admin-gui` The parameter can be replaced with the environment variable `MANTA_ADMIN_HOST=http://localhost:8181/manta-admin-gui`	`--host http://localhost:8181/manta-admin-gui`	yes	3
--inputs-file	Path to a zip file containing additional inputs for the workflow. (The zip file will be sent to Process Manager.) The path must refer to a file that is reachable from where the script is executed. A zip file is expected, containing an Input folder with the same hierarchy as when inputs are provided directly via filesystem.	--inputs-file additional-inputs.zip	no
--inputs-path	Path to a shared folder containing additional inputs for the workflow. (The path will be sent to Process Manager.) Path to a shared folder containing additional inputs for the workflow. The folder is considered to be on the machine where Manta Data Lineage is installed. See Preparing Scanner Inputs.	--inputs-path /input/path	no

Example Output

./odiExtractorScenario.sh --username User --password secretPass --host http://localhost:8181/manta-admin-gui 
Execute workflow odiExtractorScenario: START
Execute workflow 'odiExtractorScenario': status=PENDING  executionId='159'
Workflow state can be check with command
   ./_status.sh -u $USERNAME -p $PASSWORD --host http://localhost:8181/manta-admin-gui -e 159

Workflow can be terminated with command
   ./_terminate.sh -u $USERNAME -p $PASSWORD --host http://localhost:8181/manta-admin-gui -e 159

Execute workflow 'odiExtractorScenario': status=RUNNING  executionId='159' stepFinished=8/24 
Execute workflow 'odiExtractorScenario': status=DONE  executionId='159' stepFinished=24/24 
Last server response
{"executionId":"159","workflowDefinitionName":"odiExtractorScenario","status":"DONE","executedBy":"User","addedToWaitingQueue":"2022-02-17T08:16:54.841Z","executionStart":"2022-02-17T08:16:56.234Z","executionEnd":"2022-02-17T08:17:01.095Z","workflowExecutionStepsNumber":24,"workflowExecutionFinishedSteps":24,"outputExpected":false,"outputSize":-1,"workflowExecutionSteps":[]}

Script to Get Information about the Executed Workflow

Get information about the status of the workflow via:

Linux: <DIR>/_status.sh
Windows: <DIR>\_status.bat

Parameter	Description	Example	Mandatory	Notes
-h, --help	Show script help		no
-c, --certificate	Path to the certificate (public key) used by Manta Service Utility HTTPS. The certificate must be in PEM format. The hostname in the certificate should match the hostname used to access Manta Data Lineage. (Available as of R40.1)	-c /home/path/to/mantaserviceutility/https_certificate.pem	no; needed only when HTTPS is used by Manta Service Utility
--host	The Manta Admin UI base URL that the command will be executed against; enter the URL without the trailing slash, like in the example The parameter can be replaced with the environment variable `ADMIN_UI_BASE_URL=http://localhost:8181/manta-admin-gui` The parameter can be replaced with the environment variable `MANTA_ADMIN_HOST=http://localhost:8181/manta-admin-gui`	`--host http://localhost:8181/manta-admin-gui`	yes	3
-e, --executionId	Unique execution ID	-e 42	yes	4
--detailed-status	Show detailed status with all workflow step status	--long-description	no

Example Output

 ./_status.sh --executionId 159 --username User --password secretPass --host http://localhost:8181/manta-admin-gui  
Workflow 'Run': status=DONE  executionId='159' stepFinish=24/24
  Status json:
  {"executionId":"159","workflowDefinitionName":"Run","status":"DONE","executedBy":"User","addedToWaitingQueue":"2022-02-17T08:16:54.841Z","executionStart":"2022-02-17T08:16:56.234Z","executionEnd":"2022-02-17T08:17:01.095Z","workflowExecutionStepsNumber":24,"workflowExecutionFinishedSteps":24,"outputExpected":false,"outputSize":-1,"workflowExecutionSteps":[]}

./_status.sh --detailed-status --executionId 8 --username User --password secretPass --host http://localhost:8090/manta-admin-gui
Workflow 'Postgres': status=DONE  executionId='8' stepFinish=3/3
  Status json:
  {"executionId":"8","workflowDefinitionName":"Postgres","status":"DONE","executedBy":"petr","addedToWaitingQueue":"2022-02-23T07:53:39.889Z","executionStart":"2022-02-23T07:53:40.370Z","executionEnd":"2022-02-23T07:53:50.734Z","workflowExecutionStepsNumber":3,"workflowExecutionFinishedSteps":3,"outputExpected":false,"outputSize":-1,"workflowExecutionSteps":[{"scenarioName":"diagnoseRepositoryScenario","connectionName":"","executionStatus":"DONE","executionStart":"2022-02-23T07:53:40.435Z","executionEnd":"2022-02-23T07:53:40.533Z","executionProgress":100},{"scenarioName":"postgresqlDictionaryMappingScenario","connectionName":"local","executionStatus":"DONE","executionStart":"2022-02-23T07:53:40.554Z","executionEnd":"2022-02-23T07:53:40.559Z","executionProgress":100},{"scenarioName":"postgresqlExtractorScenario","connectionName":"local","executionStatus":"DONE","executionStart":"2022-02-23T07:53:40.594Z","executionEnd":"2022-02-23T07:53:40.598Z","executionProgress":100}]}

Script to Terminate a Running Workflow

Terminate a running workflow via:

Linux: <DIR>/_terminate.sh
Windows: <DIR>\_terminate.bat

Parameter	Description	Example	Mandatory	Notes
-h, --help	Show script help		no
-c, --certificate	Path to the certificate (public key) used by Manta Service Utility HTTPS. The certificate must be in PEM format. The hostname in the certificate should match the hostname used to access Manta Data Lineage. (Available as of R40.1)	-c /home/path/to/mantaserviceutility/https_certificate.pem	no; needed only when HTTPS is used by Manta Service Utility
--host	The Manta Admin UI base URL that the command will be executed against; enter the URL without the trailing slash, like in the example The parameter can be replaced with the environment variable `ADMIN_UI_BASE_URL=http://localhost:8181/manta-admin-gui` The parameter can be replaced with the environment variable `MANTA_ADMIN_HOST=http://localhost:8181/manta-admin-gui`	`--host http://localhost:8181/manta-admin-gui`	yes	3
-e, --executionId	Unique execution ID	-e 42	yes	4
--force-terminate	Enforce workflow termination even when there is an uncommitted revision; for example, an extraction step has completed and now dataflow analysis is running	--force-terminate	no

Example Output

 ./_terminate.sh --executionId 159 --username User --password secretPass --host http://localhost:8181/manta-admin-gui  
Workflow with execution id '159' was terminated

Notes

Note 1

The password entered will be visible in the terminal history. When using Linux, if the first character is a space, the command will not be saved in the history.

Saved in history
./_status.sh --executionId 159
Not saved in history
./_status.sh --executionId 159

Note 2

The --wait parameter affects whether the script will be active until the workflow completes or not.
Example

./_execute.sh --wait false --workflow Run --username User --password secretPass --host http://localhost:8181/manta-admin-gui 
Execute workflow Run: START
Execute workflow 'Run': status=PENDING  executionId='159'
Workflow state can be check with command
./_status.sh -u $USERNAME -p $PASSWORD --host http://localhost:8181/manta-admin-gui -e 159

Workflow can be terminated with command
./_terminate.sh -u $USERNAME -p $PASSWORD --host http://localhost:8181/manta-admin-gui -e 159
script end

./_execute.sh --wait true --workflow Run --username User --password secretPass --host http://localhost:8181/manta-admin-gui 
Execute workflow Run: START
Execute workflow 'Run': status=PENDING  executionId='159'
Workflow state can be check with command
./_status.sh -u $USERNAME -p $PASSWORD --host http://localhost:8181/manta-admin-gui -e 159

Workflow can be terminated with command
./_terminate.sh -u $USERNAME -p $PASSWORD --host http://localhost:8181/manta-admin-gui -e 159

Execute workflow 'Run': status=RUNNING  executionId='159' stepFinished=8/24 
Execute workflow 'Run': status=RUNNING  executionId='159' stepFinished=12/24 
Execute workflow 'Run': status=RUNNING  executionId='159' stepFinished=18/24 
Execute workflow 'Run': status=RUNNING  executionId='159' stepFinished=20/24 
Execute workflow 'Run': status=DONE  executionId='159' stepFinished=24/24 
Last server response
{"executionId":"159","workflowDefinitionName":"Run","status":"DONE","executedBy":"User","addedToWaitingQueue":"2022-02-17T08:16:54.841Z","executionStart":"2022-02-17T08:16:56.234Z","executionEnd":"2022-02-17T08:17:01.095Z","workflowExecutionStepsNumber":24,"workflowExecutionFinishedSteps":24,"outputExpected":false,"outputSize":-1,"workflowExecutionSteps":[]}
script end

Note 3

If you are using a self-signed certificate, you need to pass the certificate information to cURL before you can run the script. cURL uses the specified certificate file to verify the peer. The file may contain multiple CA certificates. The certificate(s) must be in PEM format.
cURL recognizes the environment variable named CURL_CA_BUNDLE, if it is set, and uses the given path as a path to a CA cert bundle.
The Windows version of cURL will automatically look for a CA cert file named curl-ca-bundle.crt either in the same directory as curl.exe, in the current working directory, or in any folder along your PATH.

Note 4

The executionId of the executed workflow can be obtained as output from the script _execute.sh|bat.
For example, here is executionId 159.

Execute workflow 'Run': status=PENDING  executionId='159'

Note 5

The .netrc file contains the login and initialization information used by the auto-login process. The .netrc file is typically stored in a user's home directory. The .netrc file format is simple: you specify lines with a machine name and follow that with lines for the login and password that are associated with that machine.
An example with .netrc for the host manta.com with a user named "user" using the password "pass" would look like:

    machine manta.com login user password pass

More information can be found at cULR netrc.