Contribute in GitHub: Edit online

DataStage Resource Configuration

Before you configure your scanner, make sure you meet the prerequisites. Read our guide on DataStage integration requirements to double-check.

Source System Properties

This configuration can be setup by creating a new connection on Admin UI > Connections tab or editing an existing connection in Admin UI / Connections / Data Integration Tools / DataStage / specific connection. New connection can also be created via Manta Orchestration API.

One IBM Manta Data Lineage connection for DataStage corresponds to one DataStage server that will be analyzed.

Common Scanner Properties

This configuration is common for all DataStage source systems and for all DataStage scenarios, and is configure in Admin UI / Configuration / CLI / DataStage / DataStage Common. It can be overridden on individual connection level.

Manual Mapping File

The file is optional and is expected in the path defined by the common property datastage.manual.mapping.file. The format of the CSV is:

"Full path to Stage";"Input Link name";"Input Column name";"Output Link name";"Output Column name";"Edge Type (DIRECT | FILTER)";"Description (optional)"
"manual_mapping_job/Generic_3";"DSLink2";"a";"DSLink5";"b";"DIRECT";""

The path to the stage is in the format Job/[Shared and Local containers optional]/Stage.

Parameter Override File

The file is optional and is expected in the path defined by the common property datastage.parameter.override.file. This file overrides previously-loaded parameter values, so you can set them manually or add new ones.

The datastageParameterOverride.txt File Format in Standalone DataStage

[ENVIRONMENT]
PARAM1_NAME = "param1_value"
PARAM2_NAME = "param2_value"
PARAM3_NAME = "param3_value"

[PARAMETER_SET/parameter_set_name]
param4_name  = "default_param4_value"
param5_name  = "default_param5_value"
$PARAM3_NAME = "$PROJDEF"

[VALUE_FILE/parameter_set_name/value_file1_name]
param4_name  = "some_param4_value"
param5_name  = "some_param5_value"
$PARAM3_NAME = "some_param3_value"

[VALUE_FILE/parameter_set_name/value_file2_name]
param4_name  = "other_param4_value"
param5_name  = "other_param5_value"
$PARAM3_NAME = "other_param3_value"

[JOB/job1_name]
param6_name = "param6_value"
param7_name = "param7_value"

[JOB/job2_name]
param8_name = "param8_value"

Four scopes of parameters can be added to the file.

1. Project/environment-level parameters — You can add any number of them under the [ENVIRONMENT] heading. Note that the values from the DSParam file сan be overridden.

2. Parameter set default parameters — You can add any number of them under the [PARAMETER_SET/parameter_set_name] heading by specifying the name of the parameter set. You can also refer to some default environment parameters as in the example above, referring to the value $PROJDEF.

3. Parameter set value file parameters — You can add any number of them under the [VALUE_FILE/parameter_set_name/value_file_name] heading by specifying the name of the parameter set and value file. Referring to the value $PROJDEF is also allowed here. Note: Don't forget to enter the names of the value files in the datastage.value.files property.

4. Job parameters — You can add any number of them under the [JOB/job_name] heading by specifying the name of the job. Referring to the value $PROJDEF is allowed.

Use correct file format. Spaces and tabs are allowed, but each scope definition and parameter entry must be on a separate line.

The datastageParameterOverride.txt File Format in DataStage for Cloud Pak for Data

parameter_1 = value
parameter_2 = value
$environment_variable_1 = value
$environment_variable_2 = value
parameter_set_name.parameter_1 = value
parameter_set_name.parameter_2 = value

The order of the parameters in the file is not important. Spaces around = are optional.

Parameter Override Helper File

If during the analysis any unresolved parameters were found, a "helper" file, datastageParameterOverride.txt, is generated in the location defined by the datastage.parameter.override.helper.file property. The hint that the helper file gives can also be found in the log file and in logs inside Admin UI Log Viewer relevant to the current DataStage dataflow analysis. After that, you need to perform the following steps.

• Copy the helper file to the location defined by the datastage.parameter.override property. If such a file already exists in the destination folder, you can safely replace it.

  • You can also copy the contents of the helper file from the log and create the file in the above location. If such a file already exists, you can simply replace its contents.

• Now you need to open the file and fill in the values of all unresolved parameters. Such parameters are empty and may look like this: my_param = "". The parameter value must be enclosed in double quotes; for example, my_param = "my_value".

• Save and close the file.

• Run the analysis again.

ODBC Connection Definition Settings

The connection settings for IBM DataStage are exported automatically by Manta Data Lineage during lineage analysis — but to resolve the ODBC connector, Manta Data Lineage needs more information about the relevant source/destination systems. They can be configured manually as follows.

• Create or open the file referenced by the datastage.odbc.connection.definition.file property (e.g., <MANTA_DIR_HOME>/input/datastage/${datastage.extractor.server}/connection_definition/odbcConnectionDefinition.ini).

• Follow the instructions in Manually Define a Database Connection.