IBM DataStage for Cloud Pak for Data lineage configuration

Prepare a .zip file as an external input

You can provide DataStage flows as external inputs in a .zip file. The folder must have the following structure:

• <project_export.zip> - A DataStage project exported to a .zip file.
• DSParams - A file that contains the project- or environment-level parameters if applicable. You can get this file from the project directory.
• datastageParameterOverride.txt - A file with parameter-set overrides if your jobs use parameter sets.
• connection_definition/odbcConnectionDefinition.ini - A file with connection definitions for ODBC connections. Definitions of ODBC connections are not included in the DataStage XML exports and must be specified separately.
• datastageComponentOverrides.csv - A file with component-lineage overrides.

The format of the .zip file with the exported DataStage project When you export a DataStage project, it must have the following structure:

• assets - required folder.
  • .METADATA - required folder.
  • data_intg_flow.*.json - required files that contain information about flows.
  • connection.*.json - optional files that contain information about connections.
  • parameter_set.*.json - optional files that contain information about parameter sets.
  • job.*.json - optional files that contain information about jobs.
  • job_run.*.json - optional files that contain information about particular executions of the job.
  • data_intg_flow - required folder.
  • At least one file that contains the string "schemas":[{, but does not end in px_executables.
• assettypes - required folder.
• project.json - required file. There might be multiple instances of this file as a result of ZIP decompression, which is correct.

The datastageParameterOverride.txt file format The datastageParameterOverride.txt file has the following content:

[ENVIRONMENT]
PARAM1_NAME = "param1_value"
PARAM2_NAME = "param2_value"
PARAM3_NAME = "param3_value"
[PARAMETER_SET/parameter_set_name]
param4_name  = "default_param4_value"
param5_name  = "default_param5_value"
$PARAM3_NAME = "$PROJDEF"
[VALUE_FILE/parameter_set_name/value_file1_name]
param4_name  = "some_param4_value"
param5_name  = "some_param5_value"
$PARAM3_NAME = "some_param3_value"
[VALUE_FILE/parameter_set_name/value_file2_name]
param4_name  = "other_param4_value"
param5_name  = "other_param5_value"
$PARAM3_NAME = "other_param3_value"
[JOB/job1_name]
param6_name = "param6_value"
param7_name = "param7_value"
[JOB/job2_name]
param7_name = "param8_value"

The connection_definition/odbcConnectionDefinition.ini file format The connection_definition/odbcConnectionDefinition.ini file has the following content. Create a separate [Shortcut_Name] secion for each connection.

[<Shortcut_Name>]
Type=<connection_type>
Connection_String=<connection_string>
Server_Name=<server_name>
Database_Name=<database_name>
Schema_Name=<schema_name>
User_Name=<user_name>

• Shortcut_Name: The name of the connection or data server that is used by the data integration tool.
• connection_type: The type of data source.
• connection_string: A JDBC connection string or any identification of the database such as the system ID (SID) or the host name.
• server_name: The value depends on the type of data source:
  • Db2, Microsoft SQL Server, Netezza Performance Server, SAP ASE (formerly Sybase), or Teradata: The server name.
  • FTP: The hostname.
  • Oracle and other databases: The value is ignored.
• database_name: The value depends on the type of data source:
  • Oracle: The global database name.
  • Db2, Microsoft SQL Server, Netezza Performance Server, SAP ASE (formerly Sybase), Teradata, and other databases: The name of the default database.
  • user_name: The name of the user that logs in to the database.

Add a new line at the end of the parameters for each section.

The datastageComponentOverrides.csv file format The datastageComponentOverrides.csv file has the following content:

"Full path to Stage";"Input Link name";"Input Column name";"Output Link name";"Output Column name";"Edge Type (DIRECT | FILTER)";"Description (optional)"
"manual_mapping_job/Generic_3";"DSLink2";"a";"DSLink5";"b";"DIRECT";""

The path to the stage is in the format Job/[Shared and Local containers optional]/Stage.

Upload the .zip file to the project

To use the .zip file in the metadata import, you must add it to the project where you create the metadata import.

1. In the project, click Import assets.
2. In the Local file section, click Data asset.
3. Add the .zip file with DataStage project.

When you create the metadata import, you will be able to select this file in the Add inputs from file step.

Configure the import to use only the external input

If you want to import metadata only from the provided external input, and not directly from the connected DataStage for Cloud Pak for Data instance, complete these steps:

1. Add the .zip file in the Add inputs from file section and click Next.
2. Expand the Lineage import phases list, and disable the Transformations extraction phase.