Talend lineage configuration

Connect to Talend to scan metadata and display it on lineage. To import lineage metadata, create a data source definition and the metadata import job.

Supported versions

  • Talend Studio 8.0

Processed metadata

The following Talend metadata is processed and displayed on lineage:

  • Projects, jobs, and subjobs
  • Connectors
  • Components
  • Expressions
  • SQL overrides

The following Talend components are processed:

  • Input/Output: tFileInputExcel, tFileOutputExcel, FileInputDelimited, tFileOutputDelimited, tFileInputRaw, tFileOutputRaw, tFileInputPositional, tFileOutputPositional, tS3Put
  • Database Connection/Input/Output/Row: tTeradataConnection/tTeradataInput/tTeradataOutput/tTeradataRow, tOracleConnection/tOracleInput/tOracleOutput/tOracleRow, tMSSqlConnection/tMSSqlInput/tMSSqlOutput/tMSSqlRow, tSybaseConnection/tSybaseInput/tSybaseOutput/tSybaseRow, tHiveConnection/tHiveInput/tHiveOutput/tHiveRow, tNetezzaConnection/tNetezzaInput/tNetezzaOutput/tNetezzaRow, tDB2Connection/tDB2Input/tDB2Output/tDB2Row, tPostgresqlConnection/tPostgresqlInput/tPostgresqlOutput/tPostgresqlRow, tSQLiteConnection/tSQLiteInput/tSQLiteOutput/tSQLiteRow, tMysqlConnection/tMysqlInput/tMysqlOutput/tMysqlRow, tSnowflakeConnection/tSnowflakeInput/tSnowflakeOutput/tSnowflakeRow/tSnowflakeOutputBulkExec, tRedshiftConnection/tRedshiftInput/tRedshiftOutput/tRedshiftRow/tRedshiftBulkExec, tDBRow, tGreenplumInput/tGreenplumOutput, tBigQueryInput/tBigQueryOutput, tELTOracleInput/tELTOracleOutput
  • Transformation: tPrejob, tPostjob, tHMap, tSampleRow, tFilterRow, tSortRow, tExternalSortRow, tUniqRow, tReplace, tReplicate, tConvertType, tUnite, tFilterColumns, tAggregateRow, tSplitRow, tJoin, tMap, tFixedFlowInput, tRowGenerator, tExtractJSONFields, tELTOracleMap
  • Orchestration: tRunJob, tContextLoad, tDummy
  • Other: tRESTClient, tBufferInput, tBufferOutput, tLogRow, tFileList

Limitations

The following limitations apply:

  • Dynamic schemas are not processed.

Creating a metadata import asset

Data source connection

You can import lineage metadata from Talend by providing external files only. You don't connect directly to the Talend system. It means that you need to provide a data source definition only. You can create it before you start creating a metadata import asset, or you can do it directly in the metadata import wizard.

After you select a data source definition, the connection is created automatically. You can then proceed to adding external inputs, which are required.

Data source definition

Select Talend Data as the data source type.

External inputs

External inputs are required as this is the only way to import metadata.

You can add the Talend jobs in the following two export formats:

  • Individual Talend jobs exported in the XML format (.item files). These files are analyzed independently, and direct relationships are not created between job or joblet components in different jobs.
  • Talend jobs exported in the XML format and compressed to one archive file (.zip file). The entire archive, including all jobs and joblets it contains, is analyzed as a single task, and relationships between various jobs are created. For example, when one job invokes another job through a tRunJob component, and the results from the invoked job are propagated back to the calling job, a relationship is established. In this scenario, data lineage is created between the output of the invoked job and the tRunJob component.

The export file must follow the same directory and file structure that Talend Studio generates.

You can also add the following optional files:

  • context - This directory contains the context files. It is required only when contexts are used in Talend jobs.
  • contextReplacement.txt - This file contains manual context values. Use this file to override context values as defined in Talend Context Configuration.
  • talendExpressionOverrides.csv - This file contains manual expression overrides. Use this file to override Java expressions in Talend jobs.

The external input file (a directory compressed to the .zip format) can have the following structure:

All Talend .zip and .item file exports
context (directory)
contextReplacement.txt 
talendExpressionOverrides.csv

When you export a compressed .zip file that is generated in Talend Studio, you can use it as external input. However, only the following files are processed:

  • process/
    • optional folder hierarchy
      • *.item
      • *.properties
  • maps/
    • optional folder hierarchy
      • *.xml
  • joblets/
    • optional folder hierarchy
      • *.item
      • *.properties
  • context/
    • *.item
    • *.properties

Configuring context variables

When contexts are used in Talend jobs, you must add values to context variables so that they are processed correctly. All context files are added by using external inputs. They must be added to the context folder.

The context variables are processed in the following order:

  • The static values are loaded from the .item files.
  • Implicit context values are processed.
  • Manually defined values in the contextReplacement.txt file are processed.

The manually defined values override all previously loaded values for the same variables. If only static values are used and no values are found for some variables, empty values are replaced with the context.variableName string.

Using implicit context load (tContextLoad)

If the Talend job uses an implicit tContextLoad, the context can be loaded either from a file or from a database.

File
Copy the file from Talend to the context folder in the external input .zip file. You can find the implicit context in Talend in Project Settings > Job settings > Implicit context load. The ImplicitContext file is in /inputs/talend/{connectionName}/context/ folder. This example shows how content of the file is structured:
<elementParameter field="FILE" name="IMPLICIT_TCONTEXTLOAD_FILE" value=""{path to the context}/contexts/onlyImplicit.txt""/> <elementParameter field="TEXT" name="FIELDSEPARATOR" value=""=>""/>

The IMPLICIT_TCONTEXTLOAD_FILE parameter contains the path to the context file. The structure of the context file consists of a key and value pairs, which are separated by the => delimiter, for example:

first=>impl1
second=>impl2
path=>pathino

Do not modify the content of the context file. Copy it to the context folder of the external input file.

Database
The context values can't be loaded from the database. Add values manually in the contextReplacement.txt file.

Adding values manually in the contextReplacement.txt file

Use the contextReplacement.txt file to add context values manually. Structure the content of the file in the following format:

[jobName/testing]
key=value
key2=value2

[jobName/production]
key=value5
key2=value

[jobName2/testing]
key=value
key2=value2
key3=58

Learn more