Teradata lineage configuration

Limitations

The following limitations apply:

• Assets that use dynamically executed code through the EXECUTE IMMEDIATE statement are not processed.
• The TPT, BTEQ, or SQL files that are wrapped in Linux Shell, Windows Batch, or any other programming language are not processed.

Data source connection

To connect to the data source from which you want to import lineage metadata, you need to select a data source definition and a connection. You can create them before you start creating the metadata import, or you can create them when you create and configure the metadata import asset.

Include and exclude lists

You can include or exclude databases. Each part is evaluated as a regular expression. If the assets which you add later in the data source match the conditions specified in the lists they might also be included or excluded. Example values:

• myDB/: all included databases and assets in myDB,
• myDB2/.*: all included databases and assets in in myDB2,
• myDB3/myDB4: Database myDB4 from myDB3,
• myDB5/myDB[1-5]: any database or another asset in myDB5 with a name that starts with myDB and ends with a digit between 1 and 5.

External inputs

It is possible to analyze the data flows defined in BTEQ and TPT scripts that operate on the configured database. You can provide external Teradata SQL scripts as inputs by packaging them into a .zip file. You can organize the contents of the .zip into subfolders that correspond to database names. After the file is processed, each script is automatically associated with the appropriate database based on its folder location.

The external input file must have the following format:

<bteq>
    <script_name.bteq>
    <database_name> - optional
        <script_name.bteq>
<tpt>
    <script_name.tpt>
<tpt_operators>
    <tpt_operator_file_name> OR <tpt_operators.zip>
<tpt_includes>
    <included_file_name>
<tptJobVariables.txt>
<tptIncludesPathReplacements.txt>
replace.csv
connectionsConfiguration.prm

Where:

BTEQ scripts:

• BTEQ scripts must be available on the filesystem as standalone .bteq (or equivalent) script files.
• Scripts must contain only BTEQ code. Wrappers such as Linux shell scripts, PowerShell scripts, or Windows batch files are not supported.
• If the scripts use parameter placeholders, a corresponding parameter‑mapping file must be present on the filesystem. Each entry in the file must follow the format: placeholderName=actualValue

TPT scripts:

• TPT scripts must also be available on the filesystem as standalone .tpt (or equivalent) script files.
• Scripts must contain only TPT code without embedding shell, PowerShell, or batch wrappers of any kind.

tpt_operators

Folder that contains custom operator scripts or operator templates defining producer and consumer operators, schemas, attribute blocks, and reusable job fragments. The folder can also include a .zip file that contains operator templates. Operator templates can reference code fragments located in the tpt_includes directory.

tpt_includes

Folder that contains files referenced in jobs through INCLUDE statements. These files typically provide schema definitions, operator blocks, shared SQL fragments, or configuration segments required for full script expansion.

tptJobVariables.txt

Text file that contains name–value mappings for TPT job variables that are used in scripts.

Accepted format placeholder=value. You cannot use the at sign (@) in the placeholder name and its value.

Example:DBName='WORKDB'.

tptIncludesPathReplacements.txt

File with key=value mappings used to adjust paths in INCLUDE directives. These mappings ensure that referenced script fragments are resolved relative to the tpt_includes directory.

replace.csv

File that contains placeholder replacements for scripts included in the .zip package. For more information about the format, see Placeholder replacements.

connectionsConfiguration.prm

Parameter file with connection definitions used by TPT scripts to reference connections through names rather than direct logon strings.

Advanced import options

Extract extended attributes

You can extract extended attributes like primary key, unique and referential integrity constraints of columns. By default these attributes are not extracted.

Performance profile

For selected data sources you can choose a performance profile. Depending on your current needs, the lineage metadata import might be faster or more complete. You can choose between the following profiles:

• Fast: Low time and memory consumption are the priorities in this profile. If your input is large, lineage might not be complete.

• Balanced: Both performance and lineage completness are important. It is a compromise bewteen the lineage completness and time and memory that is spent on lineage import.

• Complete: The completness for lineage is the priority in this profile. If your input is large, the lineage import might take a significant amount of resources and time.

• Custom profile: You can create your own performance profile by providing values for the following properties:

  • Dataflow Analysis Timeout Limit: Specifies the maximum estimated time (in seconds) after which the dataflow analysis of a single input is stopped. The time is checked when each node is added, or in some cases when edges are created. Therefore, in some cases, the timeout might slightly exceed the specified limit. If you set the value to 0, the analysis is not stopped. Example value: 60.
  • Dataflow Analysis Edge Limit: Specifies the maximum number of edges that are allowed for a single input during the dataflow analysis. If this limit is exceeded, all filter edges are removed and no more filter edges are added. If the limit is still exceeded even after that, the analysis is stopped and the input fails. To disable the limit, set the value to 0. Example value: 2500.

Transformation logic extraction

You can enable building transformation logic descriptions from SQL code in SQL scripts.