Creating an export asset and managing jobs

Create an export asset to generate a file with lineage metadata, and optionally upload it to the target system automatically after the file is generated. You can later edit the scope and schedule of the exports.

Prerequisites

Collibra

Before you start creating a data lineage export asset to Collibra, make sure that the following prerequisite tasks are completed:

  • A data source definition for the target system is created.
  • Collibra required configuration is complete.
  • The target Collibra instance is created and configured.
  • The Collibra mapping is created.
  • A connection to the target system is defined in the project where you want to create the export asset.

For details about Collibra configuration, instances and mappings, see Configuring Collibra export.

OpenLineage

When you create a data lineage export asset to the OpenLineage format, no prerequisites are required.

Creating a data lineage export asset

To create a data lineage export asset and a job, complete these steps:

  1. In the project, go to the Assets tab, and click New asset > Export lineage metadata.
  2. Specify a name, and optionally a description and tags.
  3. Select the export type.
  4. Collibra only: Select the data source definition that you used to create the target Collibra instance.
  5. Collibra only: If you want to upload the lineage to the target system immediately after it is generated, set Upload data to the export target to on.
    1. Collibra only: Select the connection to Collibra that you created as one of the prerequisite tasks.
  6. Select the data source definition of the data source from which you want to export the lineage.
  7. In the Export goals section, select the scope of lineage to export. See Export goals.
  8. Collibra only: Select one synchronization mode to process assets, which no longer exist in the target system. See Synchronization mode.
  9. Select the asset levels for which lineage relationships are exported. See Lineage aggregation level.
  10. Decide when to run the export job. You can run it immediately or schedule the job to run on a specific time.
  11. Review the summary of the export asset and save the changes.

When the job is finished, an export asset is created and it is ready for downloading.

Managing existing export assets

You can view the existing export assets in the project where you created them, in the Assets tab. You can download the exported .zip file at any time.

Editing the export asset

When you edit the export asset, you cannot modify the export target. After you save the changes, you need to manually run a new job to generate an export file with modified content.

Deleting the export asset

When you delete the export asset, the associated job is also deleted. The data that is already uploaded in Collibra is not affected.

Managing existing export jobs

You can view the existing export job in the project where you created the export asset, in the Jobs tab. Each job run can have one of the following statuses:

Starting
The job is initialized.
Running
Processing is in progress.
Completed
The data was successfully exported and the exported file is ready for downloading. If you choose to upload the data to Collibra immediately after the file is generated, the job is completed when the data is uploaded to Collibra.
Failed
Processing stopped unexpectedly.

The jobs are associated with the data lineage export asset and they are generated when you run the job for the export asset manually, or when the job is scheduled to run at a specific time. You can edit the name of the job and it is still associated with the same data lineage export asset.

Log files

You can find the log files for each job run in the job run details page. You access this page by clicking a specific run of a job in the job details page, or you can click the View run metrics button in the export asset details page to see the latest job run details.