Mapping OpenLineage events
Use mapping rules for OpenLineage events to represent unsupported or custom technologies in lineage. Each rule consists of conditions to define when the rule is applied, and actions to define how the event is interpreted.
If your data flow includes unsupported technologies, your lineage might not show the complete flow of data. You can import OpenLineage events and use mappings to define how those technologies are recognized and shown in lineage. Mappings are rules that define how data in OpenLineage events is interpreted and represented in lineage.
When you create mappings for OpenLineage events, lineage is improved in the following ways:
- Custom jobs and datasets are displayed under the correct technology.
- Lineage paths correctly connect nodes rather than placeholders.
Mapping rule structure
Each mapping rule consists of two parts:
- Conditions, which define when to apply a rule, based on the namespace or prefix, and applicable scope.
- Actions, which define what happens when the rule is applied. Actions include assigning the technology, setting the asset hierarchy, and linking to the right data source definition so that assets are located under the correct system.
Mapping OpenLineage events overview
Mapping OpenLineage events involves the following process:
- Create data source definitions for systems that are referenced in the event.
- Create custom technology structure to define asset types and their hierarchy.
- Create a mapping rule where you define when the rule is processed and what actions are taken.
- Run a metadata import where you add the OpenLineage event as an input file.
- Review the imported data on lineage.