What is data lineage?

Data lineage tools provide a record of data throughout its lifecycle, including source information and any data transformations that have been applied during any ETL or ELT processes.

This type of documentation enables users to observe and trace different touchpoints along the data journey, allowing organizations to validate for accuracy and consistency. This is a critical capability to ensure data quality within an organization. It is commonly used to gain context about historical processes as well as trace errors back to the root cause.

Data lineage, data provenance and data governance are closely related terms, which layer into one another. Together, they ensure that an organization can maintain data quality and data security over time.

Data governance creates structure within organizations to manage data assets by defining data owners, business terms, rules, policies, and processes throughout the data lifecycle. Data lineage solutions help data governance teams ensure data complies to these standards, providing visibility into how data changes within the pipeline. Data provenance is typically used in the context of data lineage, but it specifically refers to the first instance of that data or its source.

Data lineage provides an audit trail for data at a very granular level; this type of detail is incredibly helpful for debugging any data errors, allowing data engineers to troubleshoot more effectively and identify resolutions more quickly. While the scope of data governance is broader than data lineage and data provenance, this aspect of data management is important in enforcing organizational standards.

Reliable data is essential to drive better decision-making and process improvement across all facets of business--from sales to human resources. However, this information is valuable only if stakeholders remain confident in its accuracy as insights are only as good as the quality of the data. Data lineage gives visibility into changes that may occur as a result of data migrations, system updates, errors and more, ensuring data integrity throughout its lifecycle.

Data lineage documents the relationship between enterprise data in various business and IT applications. These details can include:

• Where data is and how it’s stored in an environment, such as on premises, in a data warehouse or in a data lake.
• How the data can be used and who is responsible for updating, using and altering data. This also includes the roles and applications which are authorized to access specific segments of sensitive data, e.g. personally identifiable information (PII).
• Tracking data generated, uploaded and altered by business users and applications. For example, this can be the addition of contacts to a customer relationship management (CRM) system, or it can a data transformation, such as the removal of duplicate records.
• Data created and integrated from different parts of the organization, such as networking hardware and servers.

Metadata allows users of data lineage tools to fully understand how data flows through the data pipeline. Metadata is the “data about the data”, which includes various information about the data assets, such as the type, format, structure, author, date created, date modified and file size. Data lineage tools provide a full picture of the metadata to guide users as they determine how useful the data will be to them.

In recent years, the ways in which we store and leverage data has evolved with the evolution of big data. Companies are investing more in data science to drive decision-making and business outcomes. However, in order for them to construct a well-formed analysis, they’ll need to utilize data lineage tools and data catalogs for data discovery and data mapping exercises. While data lineage tools show the evolution of data over time via metadata, a data catalog uses the same information to create a searchable inventory of all data assets in an organization. Together, they enable data citizens to understand the importance of different data elements to a given outcome, which is foundational in the development of any machine learning algorithms.

Companies today have an increasing need for real-time insights, but those findings hinge on an understanding of the data and its journey throughout the pipeline. Some of the ways that teams can leverage end-to-end data lineage tools to improve workflows include:

Data modeling: To create visual representations of the different data elements and their corresponding linkages within an enterprise, companies must define the underlying data structures that support them. Data lineage helps to model these relationships, illustrating the different dependencies across the data ecosystem. Since data evolves over time, there are always new data sources emerging, new data integrations that need to be made, etc. As a result, the overall data model that businesses use to manage their data also needs to adapt the changing environment. Data lineage helps to accurately reflect these changes over time through data model diagrams, highlighting new or outdated connections or tables. This, in turn, helps analysts and data scientists facilitate valuable and timely analyses as they'll have a better understanding of the data sets.

Data migration: When moving data to a new storage system or onboarding new software, organizations use data migration to understand the locations and lifecycle of the data. Since data lineage provides a view of how this data has progressed through the organization, it assists teams in planning for these system migrations or upgrades, expediting the overall transition to the new storage environment. It also provides teams with the opportunity to clean up the data system, archiving or deleting old, irrelevant data; this, in turn, can improve overall performance of the data system reducing the amount of data that it needs to manage.

Compliance: Data lineage provides a compliance mechanism for auditing, improving risk management, and ensuring data is stored and processed in line with data governance policies and regulations. For example, in 2016, GDPR legislation was created to protect the personal data of people in the European Union and European Economic Area, giving individuals more control of their data. In the United States, individual states, like California, developed policies, such as the California Consumer Privacy Act (CCPA), which required businesses to inform consumers about the collection of their data. This type of legislation makes the storage and security of this data a top priority, and without data lineage tools, organizations would find noncompliance issues to be a time-consuming and expensive undertaking.

Impact Analysis: Data lineage tools can provide visibility into the impact of specific business changes, such as any downstream reporting. For example, if the name of a data element changes, data lineage can help leaders understand how many dashboard that might affect and subsequently how many users that access that reporting. It can also help assess the impact of data errors and the exposure across the organization. Data errors can occur for a myriad of reasons, which may erode trust in certain business intelligence reports or data sources, but data lineage tools can help teams trace them to the source, enabling data processing optimizations and communication to respective teams.