Data Lineage Tools Overview

Data lineage tools are like detectives that help data professionals quickly sort through the tangled webs of interdependencies that make up the modern data stack.

Whether you’re a data scientist, data engineer, or business analyst, keeping track of your data’s origin, transformation, and movement is crucial for maintaining transparency, enforcing data governance, and ensuring data quality.

Just like Sherlock Holmes relied on his trusty magnifying glass to uncover hidden details, you too need powerful data lineage capabilities to get a better understanding of your data environment.

The first instinct of many early stage companies or budget strapped data teams is to turn to open source data lineage tools. While there are several affordable tools that we evaluate and compare below, what you will see is that their implementation and maintenance is anything but “elementary.”

Depending on your requirements (more on that in the next section), it is prudent to evaluate open source data lineage tools alongside the several other types of platforms that provide data lineage capabilities. You may even already have access to one of them.

Data lineage tools help you uncover and map the complex relationships between data elements across various sources, systems, and processes. By understanding your data’s lineage, you can ensure data accuracy, enhance data governance, meet regulatory compliance requirements, and improve decision-making.

This guide will serve as a handy resource to help you navigate the various open source data lineage tools out there, along with discussion on the pros and cons of open source, and how to select the right tool for your specific needs.

Pros and Cons of Open Source Data Lineage Tools

The choice between open source and proprietary is a matter of assessing your organization’s specific needs, resources, and future goals. We’ve written extensively on how to analyze the build vs buy decision across the modern data stack, but it’s important to speak specifically about open source data lineage options.

The reality is data lineage is incredibly complex to build as well as maintain, and as a result there aren’t many strong open source options that will address most organizations’ needs. There are a few reasons for this, but it’s mainly because data lineage involves so many layers–system, code, users– that are constantly evolving.

We know this because Monte Carlo was (and still is) on the forefront of developing data lineage for the modern data stack as one of the five pillars of data observability. We’ve been open about this work, providing very technical deep dives into how we developed column-level lineage and even some of our failures in building Spark data lineage for data lakes.

Another consideration for these tools are the integration points across the data pipeline. A question to ask is how will these data lineage source solutions account for dbt models, Fivetran connectors, and their relationship to tables? If our organization decided to pack up and migrate from one table format or data warehouse to another, would all the work we put into maintaining our open source data lineage solution for naught?

Finally, it’s important to consider first principles. Why do you need data lineage? Is it mainly a governance use case, a discovery/self-service use case, data quality use case, or all of the above?

For light data governance use cases you may already be on platforms such as Databricks or Google BigQuery which provide data catalog options that will suffice such as Unity Catalog and Dataplex respectively.

However, if there are any data quality requirements or desired data lineage use cases then you will want to consider a data observability platform. This is because data lineage, incident detection, and resolution capabilities need to be within a unified platform to enjoy accelerated resolution, incident triage, and other benefits.

Open Source Data Lineage Tools

OpenMetadata

OpenMetadata is an open-source data lineage tool with several stand out features:

• Column-level Lineage: Data transformations and dependencies can be traced down to the individual column level, enabling an incredibly granular view of data lineage.
• Query Filtering: You can isolate and focus on specific segments of data lineage using filters, facilitating better analysis and understanding.
• No-code Editor: It’s possible to augment the lineage captured from machine metadata with a drag and drop no-code editor. Tables, pipelines, and dashboards can be manually added, modified or removed on the lineage graph for a richer and more detailed understanding of the provenance of data.
• dbt Integration: This integration unveils the models used to generate tables. While data lineage broadly indicates where a table’s data came from, the dbt model provides more specific details about the transformations involved.

OpenLineage + Marquez

Technically, OpenLineage isn’t a tool. Rather it’s an open standard for metadata and data lineage collection. The actual collecting, aggregating and visualizing of metadata to construct data lineage is done by any tool that adheres to this standard. OpenLineage’s docs reference the open-source tool Marquez to do this.

Egeria

Egeria provides open APIs, event formats, types, and integration logic enabling metadata exchange and governance in organizations, useful for managing data lineage.

Apache Atlas

Apache Atlas is an open-source metadata management and governance tool that can also be used to track and manage data lineage. Atlas’ UI allows you to view the lineage of data as it moves through various processes and there is a set of REST APIs that allow you to access and update data lineage information.

Spline

Spline was originally designed for Apache Spark but the project has expanded to accommodate data lineage for other data sources as well. Spline has APIs for both collecting and querying data lineage data, supports OpenLineage integration, and provides a web UI to display stored lineage information.

Conclusion

In conclusion, open source data lineage tools can be an evaluation option for early-stage companies looking to maintain transparency and ensure data quality. However, they should be evaluated alongside other data lineage solutions such as data catalogs and data observability solutions depending on your desired use cases.