Data Pipeline Architecture Explained: 6 Diagrams And Best Practices

What is data pipeline architecture?

Data pipeline architecture is the process of designing how data is surfaced from its source system to the consumption layer.

This frequently involves, in some order, extraction (from a source system), transformation (where data is combined with other data and put into the desired format), and loading (into storage where it can be accessed). This is commonly abbreviated and referred to as an ETL or ELT pipeline.

It’s important to understand most data pipelines aren’t a linear movement of data from source A to target B, but rather consist of a series of highly complex and interdependent processes. For example, data may be extracted from multiple sources, reorganized, and joined at different times before ultimately being served to its final destination.

Why is data pipeline architecture important?

For data engineers, good data pipeline architecture is critical to solving the 5 v’s posed by big data: volume, velocity, veracity, variety, and value. A well designed pipeline will meet use case requirements while being efficient from a maintenance and cost perspective.

6 Data pipeline architecture designs and their evolution

ETL

The Hadoop era, roughly 2011 to 2017, arguably ushered in big data processing capabilities to mainstream organizations.

Data then, and even today for some organizations, was primarily hosted in on-premises databases with non-scalable storage. Despite Hadoop’s parallel and distributed processing, compute was a limited resource as well.

As a result, data engineers spent considerable time modeling data and optimizing queries to fit within these constraints. Data pipeline architecture typically consisted of hardcoded pipelines that cleaned, normalized, and transformed the data prior to loading into a database using an ETL pattern.

Some organizations choose to still leverage an ETL pattern in the cloud, particularly for production pipelines where data contracts can help reduce data downtime.

ELT

The modern data stack era, roughly 2017 to present data, saw the widespread adoption of cloud computing and modern data repositories that decoupled storage from compute such as data warehouses, data lakes, and data lakehouses. The term modern data stack refers to the multiple modular SaaS solutions that comprise the data platform and pipeline (more on those later).

With cost and physical compute/storage limitations largely lifted, data engineers started to optimize data pipeline architecture for speed and agility. Data could now be extracted and loaded prior to being transformed for its ultimate use. This scale and flexibility of the cloud and an ELT design pattern unlocked additional valuable use cases such as more widespread analytics, experimentation, and machine learning applications.

Streaming

While these batch data pipelines were ideal for analytical use cases, there has been an increased need for near-real time data. Streaming data pipeline architectures are typically run in parallel to modern data stack pipelines and used mainly for data science or machine learning use cases. Apache Kafka is the most common tool and it comes in many flavors (Confluent Cloud, AWS MSK, self-hosted, etc).

The pattern can be described as stream, collect, process, store, and analyze. The most common architecture involves sending raw data directly to the application, but sometimes Flink is used as part of the transformation process. Another common pattern is to also land this data in a lake/lakehouse, eventually making its way to a warehouse for analytical use cases.

The direct source to application streaming data pipeline architecture can present challenges such as the inability to fully validate data quality or model the data. However, the value and benefits of near-real time data are hard to deny.

Data pipeline architectures are constantly being reinvented. Two emerging data pipeline architectures include zero ETL and data sharing.

Zero ETL

Zero ETL is a bit of a misnomer. These pipelines differ from traditional ELT pipelines by doing the data cleaning and normalization prior to load. In many cases the data stays in a data lake and is queried from there versus moving to the data warehouse.

At the moment, this tight integration is possible because most zero-ETL architectures require both the transactional database and data warehouse to be from the same cloud provider. Some examples include AWS ( Aurora to Redshift), GCP ( BigTable to BigQuery) and Snowflake ( Unistore).

This trend is competing with another data pipeline architecture undercurrent where some organizations are considering decoupling their storage and compute at the vendor level with companies like Tabular.

Data Sharing

No copy data sharing is another emerging data pipeline architecture being pioneered by Snowflake ( Secure Data Sharing) and Databricks ( Delta Sharing). Unlike ZeroETL, this process actually doesn’t involve ETL and instead provides expanded access (permissions) to the data where it’s stored.

Remember, these aren’t a binary choice. Most organizations deploy some or all of these data pipeline architectures.

RAG or Vector Retrieval (unstructured data context for AI)

We’ve written about retrieval augmented generation or RAG extensively, but the main idea behind these pipelines is to provide the context an AI agent needs to complete its task successfully. A customer success agent might need to reference both structured as well as unstructured data.

For example, recent customer transactions would be structured data typically stored in a warehouse or lakehouse using the modern data stack or ELT pattern. The agent queries the information and gets what it needs (easier said than done of course). But what if the agent needs to reference some unstructured data like a refund policy PDF file?
In this case, a RAG or vector retrieval pipeline is often used. The meaning is extracted from the unstructured file in smaller chunks so that it can easily fit the context window. Those chunks “The refund policy is 60 days,” are then transformed into vectors like .56, .101, .-3, etc that represent the semantic meaning of the chunk. When the agent need to retrieve this context it understands and retrieves the relevant vector, unpacks the semantic meaning which is then injected into the prompt to inform the final output.

As you can imagine, issues occur in both the structured and unstructured portions of these pipelines, which is why you need data + AI observability. And even if the data is correct, the model can still produce an unfit output! Agent observability is the means to observe and monitor that behavior and the underlying AI configuration changes that may have led to a degradation in performance.

Common data pipeline solutions within the modern data stack

The modern data stack universe has a data warehouse, lake, or lakehouse as its center of gravity with numerous, modular solutions that integrate with one another orbiting around it.

If you zoom out to look across the entire data platform solution landscape it can get confusing and messy really quickly.

That’s why it can be more helpful to consider these solutions by the role they play within modern data pipeline architecture. This is frequently referred to as a 5 or 7 layer (depending on who you ask) data stack like in the image below.

Here are some of the most common solutions that are involved in modern data pipelines and the role they play.

Data storage and processing

Data ingestion

Batch data ingestion solutions include:

Data streaming ingestion solutions include:

Data orchestration

Data transformation

Business intelligence and analytics

Data observability

Data catalog

Some organizations choose to implement data catalog solutions for data governance and compliance use cases. Data catalogs leverage the metadata generated through a modern data platform to provide descriptive information on critical data assets such as tables, key metrics, and more. Think of an automatically updating encyclopedia for your data platform.

Access management

You want the data in your modern data platform to be highly available only to those who have a need to access it. Access management solutions have become more critical as a means to protect sensitive information and PII as well as to avoid fines from data regulations such as GDPR or CCPA.