Data Pipeline Architecture: a Guide to Layers and Patterns

Data pipeline architecture defines how raw data is collected, then transformed into structured data, stored and delivered across an organization. It encompasses the systems and processes responsible for moving data from source systems to downstream analytical, operational or AI-driven applications.

The core of a data pipeline is three fundamental stages: data ingestion, data processing and data storage. Ecah of these stages plays a different role in ensuring that data is reliable, consistent and suitable for analysis and decision-making.

A modern data pipeline architectures is commonly organized into five interconnected layers:

•            Ingestion layer: this collects raw data from source systems like consumer-facing APIs, databases, IoT devices and SaaS applications.

•            Storage layer: these store data in centralized repositories, such as data lakes or data warehouses, to support scalable access and long-term retention.

•            Processing layer: this cleans, validates, transforms and enriches raw data so that it is consistent and ready for downstream use.

•            Consumption layer: this layer delivers curated data to business intelligence (BI) platforms, machine learning (ML) models, operational applications and other data consumers.

•            Monitoring and orchestration layer: this is where pipeline execution is coordinated, workflows are scheduled, data quality is monitored and the overall system health is tracked. This layer is critical to guarantee reliability and minimize data downtime.

If we look at a modern data pipeline architecture as it moves data through a sequence of interconnected stages, we would have:

Data sources → Ingestion → Processing/Transformation → Storage → Governance/Quality → Consumption

Each of those steps involves different technologies and strategies for working with data.

A data pipeline begins with one or more source systems that generate the data to be collected and processed. These sources can include transactional databases, Software as a Service (SaaS) applications like customer relationship management (CRM) or enterprise resource planning (ERP) systems.

The sources can also be direct user inputs like a web or mobile applications, or data directly from Internet of Things (IoT) devices and sensors. Application logs and service telemetry provide measures of system or hardware health and are fed into a data pipeline.

Often many kinds of data are combined. For instance an e-commerce organization might collect data from its order database, website clickstream, payment processing system and inventory management platform. Each source contributes a different type of information that can be integrated to support operational reporting, analytics and machine learning applications.

The data ingestion layer is responsible for collecting data from source systems and transporting it into the data pipeline for downstream processing. Depending on specific business requirements that data could be ingested either in batches at scheduled intervals or as a continuous stream of events.

Some of the most common technologies for data ingestion are Apache Kafka, Apache Pulsar, Apache Flink, Apache NiFi. Managed cloud services such as Confluent Cloud, Amazon Kinesis and Google Pub/Sub are also popular and provide significant benefits over self-managed technologies.

Organizations generally employ one of two ingestion models. Batch ingestion transfers data at predefined intervals, such as hourly, daily or weekly, making it suitable for workloads that do not require immediate data availability.

Streaming ingestion on the other hand, continuously captures and processes events as they occur, enabling low-latency data movement and near real-time analytics. As organizations increasingly rely on time-sensitive applications, event streaming has become a common architectural approach to reduce latency and support real-time data processing.

Raw data is rarely suitable for direct consumption by data systems or models. To make data ready for consumption, the processing and transformation layer prepares data for downstream applications by improving its quality, consistency and usability. Some of the most common processing tasks include cleaning and validating records, standardizing data formats, enriching datasets with additional context, joining data from multiple sources, applying business rules and computing derived metrics and aggregations.

Technologies such as Apache Spark, Apache Flink, dbt and Apache Beam help simplify how these transformations. Processing may occur while data is in transit through the pipeline in a stream processing architecture or after the data has been stored in a batch processing architecture. Choosing between these approaches depends on the specific application requirements for latency, scalability and computational complexity.

The storage layer provides centralized repositories for both raw and processed data. These repositories are designed to support reliable storage, efficient querying and long-term data management. Modern data architectures typically have one or more of the following storage models: data lake, data warehouse, lakehouse.

A data lake stores raw, semi-structured and unstructured data in its native format, making it well suited for large-scale data ingestion and exploratory analysis. Data lakes are commonly built on cloud object storage platforms such as Amazon S3, Azure Data Lake Storage, Google Cloud Storage or IBM watsonx.data®.

A data warehouse is optimized for analytical queries, business intelligence and reporting. Unlike data lakes, data warehouses typically contain curated, structured datasets that have been transformed and validated for consistent analysis. Some of the most common data warehouse platforms are Snowflake, Google BigQuery and Amazon Redshift.

A lakehouse combines the flexibility of a data lake with the performance and governance capabilities of a data warehouse. Lakehouse architectures typically rely on open table formats such as Apache Iceberg, Delta Lake or Apache Hudi to provide features including ACID transactions, schema evolution and efficient analytical querying.

Modern data pipeline architectures generally will incorporate governance and data quality capabilities that apply to every stage of the pipeline. Rather than existing as a single processing step, these capabilities provide oversight and control throughout the data lifecycle.

Key governance functions include features like schema management, data contracts, metadata cataloging, data lineage tracking, data quality validation and access control. Taken all together, these capabilities help ensure that data remains accurate, secure, discoverable and compliant with regulatory requirements like GDPR or CCPA. This can also include a data catalog, a centralized inventory of data assets across an organization.

Some of the most popular governance platforms include Apache Atlas, DataHub and OpenMetadata.

The consumption layer is where processed data is delivered to end users, applications or services that generate business value from it. Depending on organizational needs that data might support reporting, decision-making at an operational level, predictive analytics, AI applications or a mixture of all of these.

Applications and systems that consume data from the consumption layer are, for example, business intelligence dashboards, reporting systems, machine learning models, real-time applications, operational analytics platforms and AI agents or retrieval systems. Some of the most widely used analytics and visualization platforms are Tableau, Power BI and Looker, which enable users to explore data and create dashboards that allow end users to derive actionable insights from data.

As an example, a modern streaming architecture could consist of the following components:

•            Web application: Generates customer events

•            Kafka on Confluent Cloud: Captures and distributes the event stream

•            Apache Flink: Processes and enriches the events in real time

•            Apache Iceberg: Stores curated analytical datasets

•            Data Warehouse or lakehouse engine: Executes analytical queries

•            Dashboards and AI applications: Present insights or make predictions based on the processed data

In this architecture the data is processed as it is generated rather than waiting for scheduled batch jobs that will process it. When a customer clicks on an item in a website, the application generates an event that is published to a Kafka topic. Apache Flink then consumes that topic event stream and enriches each event by adding customer profile information and other contextual data.

That transformed data is then written to Apache Iceberg tables which are stored in object storage where it becomes available for analytical queries. Analysts can query the data by using a data warehouse or lakehouse query engine while machine learning systems simultaneously consume the same event stream and support real-time inference and decision-making.

This type of event-driven architecture is increasingly prevalent because it helps organizations support operational workloads like fraud detection and personalized recommendations as well as analytical workloads like business intelligence and historical reporting. Both of these can be supported from a single stream of data.

The choice of processing architecture will depend on business requirements, which can include data volume, latency, scalability and processing complexity. The most common processing patterns include batch processing, stream processing and the Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) data integration models.

Batch processing executes transformations on large collections of data at scheduled intervals. That could be as commonly run as an hourly candence, or as infrequently as a weekly processing. This approach is well suited for workloads that do not require immediate results, including financial reporting, historical analysis and large-scale data preparation. Common technologies to do this include Apache Spark and data transformation frameworks such as Databricks.

Stream processing (sometimes called real-time processing) continuously processes data as events are generated, allowing organizations to respond with minimal latency. This approach is commonly used for applications such as fraud detection, IoT telemetry, operational monitoring and personalized customer experiences. A platform like Apache Kafka provides the event streaming, while processing frameworks such as Apache Flink analyze and transform the incoming data streams in real time.

Organizations must also determine whether to adopt an ETL or ELT approach. In ETL architectures, data is transformed before it is loaded into the destination system. In contrast, ELT architectures first load raw data into a centralized storage platform and perform transformations afterward. Because modern cloud data warehouses and data lakehouses provide substantial computational resources, ELT has become the predominant approach for many contemporary data platforms.

Pipeline design is usually determined by both the scale and size of the data and by what the data is needed for. Strategies for architectures have evolved to fit a wide variety of needs and requirements. One common strategy is to create service level agreements (SLAs) that define the expected levels of availability, latency, freshness, reliability and data quality that the pipeline must achieve.

In many organizations or systems the architectural decisions are driven as much by SLA requirements as by the volume or type of data being processed. For instance, an SLA might specify availability and fault tolerance. These would specify uptime targets like “99.9% availability” for a pipeline. SLAs can also specify scalability targets. A data engineer may specify that the pipeline will be able to process one million events per minute in order to ensure that downstream systems remain operational.

Another type requirement may focus on data quality, establishing a maximum percentage of missing records or required schema validation. Understanding the SLAs can help narrow what data products will satisfy the requirements of the agreements.

A Lambda architecture is a big data processing pattern designed to support both high-throughput historical analysis and low-latency real-time analytics. It achieves this by dividing data processing into two parallel pipelines—a batch layer and a speed (or streaming) layer—that process the same incoming data simultaneously.

The batch layer computes comprehensive, highly accurate views of the data by using the complete historical dataset. The speed layer, meanwhile, processes newly arriving events as they occur, providing observability and immediate insights before they are incorporated into the batch results.

This dual-path approach allows organizations to balance the tradeoff between data accuracy and processing latency. The speed layer delivers near-real-time updates for data visualization, alerts and time-sensitive applications, while the batch layer periodically recomputes results from the authoritative historical record to correct late-arriving data, processing errors or incomplete information.

A serving layer then combines the outputs of both processing paths, presenting users with a unified view that reflects both the latest events and the most accurate historical data.

Lambda architecture emerged as a solution for early large-scale data processing systems in which batch frameworks offered robust, fault-tolerant computation but could not satisfy real-time analytics requirements. Although the architecture remains useful for some workloads, it introduces additional operational complexity because organizations must develop, maintain and synchronize two independent processing pipelines.

As data platforms have increased in scale and complexity, organizations have adopted architectural patterns that promote consistency, governance and maintainability. One of the most widely used approaches is what is known as a medallion architecture.

A medallion architecture organizes data into progressively refined layers as it moves through a data lake or lakehouse. Rather than transforming data in a single step this architecture applies successive stages of validation, enrichment and aggregation, which allows each layer to serve a distinct purpose and preserves the lineage and improves the data quality.

The architecture is typically divided into three layers: Bronze, Silver, Gold.

Bronze layer (raw data)

The Bronze layer contains data in its original form as it is ingested from source systems. Data is stored with minimal or no transformation to preserve its fidelity and maintain a complete historical record. Whenever possible, the original schema is retained, allowing the Bronze layer to serve as the authoritative system of record.

For example, website clickstream events might be written directly from a Kafka topic into Bronze tables while preserving every field generated by the source application.

The primary objectives of the Bronze layer are to preserve the original source data, support future reprocessing as business requirements evolve and maintain a complete audit trail.

Silver layer (cleaned and enriched data)

The Silver layer contains data that has been validated, standardized and enriched to improve its quality and consistency. Typical transformations include removing duplicate records, correcting malformed data, validating schemas, standardizing formats and units of measure, joining related datasets and applying business rules.

For example, clickstream events might be enriched with customer profile information, validated against predefined schemas and standardized before being stored as Silver tables.

The primary objectives of the Silver layer are to improve data quality, create reusable datasets and establish consistent business definitions that can be shared across multiple applications and analytical workloads.

Gold layer (business-ready data)

The Gold layer contains curated datasets that have been optimized for business consumption and analytical performance. These datasets typically include aggregations, key performance indicator (KPI) calculations, dimensional models, machine learning features and executive reporting tables.

For example, Gold tables might contain daily revenue by region, customer lifetime value metrics, sales performance summaries or comprehensive customer 360 views.

The primary objectives of the Gold layer are to provide trusted business metrics, support dashboards and reporting, and supply high-quality data for advanced analytics and AI applications.