Data Pipeline Architecture: a Guide to Layers and Patterns

Data pipeline architecture defines the framework through which raw data is collected, transformed into structured data, stored and delivered across an organization. It encompasses the systems and processes responsible for moving data from source systems to downstream analytical, operational or AI-driven applications.

At its core, a data pipeline typically consists of three fundamental stages: data ingestion, data processing and data storage. Together, these stages ensure that data is reliable, consistent and suitable for analysis and decision-making.

Modern data pipeline architectures are commonly organized into five interconnected layers:

•            Ingestion layer: Collects raw data from source systems such as APIs, databases, IoT devices and SaaS applications.

•            Storage layer: Stores data in centralized repositories, such as data lakes or data warehouses, to support scalable access and long-term retention.

•            Processing layer: Cleans, validates, transforms and enriches raw data so that it is consistent and ready for downstream use.

•            Consumption layer: Delivers curated data to business intelligence (BI) platforms, machine learning (ML) models, operational applications and other data consumers.

•            Monitoring and orchestration layer: Coordinates pipeline execution, schedules workflows, monitors data quality and tracks system health to improve reliability and minimize data downtime.

If we look at a modern data pipeline architecture as it moves data through a sequence of interconnected stages, we would have:

Data sources → Ingestion → Processing/Transformation → Storage → Governance/Quality → Consumption

Each of those steps involves different technologies and strategies for working with data.

A data pipeline begins with one or more source systems that generate the data to be collected and processed. These sources can include transactional databases, Software as a Service (SaaS) applications like customer relationship management (CRM) or enterprise resource planning (ERP) systems.

The sources can also be direct user inputs like a web or mobile applications, or data directly from Internet of Things (IoT) devices and sensors. Application logs and service telemetry provide measures of system or hardware health and are fed into a data pipeline.

Often many kinds of data are combined. For instance an e-commerce organization might collect data from its order database, website clickstream, payment processing system and inventory management platform. Each source contributes a different type of information that can be integrated to support operational reporting, analytics and machine learning applications.

The data ingestion layer is responsible for collecting data from source systems and transporting it into the data pipeline for downstream processing. Depending on business requirements, data might be ingested either in batches at scheduled intervals or as a continuous stream of events.

Common technologies used for data ingestion include Apache Kafka, Apache Pulsar, Apache Flink, Apache NiFi and managed cloud services such as Confluent Cloud, Amazon Kinesis and Google Pub/Sub.

Organizations generally employ one of two ingestion models. Batch ingestion transfers data at predefined intervals, such as hourly, daily or weekly, making it suitable for workloads that do not require immediate data availability.

In contrast, streaming ingestion continuously captures and processes events as they occur, enabling low-latency data movement and near real-time analytics. As organizations increasingly rely on time-sensitive applications, event streaming has become a common architectural approach to reduce latency and support real-time data processing.

Raw data is rarely suitable for direct consumption by data systems or models. To make this data ready, the processing and transformation layer prepares data for downstream applications by improving its quality, consistency and usability. Common processing tasks include cleaning and validating records, standardizing data formats, enriching datasets with additional context, joining data from multiple sources, applying business rules and computing derived metrics and aggregations.

Organizations commonly use technologies such as Apache Spark, Apache Flink, dbt and Apache Beam to perform these transformations. Processing might occur while data is in transit through the pipeline by using stream processing, or after the data has been stored by using batch processing. The choice between these approaches depends on the specific application requirements for latency, scalability and computational complexity.

The storage layer provides centralized repositories for both raw and processed data. These repositories are designed to support reliable storage, efficient querying and long-term data management. Modern data architectures typically employ one or more of the following storage models.

A data lake stores raw, semi-structured and unstructured data in its native format, making it well suited for large-scale data ingestion and exploratory analysis. Data lakes are commonly built on cloud object storage platforms such as Amazon S3, Azure Data Lake Storage, Google Cloud Storage or IBM watsonx.data®.

A data warehouse is optimized for analytical queries, business intelligence and reporting. Unlike data lakes, data warehouses typically contain curated, structured datasets that have been transformed and validated for consistent analysis. Common data warehouse platforms include Snowflake, Google BigQuery and Amazon Redshift.

A lakehouse combines the flexibility of a data lake with the performance and governance capabilities of a data warehouse. Lakehouse architectures typically rely on open table formats such as Apache Iceberg, Delta Lake or Apache Hudi to provide features including ACID transactions, schema evolution and efficient analytical querying.

Modern data pipeline architectures generally incorporate governance and data quality capabilities that span every stage of the pipeline. Rather than existing as a single processing step, these capabilities provide oversight and control throughout the data lifecycle.

Key governance functions include features like schema management, data contracts, metadata cataloging, data lineage tracking, data quality validation and access control. Together, these capabilities help ensure that data remains accurate, secure, discoverable and compliant with regulatory requirements like GDPR. This can also include a data catalog, a centralized inventory of data assets across an organization.

Popular governance platforms include Apache Atlas, DataHub and OpenMetadata.

The consumption layer delivers processed data to the users, applications and services that generate business value from it. Depending on organizational needs, data might support reporting, operational decision-making, predictive analytics, AI applications or a mixture of all of these and more.

Common consumers include business intelligence dashboards, reporting systems, machine learning models, real-time applications, operational analytics platforms and AI agents or retrieval systems. Widely used analytics and visualization platforms include Tableau, Power BI and Looker, which enable users to explore data and create dashboards that allow end users to derive actionable insights from data.

As an example, a modern streaming architecture could consist of the following components:

•            Web application: Generates customer events

•            Kafka on Confluent Cloud: Captures and distributes the event stream

•            Apache Flink: Processes and enriches the events in real time

•            Apache Iceberg: Stores curated analytical datasets

•            Data Warehouse or lakehouse engine: Executes analytical queries

•            Dashboards and AI applications: Present insights or make predictions based on the processed data

In this architecture, data is processed as it is generated rather than waiting for scheduled batch jobs. When a customer clicks on a website, the application generates an event that is published to the Kafka topic. Apache Flink consumes that topic event stream and enriches each event with customer profile information and other contextual data.

The transformed data is then written to Apache Iceberg tables stored in object storage, where it becomes available for analytical queries. Analysts can query the data within seconds by using a data warehouse or lakehouse query engine, while machine learning systems simultaneously consume the same event stream to support real-time inference and decision-making.

This event-driven architecture is increasingly prevalent because it enables organizations to support operational workloads like fraud detection and personalized recommendations as well as analytical workloads like business intelligence and historical reporting. Both of these can be supported from a single stream of data.

The choice of processing architecture depends primarily on business requirements, including data volume, latency, scalability and processing complexity. The most common processing patterns include batch processing, stream processing and the Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) data integration models.

Batch processing executes transformations on large collections of data at scheduled intervals, such as hourly, nightly or weekly. This approach is well suited for workloads that do not require immediate results, including financial reporting, historical analysis and large-scale data preparation. Common technologies include Apache Spark and data transformation frameworks such as Databricks.

Stream processing (sometimes called real-time processing) continuously processes data as events are generated, allowing organizations to respond with minimal latency. This approach is commonly used for applications such as fraud detection, IoT telemetry, operational monitoring and personalized customer experiences. A platform like Apache Kafka provides the event streaming, while processing frameworks such as Apache Flink analyze and transform the incoming data streams in real time.

Organizations must also determine whether to adopt an ETL or ELT approach. In ETL architectures, data is transformed before it is loaded into the destination system. In contrast, ELT architectures first load raw data into a centralized storage platform and perform transformations afterward. Because modern cloud data warehouses and data lakehouses provide substantial computational resources, ELT has become the predominant approach for many contemporary data platforms.

Pipeline design is often driven by both the scale and size of the data and by what the data is needed for. Strategies for architectures have evolved to fit a wide variety of needs and requirements. For instance, service level agreements (SLAs) define the expected levels of availability, latency, freshness, reliability and data quality that the pipeline must achieve.

In many organizations, architectural decisions are driven as much by SLA requirements as by the volume or type of data being processed. For instance, an SLA might specify availability and fault tolerance. This often comes in the form of uptime targets like 99.9% availability. SLAs can also specify scalability targets. For instance, a data engineer might specify that the pipeline needs to process one million events per minute.

Another common requirement might focus on data quality, establishing a maximum percentage of missing records or required schema validation. Understanding the SLAs can help narrow what data products will satisfy the requirements of the agreements.

A Lambda architecture is a big data processing pattern designed to support both high-throughput historical analysis and low-latency real-time analytics. It achieves this by dividing data processing into two parallel pipelines—a batch layer and a speed (or streaming) layer—that process the same incoming data simultaneously.

The batch layer computes comprehensive, highly accurate views of the data by using the complete historical dataset. The speed layer, meanwhile, processes newly arriving events as they occur, providing observability and immediate insights before they are incorporated into the batch results.

This dual-path approach allows organizations to balance the tradeoff between data accuracy and processing latency. The speed layer delivers near-real-time updates for data visualization, alerts and time-sensitive applications, while the batch layer periodically recomputes results from the authoritative historical record to correct late-arriving data, processing errors or incomplete information.

A serving layer then combines the outputs of both processing paths, presenting users with a unified view that reflects both the latest events and the most accurate historical data.

Lambda architecture emerged as a solution for early large-scale data processing systems in which batch frameworks offered robust, fault-tolerant computation but could not satisfy real-time analytics requirements. Although the architecture remains useful for some workloads, it introduces additional operational complexity because organizations must develop, maintain and synchronize two independent processing pipelines.

As data platforms have increased in scale and complexity, organizations have adopted architectural patterns that promote consistency, governance and maintainability. One of the most widely used approaches is the medallion architecture.

A medallion architecture organizes data into progressively refined layers as it moves through a data lake or lakehouse. Rather than transforming data in a single step, the architecture applies successive stages of validation, enrichment and aggregation, allowing each layer to serve a distinct purpose while preserving lineage and improving data quality.

The architecture is typically divided into three layers.

Bronze layer (raw data)

The Bronze layer contains data in its original form as it is ingested from source systems. Data is stored with minimal or no transformation to preserve its fidelity and maintain a complete historical record. Whenever possible, the original schema is retained, allowing the Bronze layer to serve as the authoritative system of record.

For example, website clickstream events might be written directly from a Kafka topic into Bronze tables while preserving every field generated by the source application.

The primary objectives of the Bronze layer are to preserve the original source data, support future reprocessing as business requirements evolve and maintain a complete audit trail.

Silver layer (cleaned and enriched data)

The Silver layer contains data that has been validated, standardized and enriched to improve its quality and consistency. Typical transformations include removing duplicate records, correcting malformed data, validating schemas, standardizing formats and units of measure, joining related datasets and applying business rules.

For example, clickstream events might be enriched with customer profile information, validated against predefined schemas and standardized before being stored as Silver tables.

The primary objectives of the Silver layer are to improve data quality, create reusable datasets and establish consistent business definitions that can be shared across multiple applications and analytical workloads.

Gold layer (business-ready data)

The Gold layer contains curated datasets that have been optimized for business consumption and analytical performance. These datasets typically include aggregations, key performance indicator (KPI) calculations, dimensional models, machine learning features and executive reporting tables.

For example, Gold tables might contain daily revenue by region, customer lifetime value metrics, sales performance summaries or comprehensive customer 360 views.

The primary objectives of the Gold layer are to provide trusted business metrics, support dashboards and reporting, and supply high-quality data for advanced analytics and AI applications.