What is a data pipeline?

Data pipelines collect, transform, and store data to surface to stakeholders for a variety of data projects.

aerial top view Oil refinery at night, thailand.
What is a data pipeline?

A data pipeline is a method in which raw data is ingested from various data sources and then ported to data store, like a data lake or data warehouse, for analysis. Before data flows into a data repository, it usually undergoes some data processing. This is inclusive of data transformations, such as filtering, masking, and aggregations, which ensure appropriate data integration and standardization. This is particularly important when the destination for the dataset is a relational database. This type of data repository has a defined schema which requires alignment—i.e. matching data columns and types—to update existing data with new data. 

As the name suggests, data pipelines act as the “piping” for data science projects or business intelligence dashboards. Data can be sourced through a wide variety of places—APIs, SQL and NoSQL databases, files, et cetera, but unfortunately, that data usually isn’t ready for immediate use. Data preparation tasks usually fall on the shoulders of data scientists or data engineers, who structure the data to meet the needs of the business use case. The type of data processing that a data pipeline requires is usually determined through a mix of exploratory data analysis and defined business requirements. Once the data has been appropriately filtered, merged, and summarized, it can then be stored and surfaced for use. Well-organized data pipelines provide the foundation for a range of data projects; this can include exploratory data analyses, data visualizations, and machine learning tasks.


Types of data pipelines

There are two main types of data pipelines, which are batch processing and streaming data.

Batch processing

The development of batch processing was critical step in building data infrastructures that were reliable and scalable. In 2004, MapReduce, a batch processing algorithm, was patented and then subsequently integrated in open-source systems, like Hadoop, CouchDB, and MongoDB.

As the name implies, batch processing loads “batches” of data into a repository during set time intervals, which are typically scheduled during off-peak business hours. This way, other workloads aren’t impacted as batch processing jobs tend to work with large volumes of data, which can tax the overall system. Batch processing is usually the optimal data pipeline when there isn’t an immediate need to analyze a specific dataset (e.g. monthly accounting), and it is more associated with the ETL data integration process, which stands for “extract, transform, and load.”

Batch processing jobs form a workflow of sequenced commands, where the output of one command becomes the input of the next command. For example, one command may kick off data ingestion, the next command may trigger filtering of specific columns, and the subsequent command may handle aggregation. This series of commands will continue until the data is completely transformed and written into data repository.

Streaming data

Unlike batching processing, streaming data is leveraged when it is required for data to be continuously updated. For example, apps or point of sale systems need real-time data to update inventory and sales history of their products; that way, sellers can inform consumers if a product is in stock or not. A single action, like a product sale, is considered an “event”, and related events, such as adding an item to checkout, are typically grouped together as a “topic” or “stream.” These events are then transported via messaging systems or message brokers, such as the open-source offering, Apache Kafka. 

Since data events are processed shortly after occurring, streaming processing systems have lower latency than batch systems, but aren’t considered as reliable as batch processing systems as messages can be unintentionally dropped or spend a long time in queue. Message brokers help to address this concern through acknowledgements, where a consumer confirms processing of the message to the broker to remove it from the queue. 

Learn how to build a Kafka streaming app with IBM Streams


Data pipeline architecture

Three core steps make up the architecture of a data pipeline. 

1. Data ingestion: Data is collected from various data sources, which includes various data structures (i.e. structured and unstructured data). Within streaming data, these raw data sources are typically known as producers, publishers, or senders. While businesses can choose to extract data only when they are ready to process it, it’s a better practice to land the raw data within a cloud data warehouse provider first. This way, the business can update any historical data if they need to make adjustments to data processing jobs. 

2. Data Transformation: During this step, a series of jobs are executed to process data into the format required by the destination data repository. These jobs embed automation and governance for repetitive workstreams, like business reporting, ensuring that data is cleansed and transformed consistently.  For example, a data stream may come in a nested JSON format, and the data transformation stage will aim to unroll that JSON to extract the key fields for analysis.

3. Data Storage: The transformed data is then stored within a data repository, where it can be exposed to various stakeholders. Within streaming data, this transformed data are typically known as consumers, subscribers, or recipients.  


Data pipeline vs. ETL pipeline

You may find that some terms, such as data pipeline and ETL pipeline, are used interchangeably in conversation. However, you should think abut an ETL pipeline as a subcategory of data pipelines. The two types of pipelines are distinguished by three key features:

• ETL pipelines follow a specific sequence. As the abbreviation implies, they extract data, transform data, and then load and store data in a data repository. All data pipelines do not need to follow this sequence. In fact, ELT pipelines have become more popular with the advent of cloud-native tools. While data ingestion still occurs first with this type of pipeline, any transformations are applied after the data has been loaded into the cloud data warehouse.  

• ETL pipelines also tend to imply the use of batch processing, but as we noted above, the scope of data pipelines is broader. They can also be inclusive of stream processing. 

• Finally, while unlikely, data pipelines as a whole do not necessarily need to undergo data transformations, like ETL pipelines. It’s just rare to see a data pipeline that doesn’t utilize transformations to facilitate data analysis. 


Use cases of data pipelines

As big data continues to grow, data management becomes an ever-increasing priority. While data pipelines serve various functions, the following are three broad applications of them within business: 

• Exploratory data analysis: Exploratory data analysis (EDA) is used by data scientists to analyze and investigate data sets and summarize their main characteristics, often employing data visualization methods. It helps determine how best to manipulate data sources to get the answers you need, making it easier for data scientists to discover patterns, spot anomalies, test a hypothesis, or check assumptions.

• Data visualizations: Data visualizations represent data via common graphics, such as charts, plots, infographics, and even animations. These visual displays of information communicate complex data relationships and data-driven insights in a way that is easy to understand.

• Machine learning: Machine learning is a branch of artificial intelligence (AI) and computer science which focuses on the use of data and algorithms to imitate the way that humans learn, gradually improving its accuracy. Through the use of statistical methods, algorithms are trained to make classifications or predictions, uncovering key insights within data mining projects.


Data pipelines and IBM

IBM Cloud Pak for Data leverages microservices and its leading data and AI capabilities to enable the intelligent integration of data across distributed systems, providing companies with a holistic view of business performance. This facilitates faster collection, organization, and insight into enterprise data, allowing businesses to make decisions at scale. Data management teams can also trust that their data is secure with IBM’s competitive security frameworks, ensuring adherence to regulatory policies and reducing any compliance risks. Find out how IBM Cloud Pak® for Data and IBM Streams can improve your business’s data pipeline architecture across multiple environments.