What is a machine learning pipeline?

The end-to-end machine learning pipeline comprises three stages: 

1. Data processing: Data scientists assemble and prepare the data that will be used to train the ML model. Phases in this stage include data collection, preprocessing, cleaning and exploration. 
2. Model development: Data practitioners choose or create a machine learning algorithm that fits the needs of the project. The algorithm is trained on the data from the previous step, and the resulting model is tested and validated until it is ready for use. 
3. Model deployment: Developers and software engineers deploy the model for real-world use, integrating it into a production environment and monitoring its performance. 

Machine learning workflows are a core building block for the larger discipline of machine learning operations (MLOps). Much of the process can be automated through various automated machine learning (AutoML) techniques that manage dependencies between stages and endpoints. 

A data pipeline is an architecture designed and built by data scientists that collects data from different sources, then stores and organizes it in a centralized data repository, such as a data warehouse. A machine learning pipeline is a workflow for designing, building and deploying an AI system. 

Both phrases use the term pipeline, but whereas a data pipeline is more of a tangible system, an ML pipeline is a theoretical series of steps. An ETL pipeline is an example of a data pipeline that extracts data from various sources, transforms it into a unified format and loads it into a destination system. In machine learning, an ETL pipeline would collect data and format it into a training dataset.

Before initializing an ML workflow, business leaders, developers and other stakeholders agree on the objectives of a machine learning project. Understanding why AI is needed and what it is intended to accomplish keeps expectations realistic and aligns stakeholders around a shared purpose.

When deciding whether to incorporate AI into a workflow or product, stakeholders must first identify the business objective that the ML model is meant to solve, then demonstrate how AI can fulfill it. Some companies approach AI with this logic reversed: “We want to use AI. What should we do with it?” 

Maximizing AI return on investment (ROI) requires that leaders understand the use case, then work toward ML solution tailored to that purpose.

Clear metrics for success, such as documented KPIs (key performance indicators), inform stakeholders whether the ML project is delivering on its goals. These KPIs should reflect the goals set in the previous stage. For example, an ML model being deployed to increase efficiency might look to prioritize ROI.

Knowing the risk landscape and potential blockers helps teams navigate the project effectively. This step includes defining the data requirements and evaluating the relevant regulations, if any, for data collection and storage. The same applies for any limitations that might affect model selection, such as compute or memory requirements.

After scoping out the problem the ML model is to solve, the first step in an ML workflow is to collect, prepare and analyze the data. Practitioners must identify relevant data sources, gather and integrate data from them, prepare and clean the data, employing data science techniques including feature engineering to arrive at a prepared data set. 

The data processing stage is usually the most time-consuming. But ML model performance hinges on good data. Any errors and oversights in the data engineering phase negatively affect the model’s performance across its lifecycle. Data automation strategies can reduce the time and human effort required to produce strong training datasets. 

Data processing includes: 

• Data ingestion 

• Data preprocessing

• Data exploration 

• Feature engineering 

• Data splitting 

Data ingestion is the collection and import of data from disparate data sources into a centralized data repository through a data pipeline. Data scientists must identify appropriate data sources, such as proprietary enterprise data stored internally—sales reports, customer demographics and other organizational knowledge. 

Sometimes external data is also required. External data sources can include API connections to data providers, data scraped from the internet or synthetic data. Because new data is always being created, data ingestion is often a continuous process.

Data preprocessing, or data preparation, transforms the raw data from the previous step into clean data that is ready for analysis. After gaining an understanding of the training data through exploratory data analysis (EDA), data scientists select data preprocessing strategies. Data preprocessing steps include: 

• Data wrangling (data transformation): Transforming the data into the appropriate format 

• Identifying missing values and dealing with outliers

• Data cleaning: correcting errors in the dataset

• Data normalization: standardizing the dataset

• Denoising: removing random errors and signal interference 

• Data integration: combining the data into a unified dataset

Data exploration is the process of evaluating data to understand the information it contains. EDA aims to learn the characteristics of the data, discover patterns and relationships and identify insights with the help of data visualization tools. 

EDA findings inform the model selection choices that occur next.

Feature selection is a critical data preprocessing step that involves identifying the most relevant features, or characteristics, of the data points. Data features are extracted and selected that give the model the best possible chance of solving real-world challenges. 

Focusing on the wrong features can result in a model that doesn’t perform as intended. After applying feature extraction techniques to streamline the data, data scientists choose the features that will lead to the strongest model predictions.

After the training data has been prepared, the next step in the ML workflow is to build the machine learning model. The process of creating a deep learning model involves selecting an appropriate machine learning algorithm and exposing it to the training datasets. The result of this process is the creation of an AI model ready for real-world use with similar unseen data. 

The model development process involves: 

• Model selection 

• Hyperparameter tuning 

• Model training

• Model evaluation

Model selection is the process of choosing the type of model that is most likely to deliver top performance in the intended use case. The initial project planning stages have already given all stakeholders and participants a clear understanding of the business needs, limitations and project goals. ML practitioners base their choices on these factors, balancing optimization with feasibility. 

Choices include linear regression and logistic regression, random forests and decision trees, neural networks and large language models (LLMs), support vector machines (SVMs), ensemble models, agentic systems and many others. 

Depending on the nature of the machine learning challenge, certain types of algorithms make more suitable candidates. 

For example, neural networks can handle complex generative AI challenges but come with high compute costs and are more prone to overfitting. Regression models are compute-efficient but have limited use cases.

Model hyperparameters are external variables that control the model’s behavior during training. Hyperparameters also govern the shape of the model that the algorithm builds, such as the number of neurons and layers in a neural network. 

Hyperparameter tuning is the process of optimizing the hyperparameters so that the training process produces a top-performing model. Data scientists can set hyperparameters manually but usually automate the process through various algorithms and other techniques.

Model training is the process of optimizing a model’s performance with training datasets that are similar to the input data the model processes once deployed. A machine learning training pipeline is an extensive system that can take any number of shapes depending on the algorithm and the task for which the model is being developed. 

Many training methods revolve around minimizing a loss function that measures the model’s error: the gap between the model’s outputs and real-world data values. With each round of training, the new model updates its parameters as it fits closer to the training data. Each update iterates on previous results. 

Model training methods include: 

• Supervised learning: The model is trained on a dataset of structured data. Inputs are labeled with corresponding outputs, teaching the model how to associate input features with the correct output values. 

• Unsupervised learning: The model is trained on unstructured data and must discern the patters and relationships between data points and features on its own. 

• Semi-supervised learning: The model is trained in a hybrid method that mixes supervised and unsupervised learning. 

• Self-supervised learning: The model is trained with unlabeled data for tasks that usually call for supervised learning. 

• Reinforcement learning: The model is trained to take the actions that generate the largest possible reward, rather than minimize error. 

• Continual learning: The model is trained on a real-time stream of input data, as opposed to a preassembled training dataset.

After the model is deemed to be trained—such as when its loss function has been sufficiently minimized—its performance is evaluated before deployment. The LLM evaluation process uses the testing and validation datasets that were prepared during the data splitting phase.

Validation estimates the model’s prediction error: how good is it at making the correct predictions? During training, the machine learning algorithm often outputs multiple models with various hyperparameter configurations. Validation identifies the model with the optimal hyperparameter configuration.

Testing simulates real-world values to evaluate the best-performing model’s generalization error: how well does the model adapt to new unseen data? Test data is independent of training data and benchmarks the model’s performance after training is complete. Tests reveal whether the model will perform as intended after it is deployed.

After developing a suitable model with strong performance, it’s time to put that model to work. Model deployment serves the model to users in the intended production environment. This can be anything from a mobile app or API connection to a pharmaceutical development or robotics research facility. 

Models don’t begin working until they are actively deployed. Achieving strong results from a machine learning project means that the model must be deployed in a way that makes it easy to use, whether that is by consumers, business leaders or other computer systems. 

Model deployment includes: 

• Model serialization 

• Integration 

• Architecture 

• Monitoring 

• Updates 

• Compliance

Serialization is a common deployment method that involves converting a model into a format that can be stored and transmitted, then deserializing it in the production environment. It’s like packing up a roomful of belongings into a box, moving the box to a new home, then unpacking to set up the new room. 

For example, Python, a coding language popular with ML development, recommends the pickle framework for deployment.

Integration incorporates the model into its production environment, such as a mobile app. Models can be served through cloud computing providers such as AWS or Azure, or hosted onsite. Alternatively, it might be better to use a containerized solution such as Kubernetes and Docker. 

Depending on how the model will be served, developers need to make the model accessible with the appropriate machine learning libraries and frameworks, such as PyTorch or TensorFlow Serving.

Portability and scalability are two primary concerns to consider during ML deployment. 

• Portability is the ease with which the model can be transferred between systems. 

• Scalability is the model’s ability to handle growing workloads, such as an increasing user base, without needing to be redesigned. 

The model’s production environment must be able to support the projected growth of the machine learning project. Autoscaling and orchestration tools can help field increased demand over time.

The ML workflow isn’t complete once the model is deployed. The model’s performance must be monitored over the course of the AI lifecycle to avoid model drift: when performance suffers due to changes in data distribution. Many other metrics pertain to the model’s ability to generate and process tokens: a single unit of input or output. Some of these metrics include: 

• Time per output token (TPOT) / inter-token latency (ITL): The amount of time it takes for the model to generate a token. 

• Time to first token (TTFT): The amount of time it takes for a model to generate the first token of its response. 

• Throughput: A measure of the model’s overall token generation capacity, measured in tokens per second (TPS). 

• Latency: The amount of time it takes for the model to generate a complete output after receiving a user input.

Unless a model is being trained with continual learning, its training dataset is finite. A model’s knowledge cutoff refers to the last date on which its knowledge base was updated with new data. Over time, a model becomes less relevant as the information in the knowledge base grows more dated. 

Models must be regularly updated to mitigate model drift and keep error rates to an acceptable minimum. New data, new features and algorithmic updates can both optimize model performance. Retraining can also help models stay current.

Wherever data collection is concerned, model operators must consider all relevant legal regulations and requirements surrounding privacy, intellectual property, copyright and other concerns. For example, HIPAA protects medical data in the U.S., while GDPR provides specific data protections to people in the European Union. 

Models built for use in regulated industries such as pharmaceuticals and finance might also be subject to stricter operating controls. Any models used in an enterprise setting likely process sensitive internal data, necessitating strong cybersecurity measures. 

Model operators are obligated to protect user data and prevent their models from being used for malicious ends, such as fraud and misinformation. One advantage of open-source models is that anyone can evaluate the model to see how it works and whether it’s abiding by all relevant regulations.

Machine learning pipelines offer many benefits, such as: 

• Modularization 

• Reproducibility

• Efficiency 

• Scalability

• Experimentation 

• Deployment 

• Collaboration 

• Version control and documentation

Pipelines break down the machine learning process into modular, well-defined steps. Each step can be developed, tested and optimized independently, making it easier to manage and maintain the workflow.

Machine learning pipelines make it easier to reproduce experiments. Defining the sequence of steps and their parameters in a pipeline helps ensure consistent results. If a step fails or a model's performance deteriorates, the pipeline can be configured to raise alerts or take corrective actions.

Pipelines automate many routine tasks, such as data preprocessing, feature engineering and model evaluation. This efficiency can save time and reduce the errors.

Pipelines can be scaled to handle large datasets or complex workflows. As data and model complexity grow, you can adjust the pipeline without having to reconfigure everything from scratch.

Modifying individual steps within the pipeline opens the door to experimentation with different data preprocessing techniques, feature selections and models. This flexibility enables rapid iteration and optimization.

Pipelines facilitate the deployment of machine learning models into production. A well-defined pipeline for model training and evaluation makes deployment easier into an application or system.

Pipelines enable teams of data scientists and engineers to collaborate. Because the workflow is structured and documented, it's easier for team members to understand and contribute to the project.

Version control systems track changes in pipeline code and configurations, allowing for rollback to previous versions. A well-structured pipeline encourages better documentation of each step.

The history of machine learning pipelines is closely tied to the evolution of both machine learning and data science as fields. While the concept of data processing workflows predates machine learning, the formalization and widespread use of machine learning pipelines developed more recently. 

The history of machine learning pipelines includes the following developments: 

• Early data processing workflows (pre-2000s) 

• Emergence of machine learning (2000s) 

• Rise of data science (late 2000s to early 2010s) 

• Development of machine learning libraries and tools (2010s) 

• Rise of AutoML (2010s) 

• Integration with DevOps (2010s) 

Before the widespread adoption of machine learning, data processing workflows were used for tasks such as data cleaning, transformation and analysis. These workflows were typically manual and involved scripting or tools such as spreadsheet software. However, machine learning was not a central part of these processes during this period.

Machine learning gained prominence in the early 2000s with advancements in algorithms, computational power and the availability of large datasets. Researchers and data scientists started applying machine learning to various domains, leading to a growing need for systematic and automated workflows.

The term data science became popular as a multidisciplinary field that combined statistics, data analysis and machine learning. This era saw the formalization of data science workflows, including data preprocessing, model selection and evaluation, which are now integral parts of machine learning pipelines.

The 2010s brought the development of machine learning libraries and tools that facilitated the creation of pipelines. Libraries like scikit-learn (for Python) and caret (for R) provided standardized APIs for building and evaluating machine learning models, making it easier to construct pipelines.

Automated machine learning (AutoML) tools and platforms emerged to automate the process of building machine learning pipelines. These tools typically automate tasks such as hyperparameter tuning, feature selection and model selection, making machine learning more accessible to non-experts with visualizations and tutorials.

DevOps practices began to incorporate machine learning pipelines to enable the continuous integration and deployment (CI/CD) of machine learning models. This integration, known as machine learning operations (MLOps), emphasized the need for reproducibility, version control and monitoring in ML pipelines. 

MLOps helps data science teams effectively navigate complex AI orchestration challenges. In real-time deployment, the pipeline replies to a request within milliseconds.