What is continual learning?

Incremental learning allows models to acquire new knowledge and keep pace with the unpredictability of the real world without forgetting old knowledge. Nonstationary data means that the data distributions are not static. When implemented successfully, continual learning results in models that maintain task-specific knowledge and can also generalize across dynamic data distributions. 

Continual learning models are designed to apply new data adaptively in changing environments. Also known as lifelong learning, continual learning is inspired by neuroscience concepts relating to the way humans learn new things while also retaining what they already know. If a person learns to skateboard, they do not immediately forget how to ride a bicycle.

Traditional machine learning systems train models on large static datasets. The dataset passes through the model’s algorithm in batches as the model updates its weights, or parameters. The model processes the entire dataset multiple times, with each cycle known as an epoch. 

Developers identify the purpose of the deep learning model ahead of time, assemble a training dataset to fit the learning objective and train the model on that data. Then, the model is tested, validated and deployed. Fine-tuning the machine learning model with more data can tailor its performance to new tasks. 

Traditional learning methods do not fully reflect the dynamism of the real world. Supervised learning uses static datasets with known outcomes. Unsupervised learning lets a model sort through data on its own, but the training data is still finite and unchanging. Reinforcement learning is similarly safe and constrained. 

In contrast to traditional learning methods, continual learning attempts to apply the plasticity of the human brain to artificial neural networks. Neuroplasticity is the quality of the brain that allows it to adapt, learning without forgetting previous knowledge as it encounters changing circumstances. 

Some types of continual learning still begin with offline batch-training in multiple epochs, similar to traditional offline training. Online continual learning solely trains models with a stream of single-pass data. 

Continual learning helps deep neural networks optimize and adapt in dynamic environments. Traditional machine learning requires extensive and fixed datasets, sufficient time and compute for training and a known purpose for the model. When one or more of these requirements is not met, continual learning provides an alternative. 

• Mitigating catastrophic forgetting

• Small training datasets 

• Changing data distributions

• Resource optimization 

• Noise tolerance 

When deep learning models are trained on new data or new distributions, they can lose previous knowledge. Known as catastrophic forgetting, this phenomenon is a consequence of a model overfitting its parameters to new data. The models update their internal weights to a degree such that the new parameters are no longer relevant to the model’s original job. 

Continual learning streams training data incrementally through the AI model. The model is fed a sequence of small datasets, sometimes consisting of just a single sample. Transfer learning—when a model applies previous learning to new tasks—helps minimize the amount of new data required. 

The world is in a constant state of flux. Humans and other animals evolved the ability to learn to help them thrive in adversity. For example, if one food supply runs out, figuring out how to eat something else can ensure survival. 

But not all animals are as capable. Koalas cannot even recognize their primary food source—eucalyptus leaves—if the leaves are removed from a tree and placed in a pile on a plate. While koalas sometimes eat other leaves from other trees, they can conceive of food only as “leaves on trees.” Their smooth brains cannot deviate from this expectation. 

Consider a computer vision model intended for use in self-driving cars. The model must know how to recognize other vehicles on the road, but also pedestrians, cyclists, motorcyclists, animals and hazards. It must perceive and adapt flawlessly to changing weather and traffic patterns, such as a sudden downpour or if an emergency vehicle is approaching with its lights and siren on. 

Languages change over time. A natural language processing (NLP) model should be able to process shifts in what words mean and how they are used. Similarly, a model designed for robotics must be able to adapt if the robot’s environment changes. 

AI models are resource-intensive. They can cost millions of dollars to train and consume large amounts of electricity and water. It isn’t always possible to deploy new models whenever new tasks arise. Nor is it computationally feasible to preserve every single previous task in a model’s available memory. 

Continual learning allows large language models (LLMs) and other neural networks to adapt to shifting use cases without forgetting how to handle previous challenges. Enterprises can minimize the number of models in operation by expanding the potential capabilities of each model they use. 

If trained well, continual learning algorithms should be able to confidently identify relevant data while ignoring noise: meaningless data points that do not accurately reflect real-world values. Noise results from signal errors, measurement errors and input errors and also covers outliers. Outliers are data points so dissimilar to the rest of the data as to be irrelevant. 

Continual learning challenges can be broadly divided into three categories, depending on how the data stream is changing over time¹:

• Task-incremental continual learning

• Domain-incremental continual learning

• Class-incremental continual learning

Task-incremental learning is a step-by-step approach to multitask learning in which an algorithm must learn to accomplish a series of different tasks. It must be clear to the algorithm which task is expected of it, either by the tasks being sufficiently distinct from one another or by labeling inputs with the appropriate output. 

A real-world example of task-incremental learning would be learning how to speak Japanese, then Mandarin, then Czech and then Spanish. It is usually clear which language the speaker should use at any particular time. 

Because tasks are streamed to the model in sequence, the challenge is one of helping ensure that the model can sufficiently transfer learning from one to the next. The total number of tasks is also not always known in advance, especially with models already in deployment. 

The prevention of catastrophic forgetting is a given—getting the model to apply transfer learning is the real goal with task-incremental learning methodologies. 

Domain-incremental learning covers challenges in which the data distribution changes, but the type of challenge stays the same. The conditions surrounding the task have changed in some way, but the potential outputs have not. Unlike task-incremental learning, the model is not required to identify the specific task to solve. 

For example, a model built for optical character recognition (OCR) would need to recognize various document formats and font styles. It is not important to know how or why the environment has changed, but to recognize that it has and complete the task regardless. 

Changes in data distribution are a longstanding challenge in machine learning because models are typically trained on a discrete, static dataset. When data distributions change post-deployment, domain-incremental learning can help models mitigate performance losses.

Class-incremental learning is when a classifier model must perform a series of classification tasks with a growing number of output classes. The model must be able to both correctly solve each instance while also recalling classes encountered in previous instances. 

A model trained to classify vehicles as cars or trucks might later be asked to identify buses and motorcycles. The model will be expected to maintain its understanding of all classes learned over time, not just the options in each instance. If trained on “cars versus trucks” and later given “buses versus motorcycles,” the model should also successfully determine whether a vehicle is a car or a bus. 

State-of-the-art class-incremental learning is one of the most difficult continual learning challenges because the emergence of new classes can erode the distinctions between previously established classes.

The goal of all continual learning techniques all aim is to balance the stability-plasticity dilemma: making a model stable enough to retain previously learned knowledge while still plastic enough to cultivate new knowledge. Though researchers have identified numerous approaches to continual learning, many can be assigned into one of three categories:

• Regularization techniques

• Parameter isolation techniques

• Replay techniques

Regularization is a set of techniques that restrict a model’s ability to overfit to new data. The model is not allowed to update its architecture during incremental training, while techniques such as knowledge distillation—where a larger model “teaches” a smaller one—help preserve knowledge. 

• Elastic weight consolidation (EWC) adds a penalty to the learning algorithm’s loss function that restricts it from making drastic changes to a model’s parameters. Optimization algorithms use the gradient of the loss function as a metric to benchmark model performance. 

• Synaptic intelligence (SI), which limits parameters updates based on a cumulative understanding of each parameter’s relative importance. 

• Learning without forgetting (LWF) trains models with new task data and maintains old knowledge by preserving output probabilities of previous tasks.

Parameter isolation methods alter a portion of a model’s architecture to accommodate new tasks while freezing the parameters for previous tasks. The model rebuilds itself to broaden its capabilities, but with the caveat that some parameters can’t be adjusted. Subsequent training is done on only the parameters that are eligible for new tasks. 

For example, progressive neural networks (PNNs) create task-specific columns of neural networks for new tasks. Parallel connections to other columns enable transfer learning while preventing these columns from being changed.

Replay techniques involve regularly exposing a model during training activations to samples from previous training datasets. Replay-based continual learning saves samples of older data in a memory buffer and incorporates it into subsequent training cycles. The continued exposure to older data prevents the model from overfitting to new data. 

Memory techniques are reliably effective but come at the cost of regular access to previous data, which requires sufficient storage space. Situations that involve the use of sensitive personal data can also prevent problems for memory technique implementation. 

Generative replay uses a generative model to synthesize samples of previous data to feed to the model being trained, such as a classifier that needs to learn new classes without forgetting old ones.