Home Think Topics Recommendation Engine What is a recommendation engine?
Explore with watsonx.ai Subscribe for AI updates
Pictogram collage with clouds, pie chart and graphs

 

 

Published: 19 June 2024
Contributors: Rina Caballar, Cole Stryker

What is a recommendation engine?

A recommendation engine, also called a recommender, is an artificial intelligence (AI) system that suggests items to a user. Recommendation systems rely on big data analytics and machine learning (ML) algorithms to find patterns in user behavior data and recommend relevant items based on those patterns.

Recommendation engines help users discover content, products or services they might not have found on their own. These systems are an integral part of generating sales and cultivating engagement for many online businesses, including e-commerce websites, media streaming platforms, search engines and social media networks.

A recommender suggests the next movie or video to watch, a similar song to listen to, relevant search results or a product that complements a certain order.

The suggestions created by recommendation systems also play a vital role in personalizing user experiences. According to research by management consulting firm McKinsey (link resides outside ibm.com), personalization can raise revenues by 5%–15%. Also, 76% of customers feel frustrated when they do not experience personalized interactions.

The market for recommendation systems is growing. In 2024, the recommendation engine market (link resides outside ibm.com) is estimated at USD 6.88 billion, and this market size is projected to triple in 5 years.

IDC Spotlight: The truth about successful generative AI

Uncover the benefits of AI platforms that enable foundation model customization.

Related content

Register for the guide to foundation models

How recommendation engines work

To target users with suitable suggestions, a recommendation engine blends data science and machine learning.

Recommenders typically operate in 5 phases to predict the most accurate recommendations:

1. Data gathering

Data is the foundation of a recommendation system, so collecting data is a crucial first step. The 2 key types of data to be gathered include explicit data and implicit data.

Explicit data encompasses user actions and activities such as comments, likes, ratings and reviews. Implicit data comprises user behavior such as browsing history, shopping cart events, clicks, past purchases and search history.

Recommenders also use other customer data such as demographics (age or gender) and psychographics (interests or lifestyle) to find similar users, and feature data (such as price range or item type) to determine related products or services.

2. Storage

After data has been gathered, the next step is to store it. The kind of storage system depends on the types of data collected.

A data warehouse can aggregate data from different sources to support data analysis and machine learning, while data lakes can store both structured and unstructured data.

A data lakehouse combines the best aspects of data warehouses and data lakes into a single data management solution.

3. Analysis

The analysis phase employs machine learning algorithms to process and examine data sets. These algorithms detect patterns, identify correlations and weigh the strength of those patterns and correlations. Machine learning models can be trained on large data sets to make fitting recommendations.

4. Filtering

The final step is to filter the data, showing the most relevant items from the previous analysis stage. Data filtering involves applying certain mathematical rules and formulas to the data depending on the type of recommendation engine used.

5. Refining

An optional refining step might be added to regularly assess the outputs of a recommendation system and further optimize the model to continually enhance its accuracy and quality.

Types of recommendation engines 

Recommenders differ based on the filtering method that they use. There are generally 3 types of recommendation engines:

1. Collaborative filtering 

A collaborative filtering system filters suggestions based on a particular user’s likeness to others. Collaborative recommender systems rely on explicit and implicit data and assume that users with comparable preferences will likely be interested in the same items and potentially interact with them in similar ways in the future.

Amazon (link resides outside ibm.com), for instance, uses collaborative filtering for its product recommendations, just like Spotify (link resides outside ibm.com) does for its audio content.

Collaborative filtering recommenders can offer effective suggestions and won’t usually need detailed item descriptions. However, collaborative filtering is also prone to the cold start problem, which happens when the system has limited historical data to draw from, especially for new users.

There are 2 main kinds of collaborative filtering systems: memory-based and model-based.

Memory-based

Memory-based systems represent users and items as a matrix. They are an extension of the k-nearest neighbors (KNN) algorithm because they aim to find their “nearest neighbors,” which can be similar users or similar items. Memory-based systems are further subdivided into 2 types:

  • User-based filtering computes similarities between a particular user and all other users in the matrix. User-based similarity functions are calculated between rows in the user-item matrix.
  • Item-based filtering computes item similarity through user behavior (how users interact with items and not item features). Item-based similarity functions are calculated between columns in the user-item matrix.

Model-based

Alternatively, model-based systems create a predictive machine learning model of the data. The user-item matrix serves as the training data set for the model, which then yields predictions for missing values, that is, items that a user has not yet found and will therefore be recommended.

One of the most commonly used model-based collaborative filtering algorithms is matrix factorization. This dimensionality reduction method decomposes the often large user-item matrix into two smaller matrices—one for users and another for items—having a select few dimensions. The 2 matrices are then multiplied together to predict the missing values (or the recommendations) in the larger matrix.

A more advanced implementation of matrix factorization harnesses deep learning neural networks. Other model-based systems employ machine learning algorithms such as Bayes classifiers, clustering and decision trees.

2. Content-based filtering

A content-based filtering system filters recommendations based on an item’s features. Content-based recommender systems assume that if a user likes a particular item, they will also like another similar item. Content-based filtering considers item descriptions such as color, category, price and other metadata assigned by keywords and tags, along with explicit and implicit data.

Content-based filtering systems represent items and users as vectors in a vector space. Proximity is used to determine the similarity between items. The closer 2 vectors are in space, the more similar they’re considered to be. Vectors similar to previous items according to their supplied features will be recommended to the user.

Content-based recommenders apply a user-based classifier or regression model. Descriptions and features of items a user is interested in act as the model’s training data set, which then yields predictions for recommended items.

Content-based recommendation systems can be further improved by using natural language processing tags. However, this tagging process can be tedious for huge volumes of data.

Unlike collaborative filtering, the cold start problem is less of an issue since content-based filtering is based on metadata characteristics rather than past user interaction. However, content-based filtering can be limited in exploring new items, as it often suggests those similar to what users liked previously.

3. Hybrid recommendation system 

As its name implies, a hybrid recommendation system merges collaborative filtering and content-based filtering.

As a result, a hybrid approach can greatly enhance the performance of a recommendation engine, but it requires advanced architectures and intensive computational power.

Netflix, for instance, uses a hybrid recommendation system (link resides outside ibm.com) for its movie and TV show recommendations.

Benefits of using a recommendation engine 

A recommendation engine can bring value to both businesses and users. Here are some advantages organizations can gain from investing in recommendation systems:

Improved user experience

Recommending the right product or service saves users time from scrolling endlessly through an extensive catalog. For instance, 80% of what viewers watch on Netflix comes from suggestions powered by recommendation algorithms. Moreover, suggesting relevant content leads to personalized experiences.

Better customer retention

According to research from McKinsey, an enhanced customer experience (link resides outside ibm.com) translates to higher customer satisfaction, 20% higher, to be exact. Satisfied customers become more engaged and develop loyalty toward a brand, enabling enterprises to build trust and retain more customers.

Increased customer conversions

Delivering personalized recommendations encourages users to view and click through more items, which might eventually convert perusers into purchasers. McKinsey found a 10%–15% rise in sales-conversion rates from more positive and personalized customer experiences.

A boost in revenue

Customer conversions drive sales, and sales drive revenue. McKinsey notes that 35% of what shoppers buy on Amazon comes from product recommendations (link resides outside ibm.com). Meanwhile, Netflix estimates savings of more than USD 1 billion due to its recommender system.

Challenges of recommendation engines

Recommendation systems have their limitations and pose challenges for organizations. Here are a few of the most common ones:

Cost and complexity

Recommendation engines entail analyzing and filtering massive amounts of data. This requires complex architectures and a significant investment in computing resources.

Scale and speed

Recommender systems need to be fast enough to determine and display the right recommendations in real time. This becomes even harder when real-time suggestions are served simultaneously to hundreds or thousands of users, let alone millions of them.

Irrelevant recommendations

Optimizing machine learning algorithms around the wrong metrics can lead to irrelevant recommendations. Items that are often highly rated might be suggested more frequently than new or obscure ones or those with fewer reviews. However, the most popular or most purchased items might not be what customers are interested in.

Bias

Machine learning algorithms might learn societal biases present in data—be it the collected data, the tagged data, the training data or external data sources—or from human evaluators who tune the model. This can result in inaccurate recommendations.

Learn more about AI bias
Privacy and compliance

Some users might opt out of an enterprise’s data-gathering initiatives for recommendation systems due to privacy concerns. Businesses might also need to consider regulatory requirements and compliance standards when collecting and storing data.

Use cases for recommendation engines

Here are some ways enterprises can use recommendation systems. Other use cases and applications might crop up as recommender technology evolves.

E-commerce and retail

Retailers and online sellers can use recommendation engines to amplify sales. Recommenders can suggest items other shoppers also bought or products that go well with what a shopper has already ordered.

Recommendation systems can be used for location-based or seasonal recommendations, and to promote new products or discounted items. These systems can also be used to increase the reach of seldom-bought items, recommending them as a bundle or as frequently bought products along with more popular ones.

Amazon is a prime example of an e-commerce company that uses recommendation engines.

Media and entertainment

Based on historical data and user preferences, recommendation systems can suggest relevant content users are likely to engage with and enjoy.

Whether it’s the next TV series to watch, an ebook to read, an artist to listen to, a game to play or a concert to attend, these systems enhance user experience by providing personalized recommendations.

Netflix, Spotify and YouTube are companies in the media and entertainment space employing recommenders.

Travel and hospitality

In the travel and hospitality realm, recommendation engines can suggest hotel and accommodation options, restaurants, activities and experiences according to a person’s budget and travel history.

These personalized trip recommendations improve customer satisfaction by addressing the needs of travelers.

Marketing

Organizations can use recommendation engines to expand their qualified leads. Recommenders can suggest content, such as blog posts, case studies, webinars or white papers, to introduce new services and entice prospective customers.

Marketing teams can then deliver these suggestions via newsletters, social media ads and targeted email messages.

AIOps

AIOps or artificial intelligence for IT operations, applies AI to automate and streamline IT service management and operational workflows.

Recommendation engines can be used in AIOps to suggest solutions, helping IT operations teams act swiftly and respond appropriately to technical issues.

Related solutions
IBM® watsonx.ai™

Train, validate, tune and deploy generative AI, foundation models and machine learning capabilities with ease and build AI applications in a fraction of the time with a fraction of the data. IBM watsonx.ai brings together new generative AI capabilities, powered by foundation models and traditional machine learning, into a powerful studio spanning the AI lifecycle. 

Explore IBM watsonx.ai

IBM Watson® natural language processing solutions

Accelerate the business value of AI with a powerful and flexible portfolio of libraries, services and applications.

Explore IBM natural language processing solutions

Data management solutions

Design a data strategy that eliminates data silos, reduces complexity and improves data quality for exceptional customer and employee experiences.

Explore IBM data management solutions

IBM Cloud Pak® for Data

Predict outcomes faster by using a platform built with data fabric architecture. Collect, organize and analyze data, no matter where it resides.

Explore IBM Cloud Pak for Data
Resources Build a recommendation engine with IBM Watson® Natural Language Understanding

Use IBM Watson® Knowledge Studio to create a customized language analysis model for a specific domain.

IBM and a department store build a recommender

Walk through a real-world example of implementing a recommendation system in this case study.

Use AI recommendation models for insurance claims with Watson AI

Learn how IBM Case Manager can use AI recommendation models to check the validity of insurance claims by using IBM Watson® Machine Learning, IBM Watson® Visual Recognition and IBM Watson® OpenScale.

10 everyday machine learning use cases

Explore some real-world applications of machine learning that have become part of our everyday lives.

What is predictive analytics?

Discover what predictive analytics is, its benefits, types and use cases.

What is a modern data platform?

Explore in-depth how a modern data platform enables the transformation and analysis of an organization’s data to help improve decision-making.

Take the next step

Train, validate, tune and deploy generative AI, foundation models and machine learning capabilities with IBM watsonx.ai, a next-generation enterprise studio for AI builders. Build AI applications in a fraction of the time with a fraction of the data.

Explore watsonx.ai Book a live demo