Have you ever used a chatbot to get help? If you have, chances are that you’re all too familiar with answers like, “I’m sorry, I don’t understand” or “Can you try rephrasing?” With frustrating responses like these, many businesses still see chatbots as immature tools for customer service. The biggest impediment to automating interactions is correctly understanding the user’s need.

IBM watsonx Assistant uses machine learning and deep learning techniques to understand how to answer end-user questions accurately with relatively small data sets. The artificial intelligence at the core of watsonx Assistant is designed to correctly identify the countless permutations of intent in real-world interactions. In short, we designed watsonx Assistant to be easy to train and to recognize accurately what the user wants.

But we never settle. Our vision is for watsonx Assistant to be the heart of any company’s customer service operation. To do that, we continuously improve our AI, aiming to increase precision, decrease the amount of training data, and shorten the time to production. We’re excited to announce that watsonx Assistant has a new and improved intent detection algorithm, which is more accurate versus commercial and open-source solutions in a recently published benchmark (see Table). 1

Because of these improvements, the accuracy of watsonx Assistant’s latest model is 79%, up from 76.3% in the immediately previous version. This means a watsonx Assistant virtual agent can answer customer help requests much more often on its own without human agent involvement (known in the industry as containment), which can save money and increase user satisfaction.

Better AI for understanding customers

Before we dive into our latest performance analysis, let’s talk about the challenge of intent classification.

Consider the various ways customers can express their problem:

  • “I can’t log in”
  • “My password doesn’t work”
  •  “Forgot password”

The AI technology has to understand that the intent behind these sentences (and infinite variations of wording and misspellings) is getting help resetting the password. Even for a basic intent like this, it takes complex natural language processing (NLP) and classification techniques to get it right. Now imagine the complexity when trying to set up a system to help with, say, mortgage applications.


[1] In November 2020, Jio Haptik Technologies, a conversational AI software company, published a technical paper in which they compared the performance of their product against similar offerings from Google, Microsoft, and RASA. The performance of the other commercial solutions aside from IBM watsonx Assistant was taken from the Arora et al. (2020) benchmarking study. IBM ran the same performance tests on IBM watsonx Assistant as were reported by Arora et al. for purposes of this analysis. IBM’s full results are available in this technical paper: https://arxiv.org/pdf/2012.03929.pdf


There’s always room for improvement — and multiple ways to achieve it.

In the latest version of watsonx Assistant, we added AutoML. This is a technique that tries various algorithms and combinations of features and parameters to find the best results for a given data set without human intervention.

But AutoML requires a lot of computing power and time. So we supplemented it with meta-learning techniques that dramatically speed up and improve intent detection. These replace painstaking human-tweaked feature engineering and algorithm selection with an automated, data-driven process. Using meta-learning, our system observes how the different machine learning algorithms perform across various datasets — and learns how to adapt the algorithm to new datasets.

Finally, we fortified our transfer learning capabilities. These allow the system to transfer what it learned in one domain or task (say, understanding a user’s request to apply for a credit card) to a similar domain or task (applying for a mortgage).

The result: higher accuracy

Our work has resulted in improvements in accuracy while requiring even less data to train the models.

In November 2020, Jio Haptik Technologies, a conversational AI software company, published a technical paper in which they compared the performance of their product against similar commercial offerings from Google, Microsoft and RASA, as well as BERT, an open-source project sponsored by Google. While Haptik did not include watsonx Assistant in their analysis, we used the same publicly available data sets and experimental setup as Haptik to evaluate our performance, and we appended our results to their analysis:

According to the benchmark results, watsonx Assistant is 5.6 percentage points more accurate than Google Dialogflow, and 14.7 percentage points more accurate than Microsoft LUIS.

You can read the full findings in the IBM’s recently published technical paper, which provides additional detail around the improvements and testing methodology.

Better accuracy can mean better business results

With these enhancements, watsonx Assistant is able to improve containment rates (how often the AI solves customer help requests without intervention from human agents) and first contact resolution (how often the system resolves the problem with AI or human agents on first try).

In addition, intent recognition can cut the time to value. Setting up, configuring, and tweaking the performance of an AI-powered customer service system has traditionally taken weeks, if not months. But the new features are engineered to reduce the time and data required to bring watsonx Assistant to production.


CTA: Ready for more? Read about the newest NLP features in IBM watsonx Assistant.

Was this article helpful?

More from watsonx Assistant

Chatbot examples: A beginner’s guide 

7 min read - A chatbot is a program or script designed to interact and respond to humans in real-time conversation. Different organizations and individuals employ chatbots for a variety of different uses and business functions. Broadly, chatbots provide pre-written responses and information to handle basic requests or to get enough information from customers to connect them to a live agent for better and more specific service. More advanced chatbots use machine learning, artificial intelligence (AI) and generative AI technology to generate real-time responses…

Beyond basics: Six tips for an exceptional customer service strategy

7 min read - Enhancing the customer experience through customer service is among the most important disciplines for any organization for one simple reason: without customers, organizations would fail overnight. Customer service, sometimes called customer care or customer support, relates to the activities organizations take to ensure their customers’ needs are being met. While every customer interaction is different, organizations that want to improve customer retention and grow their customer base must create an effective customer service strategy. Doing so requires combining customization with…

Transform digital experiences and unlock productivity with advanced generative AI

2 min read - AI and automation are driving business transformation by empowering individuals to do work without expert knowledge of business processes and applications. Whether it’s an employee who knows what they need but doesn’t know how to do it, a knowledge worker who knows how to do the task but needs help doing it more efficiently, or a customer who wants to resolve an issue but struggles with self-help tools, artificial intelligence (AI) unlocks new levels of productivity by empowering individuals to…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters