At IBM, we’ve put a focus on developing and expanding enterprise natural language processing (NLP) capabilities designed to help your businesses unearth insights, answer questions and make more informed decisions – even with a small data set or lack of expertise.

While human language is simple enough for a child to grasp, it is incredibly complex for even the most advanced machines – the most challenging part of teaching AI to understand human intent is that it requires massive amounts of data, lots of time, and expertise.

When you ask a question, what are you really trying to say? What goal are you trying to achieve? What information are you really trying to access? Human language is full of nuance, resulting in many ways to express a particular intent. This can be problematic for most AI – like chatbots – which stumble when confronted with the complexity of syntax and latch onto specific words rather than the broader context.

To help enterprises address this challenge, IBM launched a new and improved natural language understanding (NLU) model in IBM watsonx Assistant for intent classification. The new intent detection algorithm is more accurate versus compared commercial solutions in benchmark testing. (1)

Bringing continued NLP advancements from IBM Research to IBM Watson

In addition, we are introducing new NLP advancements within IBM watsonx Assistant and Watson Discovery, now available in beta. Pioneered by IBM Research, the new capabilities are designed to improve the automation of AI and provide a higher degree of precision in NLP.

Reading Comprehension is a feature that returns a specific fact or short answer contained within a long passage. Today, Watson Discovery identifies the best “passages” that correspond with queries. Reading Comprehension retrieves a large number of candidate paragraphs from the set of enterprise documents, searches for an answer to the question at hand and returns the corresponding answers. Reading Comprehension applies contextual understanding to understand queries and leverages massive language models to extract specific answers from the document at hand – and then the user receives a confidence score that indicate how confident the system is in each answer.

This capability is ideal for organizations in the financial industry. For example, if you are trying to make a lending decision, you may need to identify precise facts in complex documents that you would normally be reading and reviewing manually. Previously, Watson Discovery would return suggested paragraphs. With Reading Comprehension, a user will be provided with the precise answer (i.e. “What is the term of this current loan?” “2.9%”), saving them the time of having to manually search through large portfolios of documents. This feature is now available in beta to select Watson Discovery users.

FAQ Extraction, currently available in beta, is a novel answer retrieval technique that crawls web pages to detect FAQs and question-answer pairs, using this content to provide concise, up-to-minute answers through watsonx Assistant.

FAQ Extraction is designed to work in watsonx Assistant’s Search Skill, which looks for answers to end-users’ questions in documentation. This functionality makes it more likely that end-users find the answers they need when interacting with AI-powered virtual agents.

For example, businesses may struggle to keep up with ever-changing public guidance around permitted returns to the workplace or re-opening brick and mortar stores. It would require enormous resources to keep AI-powered customer care solutions up to date without a mechanism like FAQ Extraction. Instead, watsonx Assistant can keep up with the latest information simply by knowing the URL of authoritative FAQ content.

Finally, Watson NLP solutions now support 10 additional languages. IBM Watson Discovery now supports Bosnian, Croatian, Danish, Finnish, Hebrew, Hindi, Norwegian (Bokmål), Norwegian (Nynorsk), Serbian, and Swedish, while Watson Natural Language Understanding (NLU) now provides support for Danish, Norwegian Bokmal, Norwegian Nynorsk, Finnish, Czech, Hebrew, Polish, and Slovak (for Keywords).

These advancements build on a pipeline of NLP innovation from IBM Research. Earlier this year, we announced that we were taking some of the core NLP technologies powering IBM Research’s Project Debater – including advanced sentiment analysis (idiom understanding), summarization, topic clustering and key point analysis — and commercializing them within IBM’s NLP products, like Watson Discovery.

These innovations can help businesses to further understand and derive real value from their business data, so they can make more informed decisions, and provide customers and employees with more efficient insights.

Statements regarding IBM’s future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.


(1) In November 2020, Jio Haptik Technologies, a conversational AI software company, published a technical paper in which they compared the performance of their product against similar offerings from Google, Microsoft, and RASA. The performance of the other commercial solutions aside from IBM watsonx Assistant was taken from the Arora et al. (2020) benchmarking study. IBM ran the same performance tests on IBM watsonx Assistant as were reported by Arora et al. for purposes of this analysis. IBM’s full results are available in this technical paper.


More from Artificial intelligence

IBM TechXchange underscores the importance of AI skilling and partner innovation

3 min read - Generative AI and large language models are poised to impact how we all access and use information. But as organizations race to adopt these new technologies for business, it requires a global ecosystem of partners with industry expertise to identify the right enterprise use-cases for AI and the technical skills to implement the technology. During TechXchange, IBM's premier technical learning event in Las Vegas last week, IBM Partner Plus members including our Strategic Partners, resellers, software vendors, distributors and service…

Generative AI as a catalyst for change in the telecommunications industry

4 min read - Generative artificial intelligence (AI) burst into the mainstream in 2023, lighting a fire under businesses to integrate enterprise-grade versions into their processes. By 2024, 60% of C-suite executives are planning to pilot or operate generative AI in some way, indicating that generative AI's public-facing platforms have awakened the world to its groundbreaking capabilities For Communications Service Providers (CSPs) and Network Equipment Providers (NEPs), in particular, generative AI holds tremendous potential to help improve all manner of operations and customer engagement.…

AI code-generation software: What it is and how it works

4 min read - Using generative artificial intelligence (AI) solutions to produce computer code helps streamline the software development process and makes it easier for developers of all skill levels to write code. The user enters a text prompt describing what the code should do, and the generative AI code development tool automatically creates the code. It can also modernize legacy code and translate code from one programming language to another. By infusing artificial intelligence into the developer toolkit, these solutions can produce high-quality…

Data science vs data analytics: Unpacking the differences

5 min read - Though you may encounter the terms “data science” and “data analytics” being used interchangeably in conversations or online, they refer to two distinctly different concepts. Data science is an area of expertise that combines many disciplines such as mathematics, computer science, software engineering and statistics. It focuses on data collection and management of large-scale structured and unstructured data for various academic and business applications. Meanwhile, data analytics is the act of examining datasets to extract value and find answers to…