IBM Research addressing Enterprise NLP challenges in 2020

Share this post:

The field of Natural Language Processing (NLP) has made large strides over the last decade. In fact, NLP is so common in today’s AI applications that whether consumers are communicating with a virtual assistant, asking for travel directions or searching for weather predictions, chances are they’re interacting with some form of NLP.

This technology, however, still faces significant hurdles. At its core, NLP attempts to help an AI communicate with humans in natural language. Yet, for an NLP system to master language, it must be able to both generalize and reason over new text and recognize relationships between different words in context, all while adapting to the unique cultural nuances and idioms that encompass human communication. This is an incredibly difficult task and the reason today’s NLP landscape is active with many researchers working to find solutions.

At IBM Research AI, we’re focused on creating state-of-the-art NLP for enterprises. As opposed to NLP in business-to-consumer applications like Apple’s Siri or Amazon’s Alexa, our mission is to empower businesses to deploy and scale sophisticated AI systems that leverage NLP with cutting-edge accuracy and efficiency, all while requiring less data and human oversight.

NLP in the world today

There has been significant progress in basic NLP tasks over the past few years. This progress is mostly due to deep learning and transformer technology, which creates very large deep learning models with billions of parameters that can “mimic” language based on very large data sets. This technology, however, cannot deal with more advanced tasks that require reasoning and deep understanding of language. And, the compute resources required to train and leverage such models are prohibitive in many cases.

NLP in the enterprise introduces a number of additional challenges:

  • Extraction of meaning from a variety of complex, multi-format documents. Enterprise information typically resides in PDF documents. A simple PDF is difficult to process, for example, because such documents come in many formats, leverage structure (including titles, sections, headers and lists) to convey meaning, and important information is often included in embedded tables, diagrams, charts and figures.
  • Support for multiple languages. Unlike numbers and images, language varies from country to country and even within specific regions within the same country. As a result, enterprise NLP solutions must work in many languages and without the need to undergo retraining each time they encounter a new language.
  • Empowerment of subject matter experts. For automated systems to be effective, they must capture the knowledge of an organization’s lawyers, executives, customer support agents, marketers, HR employees and other professionals. Tools that enable these subject matter experts (SMEs) to easily and effectively customize NLP are critical because most companies do not have access to NLP-trained developers. And, if they do, those developers are often not familiar with the business’s specific semantics.
  • Integration of pre-existing, text-based knowledge. In addition to SMEs, a significant portion of an organization’s IP exists in the form of knowledge held in databases, applications and business processes. As a result, enterprise NLP solutions require additional processing to integrate this pre-existing information with the processing of textual content. Such integration is necessary for applications leveraging this textual content — including virtual agents, business process automation, and financial applications — to be more precise and effective.

Meeting the challenges

IBM Research is focused on advancing next-generation enterprise NLP in a number of ways to support multi-format documents, multiple languages, SME customization and knowledge extraction. This work includes improvements to basic NLP components as well as the addition of more advanced features. Our taxonomy includes:


  • Understand: The first step on the enterprise NLP ladder is understanding language at a basic level. This includes semantic comprehension of sentences, extracting key entities or concepts in a document and identifying the relations between the entities, and also understanding the ­­format of complex documents. Some of this work is already integrated in IBM’s AI offerings including Watson Discovery Smart Document Understanding (SDU) which is trained to extract and interpret custom fields and artifacts in documents. Additionally, IBM Research developed and released PubLayNet, the largest dataset ever aimed at document layout analysis (i.e., streamlining the process of extracting information embedded in PDF documents).
  • Classify: The second step is to classify the text or documents into higher-level constructs. These can include the overall sentiment of a document or parts of it — associating concepts within the document — or more generally element classification of sentences, paragraphs, tables, graphs and figures in the documents. In March 2020, IBM began integrating NLP featuresfrom IBM Research’s Project Debater into Watson DiscoveryWatson Assistant and Watson Core Services to allow enterprises to take advantage, for the first time, of advanced sentiment analysis, advanced topic clustering and customizable classification of elements in business documents.
  • Retrieve: Once the text is understood and classified, applications can exploit this and enable retrieval and exploration of the data. This could include fine-grained retrieval of documents, paragraphs, tables based on user queries, question answering (QA) or some form of visualization and navigation of the knowledge encapsulated in the documents. IBM Research recently deployed aQA system on the CORD-19 corpus of scientific articles on COVID-19 that demonstrates the functionality enterprises can expect on their own content with Watson Discovery when the QA technology is integrated. In addition, IBM Research introduced last fall its TechQA leaderboard — based on IBM Research’s top performing Go Ahead Ask Me Anything (GAAMA) system — the first leaderboard to address enterprise QA use cases.
  • Generate: Finally, new language may be generated. Examples of this include summaries that can be either single-document or across documents and in the context of a query. A broader use case becoming extremely popular is dialog generation, whereby NLP systems converse with humans trying to solve requests such as customer support questions, transactions or general guidance. IBM’s leading solution for dialog systems, Watson Assistant, is based on several innovations from IBM Research. We’re also currently developing more ways to automate the process of dialog generation through the analysis of log files, user manuals and Web pages that document the structure and content of a business’s communications. Notably, IBM Research summarization technology was used to provide live insights during the recent Grammy awardswhere it analyzed 18 million news articles, blogs and bios to add deeper context and information to the red-carpet livestream.

Holding NLP to a higher standard

IBM believes that while NLP models must deliver high accuracy, it’s equally critical to ensure they are explainable. An enterprise user must know how its AI arrived at a decision. Explainability and transparency also facilitate trust, which we believe is one of the most important factors in making an AI successful. That is why we have an entire area dedicated to what we call Trusted AI that is developing solutions to trust the output of an AI.

IBM is also enabling trust by allowing enterprise SMEs to easily and effectively customize the NLP for their needs, without special training or complex layers of guidance and information to learn. Referred to as “human in the loop”, these steps increase accuracy and transparency of an NLP system by allowing an enterprise user to feel comfortable deploying the solution and understand the outputs. One example of this work is HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop.

While customization of NLP models is important to facilitate trust, it is also crucial from a process standpoint. Each company, industry and use case has its own language, a unique taxonomy of concepts and relationships, commonly accepted short forms and abbreviations, accepted patterns of communication and challenges to conquer. As a result, these systems must be easily adaptable to understand and process these individualized needs, which is why IBM is building NLP that can streamline and expedite the process of searching for, identifying, gathering and ultimately returning information and insights.

The Road Ahead

IBM Research AI is proud to continue its role as a leader in enterprise NLP. And, we’re looking forward to pushing the boundaries of NLP even further in the coming months and years.

Some of our notable near-term efforts include:

  • COVID-19: Like most businesses today, IBM is pledging its time, energy and resources to helping fight the COVID-19 pandemic. And, our NLP technology has fueled multiple tools and resources released by IBM recently, including:
  • ACL 2020: We’re looking forward to having a robust presence at the 2020 Annual Conference of the Association for Computational Linguistics (ACL), taking place July 5–10, and showcasing a number of accepted papers in key areas of NLP including: automatic argument summarization (leveraging technology from IBM’s Project Debater), Frequently Asked Questions (FAQ) retrieval, automatic taxonomy construction, insight analysis of the BERT (Bidirectional Encoder Representations from Transformers) NLP technique and automatic question generation (QG) for fast domain adaptation of QA systems.

To learn more about IBM Research AI and our work in NLP, visit:


IBM Fellow & CTO Translation Technologies, IBM Research

Aya Soffer

Vice President of AI Tech, IBM Research AI

More AI stories

We’ve moved! The IBM Research blog has a new home

In an effort better integrate the IBM Research blog with the IBM Research web experience, we have migrated to a new landing page:

Continue reading

Pushing the boundaries of human-AI interaction at IUI 2021

At the 2021 virtual edition of the ACM International Conference on Intelligent User Interfaces (IUI), researchers at IBM will present five full papers, two workshop papers, and two demos.

Continue reading

From HPC Consortium’s success to National Strategic Computing Reserve

Founded in March 2020 just as the pandemic’s wave was starting to wash over the world, the Consortium has brought together 43 members with supercomputing resources. Private and public enterprises, academia, government and technology companies, many of whom are typically rivals. “It is simply unprecedented,” said Dario Gil, Senior Vice President and Director of IBM Research, one of the founding organizations. “The outcomes we’ve achieved, the lessons we’ve learned, and the next steps we have to pursue are all the result of the collective efforts of these Consortium’s community.” The next step? Creating the National Strategic Computing Reserve to help the world be better prepared for future global emergencies.

Continue reading