Advancing Natural Language Processing for Enterprise Domains

Share this post:

Finding information in a company’s vast trove of documents and knowledge bases to answer users’ questions is never as easy as it should be. The answers may very well exist, but they often remain out of reach for a number of reasons.

For starters, unlike the Web, where information is connected through a rich set of links and is often captured redundantly in multiple forms (making it easier to find), enterprise content is usually stored in silos with much less repetition of key information. In addition, users searching enterprise content typically ask intricate questions and expect more detailed answers than they would get from a Web search engine. These may include questions about product support, one’s bills, the latest regulation as it applies to contracts with customers, the implications of events discovered in news sites and so on. Finally, enterprises are often reluctant to rely on ‘black box’ AI that can’t explain its recommendations and may require techniques that are explainable to decision makers or end-users.

Enterprise NLPNatural language processing (NLP) holds great promise to help find such deep information in enterprise content by allowing users to more freely express their information needs and providing accurate answers to increasingly complex questions. However, enterprise NLP systems are often challenged by a number of factors, which include making sense of heterogonous silos of information, dealing with incomplete data, training accurate models from small amounts of data and navigating a changing environment in which new content, products, terms and other information is continuously being added.

IBM Research AI is exploring along three different themes to tackle these challenges and improve NLP for enterprise domains. The first seeks to advance AI where systems can learn from small amounts of data, leverage external knowledge and use techniques that include neurosymbolic approaches to language that combine neural and symbolic processing. The second focuses on trusting AI where explainability on how a system reaches a decision is provided. The third approach involves scaling AI to allow continuous adaptation and better monitoring and testing of systems to support the deployment of language systems under the rigorous expectations of enterprises.

In my post on Towards Data Science, I provide specifics on IBM Research’s enterprise NLP work by highlighting four papers we’re presenting at the ACL 2019 conference (a complete list of all our ACL papers is here). The first two papers address semantic parsing: the first uses Abstract Meaning Representation (AMR) language to represent the meaning of a sentence, and the second creates a semantic parser that converts the user’s question into a program to query a knowledge base. I also briefly explore our work integrating incomplete knowledge bases with text to improve the coverage in answering questions. The fourth paper describes a system enabling subject matter experts to fine tune the rules for an interpretable rules-based system.

Read my entire Towards Data Science article, here.

IBM Fellow & CTO Translation Technologies, IBM Research

More AI stories

Characterizing the Evolution of Tasks Within Occupations

The team at the IBM-MIT Watson AI Lab presented a new study at the AAAI Conference on AI, Ethics, and Society on “Learning Occupational Task-Shares Dynamics for the Future of Work” that shows how to predict changes in the economy’s demand for different tasks.

Continue reading

Deriving Complex Insights from Event-driven Continuous Time Bayesian Networks

Real-world decision making often involves situations and systems whose uncertain and inter-dependent variables interact in a complex and dynamic way. Additionally, many scenarios are influenced by external events that affect how system variables evolve. To address these complex scenarios for decision making, together with colleagues at the IBM T. J. Watson Research Center, we have developed a new dynamic, probabilistic graphical model called - Event-driven Continuous Time Bayesian Networks.

Continue reading

Progressing IBM Project Debater at AAAI-20 — and Beyond

At the thirty-fourth AAAI conference on Artificial Intelligence (AAAI-20), we will present two papers on recent advancements in Project Debater on two core tasks, both utilizing BERT.

Continue reading