Advancing Natural Language Processing for Enterprise Domains

Share this post:

Finding information in a company’s vast trove of documents and knowledge bases to answer users’ questions is never as easy as it should be. The answers may very well exist, but they often remain out of reach for a number of reasons.

For starters, unlike the Web, where information is connected through a rich set of links and is often captured redundantly in multiple forms (making it easier to find), enterprise content is usually stored in silos with much less repetition of key information. In addition, users searching enterprise content typically ask intricate questions and expect more detailed answers than they would get from a Web search engine. These may include questions about product support, one’s bills, the latest regulation as it applies to contracts with customers, the implications of events discovered in news sites and so on. Finally, enterprises are often reluctant to rely on ‘black box’ AI that can’t explain its recommendations and may require techniques that are explainable to decision makers or end-users.

Enterprise NLPNatural language processing (NLP) holds great promise to help find such deep information in enterprise content by allowing users to more freely express their information needs and providing accurate answers to increasingly complex questions. However, enterprise NLP systems are often challenged by a number of factors, which include making sense of heterogonous silos of information, dealing with incomplete data, training accurate models from small amounts of data and navigating a changing environment in which new content, products, terms and other information is continuously being added.

IBM Research AI is exploring along three different themes to tackle these challenges and improve NLP for enterprise domains. The first seeks to advance AI where systems can learn from small amounts of data, leverage external knowledge and use techniques that include neurosymbolic approaches to language that combine neural and symbolic processing. The second focuses on trusting AI where explainability on how a system reaches a decision is provided. The third approach involves scaling AI to allow continuous adaptation and better monitoring and testing of systems to support the deployment of language systems under the rigorous expectations of enterprises.

In my post on Towards Data Science, I provide specifics on IBM Research’s enterprise NLP work by highlighting four papers we’re presenting at the ACL 2019 conference (a complete list of all our ACL papers is here). The first two papers address semantic parsing: the first uses Abstract Meaning Representation (AMR) language to represent the meaning of a sentence, and the second creates a semantic parser that converts the user’s question into a program to query a knowledge base. I also briefly explore our work integrating incomplete knowledge bases with text to improve the coverage in answering questions. The fourth paper describes a system enabling subject matter experts to fine tune the rules for an interpretable rules-based system.

Read my entire Towards Data Science article, here.

IBM Fellow & CTO Translation Technologies, IBM Research

More AI stories

MIT-IBM Watson AI Lab Welcomes Inaugural Members

Two years in, and the MIT-IBM Watson AI Lab is now engaging with leading companies to advance AI research. Today, the Lab announced its new Membership Program with Boston Scientific, Nexplore, Refinitiv and Samsung as the first companies to join.

Continue reading

Adversarial Robustness 360 Toolbox v1.0: A Milestone in AI Security

IBM researchers published the first major release of the Adversarial Robustness 360 Toolbox (ART). Initially released in April 2018, ART is an open-source library for adversarial machine learning that provides researchers and developers with state-of-the-art tools to defend and verify AI models against adversarial attacks. ART addresses growing concerns about people’s trust in AI, specifically the security of AI in mission-critical applications.

Continue reading

Making Sense of Neural Architecture Search

It is no surprise that following the massive success of deep learning technology in solving complicated tasks, there is a growing demand for automated deep learning. Even though deep learning is a highly effective technology, there is a tremendous amount of human effort that goes into designing a deep learning algorithm.

Continue reading