Advancing Natural Language Processing for Enterprise Domains

Share this post:

Finding information in a company’s vast trove of documents and knowledge bases to answer users’ questions is never as easy as it should be. The answers may very well exist, but they often remain out of reach for a number of reasons.

For starters, unlike the Web, where information is connected through a rich set of links and is often captured redundantly in multiple forms (making it easier to find), enterprise content is usually stored in silos with much less repetition of key information. In addition, users searching enterprise content typically ask intricate questions and expect more detailed answers than they would get from a Web search engine. These may include questions about product support, one’s bills, the latest regulation as it applies to contracts with customers, the implications of events discovered in news sites and so on. Finally, enterprises are often reluctant to rely on ‘black box’ AI that can’t explain its recommendations and may require techniques that are explainable to decision makers or end-users.

Enterprise NLPNatural language processing (NLP) holds great promise to help find such deep information in enterprise content by allowing users to more freely express their information needs and providing accurate answers to increasingly complex questions. However, enterprise NLP systems are often challenged by a number of factors, which include making sense of heterogonous silos of information, dealing with incomplete data, training accurate models from small amounts of data and navigating a changing environment in which new content, products, terms and other information is continuously being added.

IBM Research AI is exploring along three different themes to tackle these challenges and improve NLP for enterprise domains. The first seeks to advance AI where systems can learn from small amounts of data, leverage external knowledge and use techniques that include neurosymbolic approaches to language that combine neural and symbolic processing. The second focuses on trusting AI where explainability on how a system reaches a decision is provided. The third approach involves scaling AI to allow continuous adaptation and better monitoring and testing of systems to support the deployment of language systems under the rigorous expectations of enterprises.

In my post on Towards Data Science, I provide specifics on IBM Research’s enterprise NLP work by highlighting four papers we’re presenting at the ACL 2019 conference (a complete list of all our ACL papers is here). The first two papers address semantic parsing: the first uses Abstract Meaning Representation (AMR) language to represent the meaning of a sentence, and the second creates a semantic parser that converts the user’s question into a program to query a knowledge base. I also briefly explore our work integrating incomplete knowledge bases with text to improve the coverage in answering questions. The fourth paper describes a system enabling subject matter experts to fine tune the rules for an interpretable rules-based system.

Read my entire Towards Data Science article, here.

IBM Fellow & CTO Translation Technologies, IBM Research

More AI stories

New research helps make AI fairer in decision-making

To tackle bias in AI, our IBM Research team in collaboration with the University of Michigan has developed practical procedures and tools to help machine learning and AI achieve Individual Fairness. The key idea of Individual Fairness is to treat similar individuals well, similarly, to achieve fairness for everyone.

Continue reading

MIT and IBM announce ThreeDWorld Transport Challenge for physically realistic Embodied AI

MIT Brain and Cognitive Sciences, in collaboration with the MIT-IBM Watson AI Lab, has developed a new Embodied AI benchmark, the ThreeDWorld Transport Challenge, which aims to measure an Embodied AI agent’s ability to change the states of multiple objects to accomplish a complex task, performed within a photo- and physically realistic virtual environment.

Continue reading

Mimicking the brain: Deep learning meets vector-symbolic AI

To better simulate how the human brain makes decisions, we’ve combined the strengths of symbolic AI and neural networks. Specifically, we combined the learning representations that neural networks create with the symbol-like entities represented by high-dimensional and distributed vectors. The idea is to guide a neural network to represent unrelated objects with dissimilar high-dimensional vectors.

Continue reading