AI

Causal Knowledge Extraction: An Evaluation using Automated Binary Causal Question Answering

Share this post:

The extraction of cause-effect relationships from text is an important task in knowledge discovery with numerous applications in medicine, finance, scientific discovery, and risk management.  At the 28th International Joint Conference on Artificial Intelligence (IJCAI’19), we present new results on causal knowledge extraction from large amounts of text for applications in enterprise risk management in our paper, “Answering Binary Causal Questions Through Large-Scale Text Mining: An Evaluation Using Cause-Effect Pairs from Human Experts.”

The system employs a novel approach for extracting causal statements using state-of-the-art neural network-based techniques to provide an unsupervised method of answering questions in the form “Could cause Y?,” where and are phrases without any semantic constraints.

This extracted causal knowledge is used to build a domain model for scenario planning and decision support. This work is inspired by a core problem in our work on Scenario Planning, where users need to provide the system with a causal “mind map” — a graphical representation of important “drivers” (events or conditions) and potentially causal links between them. This is a complex task that requires deep expertise in the target domain and applications and many iterations involving hours of experts’ time.

A screenshot showing the outcome of scenario planning in IBM Scenario Planning Advisor project. The indicators/risks/implications are “drivers” (events or conditions) that are manually defined and linked, a tedious task that our work aims to automate.

A screenshot showing the outcome of scenario planning in IBM Scenario Planning Advisor project. The indicators/risks/implications are “drivers” (events or conditions) that are manually defined and linked, a tedious task that our work aims to automate.

Our causal knowledge extraction system is capable of ingesting a large corpus of text, e.g. a corpus of 180 million news articles, extracting causal statements, and building index structures to effectively support automated question answering.

We evaluated three classes of question answering methods:

1) Methods that rely on statistical analysis of the full corpus, used as baseline methods.

2) Methods that rely on extraction of cause-effect pairs from sentences in the input corpus and building indices for querying them. In particular, our “DCC-Embed” method uses a semantic index of all phrases in the input corpus. This index is built using a neural network-based embeddings model trained over the input corpus.

3) Methods that rely on a semantic index of causal sentences.

The “NLM-BERT” method uses the newly popular Bidirectional Encoder Representations from Transformers (BERT) Neural Language Model to turn the sentences into vector representations to enable semantic querying. Our DCC-Embed and NLM-BERT methods showed very promising results in our evaluation.

Our “DCC-Embed” method results over the results reported by [Sharp et al., 2016]: R. Sharp, M. Surdeanu, P. Jansen, P. Clark, and M. Hammond. Creating Causal Embeddings for Question Answering with Minimal Supervision. In EMNLP, 2016.

Our “DCC-Embed” method results over the results reported by [Sharp et al., 2016]: R. Sharp, M. Surdeanu, P. Jansen, P. Clark, and M. Hammond. Creating Causal Embeddings for Question Answering with Minimal Supervision. In EMNLP, 2016.

We faced a major challenge in evaluation of our binary causal question answering methods as the existing benchmarks had several limitations and did not address the requirements of our target applications in Enterprise Risk Management. As a result, we developed three novel benchmark data sets of cause-effect pairs and released them publicly (https://doi.org/10.5281/zenodo.3214925). We hope that these data sets will promote further research on causal knowledge extraction and binary causal question answering methods.

We intend to use the outcome of this work as a part of the IBM Scenario Planning Advisor [Sohrabi et al., 2018 [1], 2019 [2] to enable users to prune a large space of potential pairs, and to provide hints for each question posed to the user.

  1. Sohrabi, A. Riabov, M. Katz and O. Udrea. An AI Planning Solution to Scenario Generation for Enterprise Risk Management. Proceedings of the 32nd Conference on Artificial Intelligence (AAAI-18), 2018
  2. Sohrabi, M. Katz, O. Hassanzadeh, O. Udrea, M. D. Feblowitz, and A. Riabov. IBM scenario planning advisor: Plan recognition as AI planning in practice. AI Commun., 32(1):1–13, 2019.

 

IBM Research Staff Member

More AI stories

High quality, lightweight and adaptable Text-to-Speech (TTS) using LPCNet

Recent advances in deep learning are dramatically improving the development of Text-to-Speech systems through more effective and efficient learning of voice and speaking styles of speakers and more natural generation of high-quality output speech.

Continue reading

IBM Research AI at INTERSPEECH 2019

IBM Research's papers at INTERSPEECH 2019 showcase our focus on improving the underlying speech technologies that enable companies provide their customers with a uniformly good experience across different channels and extract actionable insights from these interactions.

Continue reading

IBM Project Debater Demonstrates the Future of Democracy in Switzerland

Can Artificial Intelligence (AI) capture the narrative of a community on a controversial topic to provide an unbiased outcome? Recently, the citizens of Lugano, a city of more than 60,000 citizens on the Swiss-Italian border, provided the answer. What is Project Debater? In February 2019, IBM unveiled Project Debater to the world. It’s the first ever AI technology […]

Continue reading