AI

Causal Knowledge Extraction: An Evaluation using Automated Binary Causal Question Answering

Share this post:

The extraction of cause-effect relationships from text is an important task in knowledge discovery with numerous applications in medicine, finance, scientific discovery, and risk management.  At the 28th International Joint Conference on Artificial Intelligence (IJCAI’19), we present new results on causal knowledge extraction from large amounts of text for applications in enterprise risk management in our paper, “Answering Binary Causal Questions Through Large-Scale Text Mining: An Evaluation Using Cause-Effect Pairs from Human Experts.”

The system employs a novel approach for extracting causal statements using state-of-the-art neural network-based techniques to provide an unsupervised method of answering questions in the form “Could cause Y?,” where and are phrases without any semantic constraints.

This extracted causal knowledge is used to build a domain model for scenario planning and decision support. This work is inspired by a core problem in our work on Scenario Planning, where users need to provide the system with a causal “mind map” — a graphical representation of important “drivers” (events or conditions) and potentially causal links between them. This is a complex task that requires deep expertise in the target domain and applications and many iterations involving hours of experts’ time.

A screenshot showing the outcome of scenario planning in IBM Scenario Planning Advisor project. The indicators/risks/implications are “drivers” (events or conditions) that are manually defined and linked, a tedious task that our work aims to automate.

A screenshot showing the outcome of scenario planning in IBM Scenario Planning Advisor project. The indicators/risks/implications are “drivers” (events or conditions) that are manually defined and linked, a tedious task that our work aims to automate.

Our causal knowledge extraction system is capable of ingesting a large corpus of text, e.g. a corpus of 180 million news articles, extracting causal statements, and building index structures to effectively support automated question answering.

We evaluated three classes of question answering methods:

1) Methods that rely on statistical analysis of the full corpus, used as baseline methods.

2) Methods that rely on extraction of cause-effect pairs from sentences in the input corpus and building indices for querying them. In particular, our “DCC-Embed” method uses a semantic index of all phrases in the input corpus. This index is built using a neural network-based embeddings model trained over the input corpus.

3) Methods that rely on a semantic index of causal sentences.

The “NLM-BERT” method uses the newly popular Bidirectional Encoder Representations from Transformers (BERT) Neural Language Model to turn the sentences into vector representations to enable semantic querying. Our DCC-Embed and NLM-BERT methods showed very promising results in our evaluation.

Our “DCC-Embed” method results over the results reported by [Sharp et al., 2016]: R. Sharp, M. Surdeanu, P. Jansen, P. Clark, and M. Hammond. Creating Causal Embeddings for Question Answering with Minimal Supervision. In EMNLP, 2016.

Our “DCC-Embed” method results over the results reported by [Sharp et al., 2016]: R. Sharp, M. Surdeanu, P. Jansen, P. Clark, and M. Hammond. Creating Causal Embeddings for Question Answering with Minimal Supervision. In EMNLP, 2016.

We faced a major challenge in evaluation of our binary causal question answering methods as the existing benchmarks had several limitations and did not address the requirements of our target applications in Enterprise Risk Management. As a result, we developed three novel benchmark data sets of cause-effect pairs and released them publicly (https://doi.org/10.5281/zenodo.3214925). We hope that these data sets will promote further research on causal knowledge extraction and binary causal question answering methods.

We intend to use the outcome of this work as a part of the IBM Scenario Planning Advisor [Sohrabi et al., 2018 [1], 2019 [2] to enable users to prune a large space of potential pairs, and to provide hints for each question posed to the user.

  1. Sohrabi, A. Riabov, M. Katz and O. Udrea. An AI Planning Solution to Scenario Generation for Enterprise Risk Management. Proceedings of the 32nd Conference on Artificial Intelligence (AAAI-18), 2018
  2. Sohrabi, M. Katz, O. Hassanzadeh, O. Udrea, M. D. Feblowitz, and A. Riabov. IBM scenario planning advisor: Plan recognition as AI planning in practice. AI Commun., 32(1):1–13, 2019.

 

IBM Research Staff Member

More AI stories

Pushing the boundaries of convex optimization

Convex optimization problems, which involve the minimization of a convex function over a convex set, can be approximated in theory to any fixed precision in polynomial time. However, practical algorithms are known only for special cases. An important question is whether it is possible to develop algorithms for a broader subset of convex optimization problems that are efficient in both theory and practice.

Continue reading

Making Neural Networks Robust with New Perspectives

IBM researchers have partnered with scientists from MIT, Northeastern University, Boston University and University of Minnesota to publish two papers on novel attacks and defenses for graph neural networks and on a new robust training algorithm called hierarchical random switching at IJCAI 2019.

Continue reading

Improving the Scalability of AI Planning when Memory is Limited

We report new research results relevant to AI planning in our paper, "Depth-First Memory-Limited AND/OR Search and Unsolvability in Cyclic Search Spaces," presented at the International Joint Conference on Artificial Intelligence, IJCAI-19.

Continue reading