AI

Causal Knowledge Extraction: An Evaluation using Automated Binary Causal Question Answering

Share this post:

The extraction of cause-effect relationships from text is an important task in knowledge discovery with numerous applications in medicine, finance, scientific discovery, and risk management.  At the 28th International Joint Conference on Artificial Intelligence (IJCAI’19), we present new results on causal knowledge extraction from large amounts of text for applications in enterprise risk management in our paper, “Answering Binary Causal Questions Through Large-Scale Text Mining: An Evaluation Using Cause-Effect Pairs from Human Experts.”

The system employs a novel approach for extracting causal statements using state-of-the-art neural network-based techniques to provide an unsupervised method of answering questions in the form “Could cause Y?,” where and are phrases without any semantic constraints.

This extracted causal knowledge is used to build a domain model for scenario planning and decision support. This work is inspired by a core problem in our work on Scenario Planning, where users need to provide the system with a causal “mind map” — a graphical representation of important “drivers” (events or conditions) and potentially causal links between them. This is a complex task that requires deep expertise in the target domain and applications and many iterations involving hours of experts’ time.

A screenshot showing the outcome of scenario planning in IBM Scenario Planning Advisor project. The indicators/risks/implications are “drivers” (events or conditions) that are manually defined and linked, a tedious task that our work aims to automate.

A screenshot showing the outcome of scenario planning in IBM Scenario Planning Advisor project. The indicators/risks/implications are “drivers” (events or conditions) that are manually defined and linked, a tedious task that our work aims to automate.

Our causal knowledge extraction system is capable of ingesting a large corpus of text, e.g. a corpus of 180 million news articles, extracting causal statements, and building index structures to effectively support automated question answering.

We evaluated three classes of question answering methods:

1) Methods that rely on statistical analysis of the full corpus, used as baseline methods.

2) Methods that rely on extraction of cause-effect pairs from sentences in the input corpus and building indices for querying them. In particular, our “DCC-Embed” method uses a semantic index of all phrases in the input corpus. This index is built using a neural network-based embeddings model trained over the input corpus.

3) Methods that rely on a semantic index of causal sentences.

The “NLM-BERT” method uses the newly popular Bidirectional Encoder Representations from Transformers (BERT) Neural Language Model to turn the sentences into vector representations to enable semantic querying. Our DCC-Embed and NLM-BERT methods showed very promising results in our evaluation.

Our “DCC-Embed” method results over the results reported by [Sharp et al., 2016]: R. Sharp, M. Surdeanu, P. Jansen, P. Clark, and M. Hammond. Creating Causal Embeddings for Question Answering with Minimal Supervision. In EMNLP, 2016.

Our “DCC-Embed” method results over the results reported by [Sharp et al., 2016]: R. Sharp, M. Surdeanu, P. Jansen, P. Clark, and M. Hammond. Creating Causal Embeddings for Question Answering with Minimal Supervision. In EMNLP, 2016.

We faced a major challenge in evaluation of our binary causal question answering methods as the existing benchmarks had several limitations and did not address the requirements of our target applications in Enterprise Risk Management. As a result, we developed three novel benchmark data sets of cause-effect pairs and released them publicly (https://doi.org/10.5281/zenodo.3214925). We hope that these data sets will promote further research on causal knowledge extraction and binary causal question answering methods.

We intend to use the outcome of this work as a part of the IBM Scenario Planning Advisor [Sohrabi et al., 2018 [1], 2019 [2] to enable users to prune a large space of potential pairs, and to provide hints for each question posed to the user.

  1. Sohrabi, A. Riabov, M. Katz and O. Udrea. An AI Planning Solution to Scenario Generation for Enterprise Risk Management. Proceedings of the 32nd Conference on Artificial Intelligence (AAAI-18), 2018
  2. Sohrabi, M. Katz, O. Hassanzadeh, O. Udrea, M. D. Feblowitz, and A. Riabov. IBM scenario planning advisor: Plan recognition as AI planning in practice. AI Commun., 32(1):1–13, 2019.

 

IBM Research Staff Member

More AI stories

We’ve moved! The IBM Research blog has a new home

In an effort better integrate the IBM Research blog with the IBM Research web experience, we have migrated to a new landing page: https://research.ibm.com/blog

Continue reading

Pushing the boundaries of human-AI interaction at IUI 2021

At the 2021 virtual edition of the ACM International Conference on Intelligent User Interfaces (IUI), researchers at IBM will present five full papers, two workshop papers, and two demos.

Continue reading

From HPC Consortium’s success to National Strategic Computing Reserve

Founded in March 2020 just as the pandemic’s wave was starting to wash over the world, the Consortium has brought together 43 members with supercomputing resources. Private and public enterprises, academia, government and technology companies, many of whom are typically rivals. “It is simply unprecedented,” said Dario Gil, Senior Vice President and Director of IBM Research, one of the founding organizations. “The outcomes we’ve achieved, the lessons we’ve learned, and the next steps we have to pursue are all the result of the collective efforts of these Consortium’s community.” The next step? Creating the National Strategic Computing Reserve to help the world be better prepared for future global emergencies.

Continue reading