AI

Causal Knowledge Extraction: An Evaluation using Automated Binary Causal Question Answering

Share this post:

The extraction of cause-effect relationships from text is an important task in knowledge discovery with numerous applications in medicine, finance, scientific discovery, and risk management.  At the 28th International Joint Conference on Artificial Intelligence (IJCAI’19), we present new results on causal knowledge extraction from large amounts of text for applications in enterprise risk management in our paper, “Answering Binary Causal Questions Through Large-Scale Text Mining: An Evaluation Using Cause-Effect Pairs from Human Experts.”

The system employs a novel approach for extracting causal statements using state-of-the-art neural network-based techniques to provide an unsupervised method of answering questions in the form “Could cause Y?,” where and are phrases without any semantic constraints.

This extracted causal knowledge is used to build a domain model for scenario planning and decision support. This work is inspired by a core problem in our work on Scenario Planning, where users need to provide the system with a causal “mind map” — a graphical representation of important “drivers” (events or conditions) and potentially causal links between them. This is a complex task that requires deep expertise in the target domain and applications and many iterations involving hours of experts’ time.

A screenshot showing the outcome of scenario planning in IBM Scenario Planning Advisor project. The indicators/risks/implications are “drivers” (events or conditions) that are manually defined and linked, a tedious task that our work aims to automate.

A screenshot showing the outcome of scenario planning in IBM Scenario Planning Advisor project. The indicators/risks/implications are “drivers” (events or conditions) that are manually defined and linked, a tedious task that our work aims to automate.

Our causal knowledge extraction system is capable of ingesting a large corpus of text, e.g. a corpus of 180 million news articles, extracting causal statements, and building index structures to effectively support automated question answering.

We evaluated three classes of question answering methods:

1) Methods that rely on statistical analysis of the full corpus, used as baseline methods.

2) Methods that rely on extraction of cause-effect pairs from sentences in the input corpus and building indices for querying them. In particular, our “DCC-Embed” method uses a semantic index of all phrases in the input corpus. This index is built using a neural network-based embeddings model trained over the input corpus.

3) Methods that rely on a semantic index of causal sentences.

The “NLM-BERT” method uses the newly popular Bidirectional Encoder Representations from Transformers (BERT) Neural Language Model to turn the sentences into vector representations to enable semantic querying. Our DCC-Embed and NLM-BERT methods showed very promising results in our evaluation.

Our “DCC-Embed” method results over the results reported by [Sharp et al., 2016]: R. Sharp, M. Surdeanu, P. Jansen, P. Clark, and M. Hammond. Creating Causal Embeddings for Question Answering with Minimal Supervision. In EMNLP, 2016.

Our “DCC-Embed” method results over the results reported by [Sharp et al., 2016]: R. Sharp, M. Surdeanu, P. Jansen, P. Clark, and M. Hammond. Creating Causal Embeddings for Question Answering with Minimal Supervision. In EMNLP, 2016.

We faced a major challenge in evaluation of our binary causal question answering methods as the existing benchmarks had several limitations and did not address the requirements of our target applications in Enterprise Risk Management. As a result, we developed three novel benchmark data sets of cause-effect pairs and released them publicly (https://doi.org/10.5281/zenodo.3214925). We hope that these data sets will promote further research on causal knowledge extraction and binary causal question answering methods.

We intend to use the outcome of this work as a part of the IBM Scenario Planning Advisor [Sohrabi et al., 2018 [1], 2019 [2] to enable users to prune a large space of potential pairs, and to provide hints for each question posed to the user.

  1. Sohrabi, A. Riabov, M. Katz and O. Udrea. An AI Planning Solution to Scenario Generation for Enterprise Risk Management. Proceedings of the 32nd Conference on Artificial Intelligence (AAAI-18), 2018
  2. Sohrabi, M. Katz, O. Hassanzadeh, O. Udrea, M. D. Feblowitz, and A. Riabov. IBM scenario planning advisor: Plan recognition as AI planning in practice. AI Commun., 32(1):1–13, 2019.

 

IBM Research Staff Member

More AI stories

IBM Research and The Michael J. Fox Foundation Develop Modeling Methodology to Help Understand Parkinson’s Disease Using Machine Learning

In collaboration with The Michael J. Fox Foundation for Parkinson’s Research, our team of researchers at IBM is aiming to develop improved disease progression models that can help clinicians understand how the disease progresses in relation to the emergence of symptoms, even when those patients are taking symptom-modifying medications.

Continue reading

AI Could Help Enable Accurate Remote Monitoring of Parkinson’s Patients

In a paper recently published in Nature Scientific Reports, IBM Research and scientists from several other medical institutions developed a new way to estimate the severity of a person’s Parkinson’s disease (PD) symptoms by remotely measuring and analyzing physical activity as motor impairment increased. Using data captured by wrist-worn accelerometers, we created statistical representations of […]

Continue reading

Image Captioning as an Assistive Technology

IBM Research's Science for Social Good team recently participated in the 2020 VizWiz Grand Challenge to design and improve systems that make the world more accessible for the blind.

Continue reading