Mandarin Language Learners Get a Boost From AI

Share this post:

IBM Research and Rensselaer Polytechnic Institute (RPI) are collaborating on a new approach to help students learn Mandarin. The strategy pairs an AI-powered assistant with an immersive classroom environment that has not been used previously for language instruction. The classroom, called the Cognitive Immersive Room (CIR), makes students feel as though they are in restaurant in China, a garden, or a Tai Chi class, where they can practice speaking Mandarin with an AI chat agent. The CIR was developed by the Cognitive and Immersive Systems Lab (CISL), a research collaboration between IBM Research and RPI.

CISL researchers demonstrate an AI-assisted Mandarin Chinese language learning aid.

CISL researchers demonstrate an AI-assisted Mandarin Chinese language learning aid. Photo courtesy of Rensselaer Polytechnic Institute.

When learning a new language, especially one as difficult as Mandarin, it’s important that students have many opportunities to speak and practice their conversational skills. Acquiring a new language naturally, through cultural immersion, may be more effective than non-immersive practices. Yet, experiential learning is unavailable to most new learners of Mandarin. That’s why we developed the CIR.

An immersive gamification classroom environment

The CIR brings together several state-of-the-art technologies such as speech-to-text, natural language understanding, and computer vision to enable immersion and natural multi-modal dialogue. The room includes a 360-degree panoramic display system, an audio system, multiple cameras, multiple Kinect devices, and multiple microphones, as well as computer systems to support the AI technologies, some located in the room and others in the cloud.

an AI-assisted Mandarin Chinese language learning aid

Rahul Divekar, a Rensselaer computer science graduate student, demonstrates an AI-assisted Mandarin Chinese language learning aid under development at CISL. Photo courtesy of Rensselaer Polytechnic Institute.

Our goal is to combine cognitive, immersive technologies with game-playing elements to enable students to experience a cultural environment, practice daily tasks, and get help from intelligent agents. With the Mandarin Project, we use IBM Watson within the CIR as a conversational agent to engage students while they learn the language. Our approach involves IBM Watson speech recognition and natural language understanding technologies for English and Chinese.

One of the biggest obstacles in learning a foreign language through immersion is students’ fears of being judged by native speakers. In the CIR, however, students can work with virtual conversation partners to practice vocabulary and pronunciation without the pressures of a real-world setting.

Many language learning researchers are working with virtual or augmented reality, but we are investigating human-scale, immersive, gaming environments where students can physically walk around without having to wear specialized equipment. This reflects a broader trend in human-computer interaction, as our engagement with information in our daily lives becomes ever more immersive and our interactions with intelligent machines shift towards partnership.

Our system was tested in a Chinese 1 class on RPI’s campus in the late fall of 2017. This fall, we will continue using AI as a teaching tool in Chinese 1, 2 and 3 classes, with the goal of offering a new class merging Chinese 1-3 in the summer of 2019.

The research behind—and beyond—the classroom

As part of the IBM AI Horizons Network, CISL creates and uses such technologies as context-aware systems, pitch tone contour analysis, language-switch between Chinese and English in a natural language conversation between people and AI agents, multi-user multimodal interaction, reasoning and planning in the decision-making process and multimodal storytelling with automatic generation of narrative, visualization and sonification.

The CIR is one of four use cases in CISL. The lab is also working to augment group intelligence for critical decision-making in real-world environments like boardrooms. In the future, these new cognitive and immersive systems could fundamentally change the way people live and work. The early prototypes of CIR have also provided opportunities for user interaction research, enabling us to find and address new challenges in language learning.

More Education stories

IBM Releases Novel AI-Powered Technologies to Help Health and Research Community Accelerate the Discovery of Medical Insights and Treatments for COVID-19

IBM Research is making available multiple novel, free resources to help healthcare researchers, doctors and scientists around the world accelerate COVID-19 drug discovery.

Continue reading

Meet the women in IBM Research securing your digital future

This is our fifth and final blog post in a series for Women’s History Month 2020 focused on women innovating the future of IBM Research. The tremendous power of digital technology also introduces risk. If it’s hacked, or falls into the wrong hands, it can be used against us. That’s why security research is indispensable. The four women we meet here represent every aspect of it, from blockchain and open source defense to erecting cloud-based fortifications around digital crown jewels.

Continue reading

Parquet Modular Encryption: Developing a new open standard for big data security

To help advance data security in the cloud, IBM Research has initiated and currently leads joint work with the Apache Parquet community to address critical issues in securing confidentiality and integrity of sensitive data.

Continue reading