Mandarin Language Learners Get a Boost From AI

Share this post:

IBM Research and Rensselaer Polytechnic Institute (RPI) are collaborating on a new approach to help students learn Mandarin. The strategy pairs an AI-powered assistant with an immersive classroom environment that has not been used previously for language instruction. The classroom, called the Cognitive Immersive Room (CIR), makes students feel as though they are in restaurant in China, a garden, or a Tai Chi class, where they can practice speaking Mandarin with an AI chat agent. The CIR was developed by the Cognitive and Immersive Systems Lab (CISL), a research collaboration between IBM Research and RPI.

CISL researchers demonstrate an AI-assisted Mandarin Chinese language learning aid.

CISL researchers demonstrate an AI-assisted Mandarin Chinese language learning aid. Photo courtesy of Rensselaer Polytechnic Institute.

When learning a new language, especially one as difficult as Mandarin, it’s important that students have many opportunities to speak and practice their conversational skills. Acquiring a new language naturally, through cultural immersion, may be more effective than non-immersive practices. Yet, experiential learning is unavailable to most new learners of Mandarin. That’s why we developed the CIR.

An immersive gamification classroom environment

The CIR brings together several state-of-the-art technologies such as speech-to-text, natural language understanding, and computer vision to enable immersion and natural multi-modal dialogue. The room includes a 360-degree panoramic display system, an audio system, multiple cameras, multiple Kinect devices, and multiple microphones, as well as computer systems to support the AI technologies, some located in the room and others in the cloud.

an AI-assisted Mandarin Chinese language learning aid

Rahul Divekar, a Rensselaer computer science graduate student, demonstrates an AI-assisted Mandarin Chinese language learning aid under development at CISL. Photo courtesy of Rensselaer Polytechnic Institute.

Our goal is to combine cognitive, immersive technologies with game-playing elements to enable students to experience a cultural environment, practice daily tasks, and get help from intelligent agents. With the Mandarin Project, we use IBM Watson within the CIR as a conversational agent to engage students while they learn the language. Our approach involves IBM Watson speech recognition and natural language understanding technologies for English and Chinese.

One of the biggest obstacles in learning a foreign language through immersion is students’ fears of being judged by native speakers. In the CIR, however, students can work with virtual conversation partners to practice vocabulary and pronunciation without the pressures of a real-world setting.

Many language learning researchers are working with virtual or augmented reality, but we are investigating human-scale, immersive, gaming environments where students can physically walk around without having to wear specialized equipment. This reflects a broader trend in human-computer interaction, as our engagement with information in our daily lives becomes ever more immersive and our interactions with intelligent machines shift towards partnership.

Our system was tested in a Chinese 1 class on RPI’s campus in the late fall of 2017. This fall, we will continue using AI as a teaching tool in Chinese 1, 2 and 3 classes, with the goal of offering a new class merging Chinese 1-3 in the summer of 2019.

The research behind—and beyond—the classroom

As part of the IBM AI Horizons Network, CISL creates and uses such technologies as context-aware systems, pitch tone contour analysis, language-switch between Chinese and English in a natural language conversation between people and AI agents, multi-user multimodal interaction, reasoning and planning in the decision-making process and multimodal storytelling with automatic generation of narrative, visualization and sonification.

The CIR is one of four use cases in CISL. The lab is also working to augment group intelligence for critical decision-making in real-world environments like boardrooms. In the future, these new cognitive and immersive systems could fundamentally change the way people live and work. The early prototypes of CIR have also provided opportunities for user interaction research, enabling us to find and address new challenges in language learning.

More Education stories

Kubernetes-based Control Plane to Manage Risk and Compliance for Hybrid Cloud

Today, traditional enterprises are looking for ways to leverage the cloud for their digital transformation. This is driven by the need to create new revenue streams, deliver superior user experience, and reduce capital and operational expenditure. Hybrid cloud has emerged as the new normal for traditional enterprises. This is demonstrated by the fact that 94% […]

Continue reading

IBM Research and The Michael J. Fox Foundation Develop Modeling Methodology to Help Understand Parkinson’s Disease Using Machine Learning

In collaboration with The Michael J. Fox Foundation for Parkinson’s Research, our team of researchers at IBM is aiming to develop improved disease progression models that can help clinicians understand how the disease progresses in relation to the emergence of symptoms, even when those patients are taking symptom-modifying medications.

Continue reading

AI Could Help Enable Accurate Remote Monitoring of Parkinson’s Patients

In a paper recently published in Nature Scientific Reports, IBM Research and scientists from several other medical institutions developed a new way to estimate the severity of a person’s Parkinson’s disease (PD) symptoms by remotely measuring and analyzing physical activity as motor impairment increased. Using data captured by wrist-worn accelerometers, we created statistical representations of […]

Continue reading