AI Enables Foreign Language Study Abroad, No Travel Required

Share this post:

A student learning to speak Mandarin wanders into a marketplace on the streets of China on a sunny summer afternoon. Before long, two vendors approach and begin hawking products, trying to outbid one another. The student must now grasp what’s being said and formulate an appropriate response using proper pronunciation to avoid being misunderstood. It’s a challenging, yet common, scenario for anyone trying to learn a new language by interacting with native speakers and immersing themselves in a foreign culture.

Fortunately, the student in this case is able to pause the unfolding scenario to check the accuracy and tone of her planned response. That’s because, even though the student appears to be surrounded by the sights and sounds of a typical marketplace in China, she’s actually standing in the AI-enhanced, state-of-the-art Cognitive Immersive Room (CIR) on Rensselaer Polytechnic Institute’s (RPI) campus in Troy, N.Y. and the vendors are actually avatars on a 360-degree panoramic display.

Conversational AI, narrative generation, spatial context awareness, and gesture and facial landmark recognition technologies form the foundation of RPI’s interactive new class, “AI-Assisted Immersive Chinese.” Working with IBM Research, the university has created a first-of-its-kind, six-week credit-bearing course in Mandarin taught in the school’s Cognitive and Immersive Systems Laboratory (CISL). CISL is an ongoing collaboration between the university and IBM Research, and part of our AI Horizons Network to accelerate the research and application of AI, machine learning, natural-language processing and related technologies.

Our goal is to leverage AI to provide students with a culture-rich learning environment that feels almost like the real thing and gives students meaningful tasks to perform. AI-Assisted Immersive Chinese has already begun to offer those opportunities for RPI students in its inaugural semester this summer, with classes starting July 8.

Since piloting the technology last summer, IBM Research and RPI have added a number of more advanced features, including the ability for multiple students to interact with multiple avatars in order to complete a daily task, a more flexible dialogue system for language teaching, the ability for learning assessment and assistance, and more immersive language learning games. Students can practice Mandarin in a number of different situations, including marketplace negotiations, living and studying on the campus of a university, survival phrases at an airport, visiting a city and ordering a meal in a restaurant. CISL’s Mandarin Project developed the class’s cognitive and immersive language learning environment specifically for teaching non-native speakers. The idea is to get language students out of the classroom and put them in a more realistic learning environment.

IBM Watson’s speech recognition, natural language understanding and conversation technologies for English and Chinese enable the students to ask the simulation program for help with pronunciation and clarify how to perform a particular task. A technology called pitch-tone contour analysis evaluates pronunciation and provides immediate audio and visual feedback comparing students’ pronunciations against those of native speakers.

In each virtual scenario, students use head movement and position to indicate to whom they are talking. That movement is tracked by cameras and other sensors placed throughout the immersive environment. A spatial context system tracks a person’s proximity to different areas of the panoramic screen, initiating dialogue when it detects a person is nearby.

It’s often said that the best way to learn a foreign language is to immerse oneself in an environment where it’s spoken. If students are to improve their conversational skills, they must have many opportunities to practice speaking. That is especially true of a complex language like Mandarin, which has four different tones that must be mastered in order for a speaker to be properly understood.

Our new AI-assisted course can help overcome those challenges, delivering a rich learning environment to RPI students – no travel required.

Director, Cognitive User Experience, IBM Research

More AI stories

IBM RXN for Chemistry: Unveiling the grammar of the organic chemistry language

In our paper “Extraction of organic chemistry grammar from unsupervised learning of chemical reactions,” published in the peer-reviewed journal Science Advances, we extract the "grammar" of organic chemistry's "language" from a large number of organic chemistry reactions. For that, we used RXNMapper, a cutting-edge, open-source atom-mapping tool we developed.

Continue reading

From HPC Consortium’s success to National Strategic Computing Reserve

Founded in March 2020 just as the pandemic’s wave was starting to wash over the world, the Consortium has brought together 43 members with supercomputing resources. Private and public enterprises, academia, government and technology companies, many of whom are typically rivals. “It is simply unprecedented,” said Dario Gil, Senior Vice President and Director of IBM Research, one of the founding organizations. “The outcomes we’ve achieved, the lessons we’ve learned, and the next steps we have to pursue are all the result of the collective efforts of these Consortium’s community.” The next step? Creating the National Strategic Computing Reserve to help the world be better prepared for future global emergencies.

Continue reading

Simplifying data: IBM’s AutoAI automates time series forecasting

In our recent paper “AutoAI-TS: AutoAI for Time Series Forecasting,” which we’ll present at ACM SIGMOD 2021, AutoAI Time Series for Watson Studio incorporates the best-performing models from all possible classes — as often there is no single technique that performs best across all datasets.

Continue reading