AI

AI Enables Foreign Language Study Abroad, No Travel Required

Share this post:

A student learning to speak Mandarin wanders into a marketplace on the streets of China on a sunny summer afternoon. Before long, two vendors approach and begin hawking products, trying to outbid one another. The student must now grasp what’s being said and formulate an appropriate response using proper pronunciation to avoid being misunderstood. It’s a challenging, yet common, scenario for anyone trying to learn a new language by interacting with native speakers and immersing themselves in a foreign culture.

Fortunately, the student in this case is able to pause the unfolding scenario to check the accuracy and tone of her planned response. That’s because, even though the student appears to be surrounded by the sights and sounds of a typical marketplace in China, she’s actually standing in the AI-enhanced, state-of-the-art Cognitive Immersive Room (CIR) on Rensselaer Polytechnic Institute’s (RPI) campus in Troy, N.Y. and the vendors are actually avatars on a 360-degree panoramic display.

Conversational AI, narrative generation, spatial context awareness, and gesture and facial landmark recognition technologies form the foundation of RPI’s interactive new class, “AI-Assisted Immersive Chinese.” Working with IBM Research, the university has created a first-of-its-kind, six-week credit-bearing course in Mandarin taught in the school’s Cognitive and Immersive Systems Laboratory (CISL). CISL is an ongoing collaboration between the university and IBM Research, and part of our AI Horizons Network to accelerate the research and application of AI, machine learning, natural-language processing and related technologies.

Our goal is to leverage AI to provide students with a culture-rich learning environment that feels almost like the real thing and gives students meaningful tasks to perform. AI-Assisted Immersive Chinese has already begun to offer those opportunities for RPI students in its inaugural semester this summer, with classes starting July 8.

Since piloting the technology last summer, IBM Research and RPI have added a number of more advanced features, including the ability for multiple students to interact with multiple avatars in order to complete a daily task, a more flexible dialogue system for language teaching, the ability for learning assessment and assistance, and more immersive language learning games. Students can practice Mandarin in a number of different situations, including marketplace negotiations, living and studying on the campus of a university, survival phrases at an airport, visiting a city and ordering a meal in a restaurant. CISL’s Mandarin Project developed the class’s cognitive and immersive language learning environment specifically for teaching non-native speakers. The idea is to get language students out of the classroom and put them in a more realistic learning environment.

IBM Watson’s speech recognition, natural language understanding and conversation technologies for English and Chinese enable the students to ask the simulation program for help with pronunciation and clarify how to perform a particular task. A technology called pitch-tone contour analysis evaluates pronunciation and provides immediate audio and visual feedback comparing students’ pronunciations against those of native speakers.

In each virtual scenario, students use head movement and position to indicate to whom they are talking. That movement is tracked by cameras and other sensors placed throughout the immersive environment. A spatial context system tracks a person’s proximity to different areas of the panoramic screen, initiating dialogue when it detects a person is nearby.

It’s often said that the best way to learn a foreign language is to immerse oneself in an environment where it’s spoken. If students are to improve their conversational skills, they must have many opportunities to practice speaking. That is especially true of a complex language like Mandarin, which has four different tones that must be mastered in order for a speaker to be properly understood.

Our new AI-assisted course can help overcome those challenges, delivering a rich learning environment to RPI students – no travel required.

Director, Cognitive User Experience, IBM Research

More AI stories

Pushing the boundaries of convex optimization

Convex optimization problems, which involve the minimization of a convex function over a convex set, can be approximated in theory to any fixed precision in polynomial time. However, practical algorithms are known only for special cases. An important question is whether it is possible to develop algorithms for a broader subset of convex optimization problems that are efficient in both theory and practice.

Continue reading

Making Neural Networks Robust with New Perspectives

IBM researchers have partnered with scientists from MIT, Northeastern University, Boston University and University of Minnesota to publish two papers on novel attacks and defenses for graph neural networks and on a new robust training algorithm called hierarchical random switching at IJCAI 2019.

Continue reading

Improving the Scalability of AI Planning when Memory is Limited

We report new research results relevant to AI planning in our paper, "Depth-First Memory-Limited AND/OR Search and Unsolvability in Cyclic Search Spaces," presented at the International Joint Conference on Artificial Intelligence, IJCAI-19.

Continue reading