Seeing and feeling shapes described by a stream of sound

Share this post:

Researchers at IBM’s lab in Tokyo have developed a new method to convert visual shapes on the web into a real-time stream of sound to help visually impaired better understand the shape of an object shown on a web site.

Still an early prototype, the technology converts the shape of an object, such as an image shown on a webpage, into a stream of sound. By listening to the sounds, the visually impaired can mentally feel the shape or better-understand turn-by-turn directions. The technology may also be useful as a real-time guide for online maps which shows route to destination via a mobile phone.

According to a preliminary experiment conducted by the IBM researchers, both blind and sighted people quickly adapted to following directional sound, and recognizing shapes such as randomly drawn paths, letters in the alphabet, and numbers. People with musical training tended to adopt the technology at a faster rate. And some of the participants felt as if they could mentally “see” and “feel” the shapes described by the sounds.

The prototype has the potential to not only help the visually impaired, but to also benefit any user of a mobile device. In situations where a person could use the assistance of a mobile device, but cannot easily look at a screen; or in instances when not all visual information is present (such as in online gaming), the system can provide real-time guidance in the real world or a virtual world.

IBM researchers in Tokyo are advancing the prototype technology with the goal of helping empower the visually impaired to better-navigate the ever increasing non-textual, visual information on the web.

More stories

IBM Research AI Advances Speaker Diarization in Real Use Cases

In a recent publication, IBM researchers describe a novel speaker diarization algorithm that can consider not only speaker information, but also identifying clues about individual recording environments that help differentiate between the speakers, resulting in improved diarization accuracy for our in-house, real test cases as well as public benchmark data.

Continue reading

IBM Research AI at ICASSP 2020

The 45th International Conference on Acoustics, Speech, and Signal Processing is taking place virtually from May 4-8. IBM Research AI is pleased to support the conference as a bronze patron and to share our latest research results, described in nine papers that will be presented at the conference.

Continue reading

IBM Takes Its Quantum Computer to Japan to Launch Country-Wide Quantum Initiative

IBM quantum computing hardware comes to Japan – thanks to a new initiative between IBM and the University of Tokyo.

Continue reading