Clarifying the Complex with a ‘Simpler Voice’

Share this post:

As smart as the phones in our pockets are, they’re not built to interpret the written word. Low-literate adults, of whom there are 757 million worldwide, including 32 million in the United States, instead rely on visual cues for recognition and comprehension. So, as part of IBM Research’s Science for Social Good initiative, our team in Austin, Texas, reached out to the Literacy Coalition of Central Texas with an idea for a different approach: use artificial intelligence to turn text into pictures and simple verbal messages.

The mobile app project, called Simpler Voice, can parse complex text of everything from product descriptions, instruction manuals, to even public signage. It then extracts and presents simplified images and short spoken messages. Today’s smartphone AI is good at straight text-to-speech and speech-to-text, but not so good at rendering a new way of explaining the world around us.

Simpler Voice weaves together IBM Watson natural language understanding and text-to-speech services with novel image generation code – or in AI research parlance, generative adversarial networks (GAN). These GANs provide alternative visualizations of what the smart phone is looking at.

What better place to pilot Simpler Voice than the grocery store.

Our team’s first test run with the Literacy Coalition’s students will examine a handful of common grocery store items, from shampoo bottles to canned goods. We’ll then have their students pilot the app at a local grocery store – with the ability to scan any product.

Low-literate adults often match images from television ads and newspaper coupons with what they see – and then buy – at the grocery store. This limitation excludes potentially healthier, more economical choices, or quite simply, something they might like better. Simpler Voice opens up the entire store to our students, and we hope soon, to the 19 percent of Texans who are low-literate, and beyond.

Take dishwasher detergent, for example. An LCCT student sees a box of dishwashing tablets next to the liquid he recognizes and usually buys. Curious, but unsure of what the tablets do, he scans the barcode. Simpler Voice reads the company’s product description, and begins to interpret key words and phrases to verbalize “dishwasher detergent” and provide a visualization of a person using a dishwasher. Simpler Voice can also illustrate instructions on where to put the tablet in a dishwasher. And because these tablets look like candy, it may also communicate a safety warning: “Not candy. Do not let children eat this.”

Simpler Voice may offer the biggest benefit in visualizing and explaining the fine print on over-the-counter medications. Can anyone read that two-point type? With a quick scan, the app can explain how many pain reliever pills an adult or child should to take, how often, as well as warn of potential allergic reactions.

After piloting Simpler Voice in grocery stores around Austin, we hope to work with LCCT on expanding the app’s capability to legal documents, such as apartment rental agreements, and medical documents, like the pages of paperwork required at the doctor’s office.

Simpler Voice is one of IBM’s 15 global Science for Social Good projects launched this summer. Our team, including intern Minh Nguyen, from the University of Southern California, will work alongside LCCT to further develop and deploy the app for their students. Watch this space for news about Simpler Voice’s availability in your favorite app store.

Manager, Optimized Cloud Environments, IBM Research

Anne Gattiker

Principal Research Staff Member, IBM Research

More Accessibility stories

The World Needs More Volunteers, Here’s How Employers Can Help

We celebrate the spirit of volunteerism on International Volunteer Day, December 5, by recognizing the contributions of those who give back to schools, communities, and people in need. Imagine what a difference it would make it we could focus on altruism year-round and keep it going. Working together, we can. Many do what they can […]

Continue reading

Delivering Great Content in the Form of a Film About an Unlikely Hero

Modern marketing continues to evolve and so does the role of the modern marketer particularly in content marketing. Traditionally, we create content to share news, tell stories about products or services and to drive brand favorability and awareness. IBM has been using content to connect with audiences since the launch of Think magazine in the […]

Continue reading

The University of Ottawa and IBM Team On Cybersecurity Research

In our increasingly interconnected world, improving cybersecurity and safety has reached a critical point as threats and attacks become more sophisticated and insidious. According to the World Economic Forum, cybersecurity breaches have grown by 67 percent over the past five years alone. We’ve all heard about or perhaps experienced firsthand the repercussions of such acts: […]

Continue reading