Live long and prosper: How IBM Watson is powering virtual reality

Share this post:

In 2017—the fiftieth anniversary of Star Trek—the dream of standing on the bridge of the USS Enterprise is one step closer to becoming reality, all thanks to IBM Watson! I sat down with Michael Ludden to talk all about how Watson is powering Ubisoft’s incredible virtual reality game: Star Trek Bridge Crew.

“‘Warp 6…engage,’ ‘Helm, raise shields,’ and ‘Fire torpedoes!’ are just some of the famous lines from Star Trek I dreamed of saying aboard the Starship Enterprise,” Michael told me.

So I had to ask the burning question:

Is IBM building a real-life USS Enterprise?

Michael explained, “Not quite! In the upcoming ‘Star Trek: Bridge Crew’ by Ubisoft, a VR-native experience launching this month across Playstation VR, Oculus Rift, and HTC Vive, gamers will be able interact with virtual shipmates in natural language speech, just like their human player counterparts.”

“The other headline here is the new IBM Watson ‘VR Speech Sandbox,’ the same technology powering this first-of-a-kind feature in ‘Star Trek: Bridge Crew,’ is freely available for any interested VR developer to take, adapt, and use in their own VR experience. The open source code is available on github and it wraps two services.

Which Watson Services does it use?

“First up is Watson Speech-to-Text, which is recognized as the most accurate speech recognition service,” said Michael. “That’s not hyperbole; check out this blog post for more.

“Next is Watson Conversation. This service recognizes the intent of the gamer, the entities from within the game, and the dialog tree system rolled into one. Watson Conversation also includes an in-browser GUI interface to allow anyone to easily train Watson. Building a Watson Conversation chatbot takes literally 10 minutes,” Michael said.

“This is all achieved from within the Watson SDK for Unity—allowing VR developers to quickly prototype and add interactive speech interfaces with advanced functionality into their own applications.”

How did this all start?

“I’ve long been interested in virtual and augmented reality,” said Michael. “Just before the launch of the first major consumer VR headsets, the HTC Vive and Oculus Rift, it dawned on me that my work—namely the Watson Developer Cloud suite of services—dovetailed nicely with these forthcoming headsets, given that we also happened to have a fully supported Watson Unity SDK – Unity (being the most popular 3D development environment).”

Michael continued, “Working on the Watson Developer Cloud and being a big believer in the promise of virtual and augmented reality (this was just before the eagerly awaited launches of the first major consumer-facing VR headsets, the Oculus Rift and HTC Vive), it occurred to me that these two parts of my life would dovetail nicely, given some of the unity properties of Watson and the excellent developer support from Unity for these headsets.”

“I set about getting early access to the available VR headsets, bugging anyone internally who would listen to help me hack together a proof of concept demo,” Michael said. “HTC was the first to allow me access to their Vive hardware.”


So what happened?

“Like all VR headsets, the Vive had a microphone. That was intriguing to me in the context of how speech in VR could add another dimension of immersion,” Michael said. “Since IBM had recently launched Watson Conversation, I wondered how easy it would be to combine it with speech-to-text to build a robust, natural language speech interaction system, tailor-made for VR developers.”

“After prototyping on my own, I realized that this was not only feasible but—dare I say it—could be made into a relatively elegant and easy to implement solution for developers. After finding a few IBMers willing to scrap with me to get a proof of concept built—shout out to all my Mobile Innovation Lab peeps—a few months later we had an alpha version of what would eventually become the IBM Watson ‘VR Speech Sandbox’!” Michael said.

“HTC offered me a ‘VR bay’ in their booth at VR:LA 2016, a massive VR conference in—you guessed it—Los Angeles. We demo-ed this first-of-its-kind technology to the major players in the budding VR/AR industry and gauged interest. Despite being sandwiched between two highly-polished, long-in-development VR games, interest was high.”

Michael said, “Better yet, I met Ubisoft at this conference—they were demonstrating ‘Star Trek: Bridge Crew.’ Being a Trekkie and gamer, I tried the demo—which I loved. But one thing was missing: Did I want to issue those famous lines to computer-controlled players? I pitched the idea of voice-controlled computer players in the game and thought about how incredible it would be to converse with Star Trek crewmates in virtual reality.”

“Surprisingly to me, they listened to my nerdy rantings! And they were kind enough to connect me with David Votypka, the creative director of Red Storm, the Ubisoft studio developing the game.”

“During our first mind meld (sorry, I had to), we agreed to begin prototyping for feasibility,” Michael said. “David and Red Storm were impressed with the capabilities of Watson—especially how easily they got up and running with such advanced functionality in their existing codebase. Fast forward a year and ‘Star Trek: Bridge Crew’ has Watson-powered natural language voice interaction, thanks to the ‘VR Speech Sandbox’  and the many folks on both sides that worked together to get it done.”

Our hope, and the reason we did all of this, is that game developers will see the potential of advanced natural language voice interaction in VR and adapt the “VR Speech Sandbox” for their application. The possibilities are endless.

What can game developers do with Watson in VR?

“‘Star Trek: Bridge Crew’ leverages the advanced chatbot interactive conversation functionality. That’s just one of many potential use cases,” Michael said. “In the ‘VR Speech Sandbox’ sample code, when compiled into a demo, it’s used in more of a voice control interface.

Michael continued, “There are also companies looking into non-gaming use cases as well. Imagine a collaborative productivity tool you can speak to call up different data sets. Or how about being able to use your voice to summon that special brush in a VR design application, rather than always rifling through menu after menu? Maybe you could even ask for a hint in a VR puzzle game with your voice, rather than needing to hit some button and break immersion. The extensible qualities of ‘VR Speech Sandbox’ are something developers are noticing right out of the box. It can scale to support any number of different and inventive use cases we haven’t even thought of yet—but hopefully you will!”

Put on your thinking cap

While you’re busy creating your next VR adventure, if you own an HTC Vive, have a look at the free demo in HTC’s Viveport app store. And drop me a line if you have an idea for a ‘VR Speech Sandbox use case, or a suggestion for a future VR or AR use case-based solution for developers to offer to the community. We’ve got a lot coming up in VR and AR, but the possibilities are endless—so we look forward to hearing your ideas! My Twitter handle is @Michael_Ludden.

Now get out there and explore your very own strange new worlds. Live long, and prosper!

More Watson stories
February 28, 2019

UBS Brings Financial Expertise to Customers Using Innovative Digital Assistants

UBS recognizes the potential to modernize the approach to client relationships as technology and expectations evolve, and by working with IBM Cloud and FaceMe, they are looking to humanize digital assistants to help make that transformation.

Continue reading

January 2, 2019

Chatbots Orchestration and Multilingual Challenges – Part 1

Chatbots Orchestration and Multilingual Challenges Nowadays, conversational bots are created to meet the needs for users. However, if users need to actively switch between languages and conversations, or maintain a large conversational bot, it is also a tedious and difficult task, and it often causes bad experience. We propose a multilingual chatbot architecture to expand […]

Continue reading

December 24, 2018

Chatbots Orchestration and Multilingual Challenges – Part 2

Chatbots Orchestration and Multilingual Challenges This post is the part 2 of “Chatbots Orchestration and Multilingual Challenges” post. Please check Chatbots Orchestration, if you haven’t. Part 2. Multilingual Chatbots Challenges There are some challenges when creating multilingual chatbots. A common practice is to use a baseline language chatbot then translate it to other languages chatbots. […]

Continue reading