Live long and prosper: How IBM Watson is powering virtual reality

Share this post:

In 2017—the fiftieth anniversary of Star Trek—the dream of standing on the bridge of the USS Enterprise is one step closer to becoming reality, all thanks to IBM Watson! I sat down with Michael Ludden to talk all about how Watson is powering Ubisoft’s incredible virtual reality game: Star Trek Bridge Crew.

“‘Warp 6…engage,’ ‘Helm, raise shields,’ and ‘Fire torpedoes!’ are just some of the famous lines from Star Trek I dreamed of saying aboard the Starship Enterprise,” Michael told me.

So I had to ask the burning question:

Is IBM building a real-life USS Enterprise?

Michael explained, “Not quite! In the upcoming ‘Star Trek: Bridge Crew’ by Ubisoft, a VR-native experience launching this month across Playstation VR, Oculus Rift, and HTC Vive, gamers will be able interact with virtual shipmates in natural language speech, just like their human player counterparts.”

“The other headline here is the new IBM Watson ‘VR Speech Sandbox,’ the same technology powering this first-of-a-kind feature in ‘Star Trek: Bridge Crew,’ is freely available for any interested VR developer to take, adapt, and use in their own VR experience. The open source code is available on github and it wraps two services.

Which Watson Services does it use?

“First up is Watson Speech-to-Text, which is recognized as the most accurate speech recognition service,” said Michael. “That’s not hyperbole; check out this blog post for more.

“Next is Watson Conversation. This service recognizes the intent of the gamer, the entities from within the game, and the dialog tree system rolled into one. Watson Conversation also includes an in-browser GUI interface to allow anyone to easily train Watson. Building a Watson Conversation chatbot takes literally 10 minutes,” Michael said.

“This is all achieved from within the Watson SDK for Unity—allowing VR developers to quickly prototype and add interactive speech interfaces with advanced functionality into their own applications.”

How did this all start?

“I’ve long been interested in virtual and augmented reality,” said Michael. “Just before the launch of the first major consumer VR headsets, the HTC Vive and Oculus Rift, it dawned on me that my work—namely the Watson Developer Cloud suite of services—dovetailed nicely with these forthcoming headsets, given that we also happened to have a fully supported Watson Unity SDK – Unity (being the most popular 3D development environment).”

Michael continued, “Working on the Watson Developer Cloud and being a big believer in the promise of virtual and augmented reality (this was just before the eagerly awaited launches of the first major consumer-facing VR headsets, the Oculus Rift and HTC Vive), it occurred to me that these two parts of my life would dovetail nicely, given some of the unity properties of Watson and the excellent developer support from Unity for these headsets.”

“I set about getting early access to the available VR headsets, bugging anyone internally who would listen to help me hack together a proof of concept demo,” Michael said. “HTC was the first to allow me access to their Vive hardware.”


So what happened?

“Like all VR headsets, the Vive had a microphone. That was intriguing to me in the context of how speech in VR could add another dimension of immersion,” Michael said. “Since IBM had recently launched Watson Conversation, I wondered how easy it would be to combine it with speech-to-text to build a robust, natural language speech interaction system, tailor-made for VR developers.”

“After prototyping on my own, I realized that this was not only feasible but—dare I say it—could be made into a relatively elegant and easy to implement solution for developers. After finding a few IBMers willing to scrap with me to get a proof of concept built—shout out to all my Mobile Innovation Lab peeps—a few months later we had an alpha version of what would eventually become the IBM Watson ‘VR Speech Sandbox’!” Michael said.

“HTC offered me a ‘VR bay’ in their booth at VR:LA 2016, a massive VR conference in—you guessed it—Los Angeles. We demo-ed this first-of-its-kind technology to the major players in the budding VR/AR industry and gauged interest. Despite being sandwiched between two highly-polished, long-in-development VR games, interest was high.”

Michael said, “Better yet, I met Ubisoft at this conference—they were demonstrating ‘Star Trek: Bridge Crew.’ Being a Trekkie and gamer, I tried the demo—which I loved. But one thing was missing: Did I want to issue those famous lines to computer-controlled players? I pitched the idea of voice-controlled computer players in the game and thought about how incredible it would be to converse with Star Trek crewmates in virtual reality.”

“Surprisingly to me, they listened to my nerdy rantings! And they were kind enough to connect me with David Votypka, the creative director of Red Storm, the Ubisoft studio developing the game.”

“During our first mind meld (sorry, I had to), we agreed to begin prototyping for feasibility,” Michael said. “David and Red Storm were impressed with the capabilities of Watson—especially how easily they got up and running with such advanced functionality in their existing codebase. Fast forward a year and ‘Star Trek: Bridge Crew’ has Watson-powered natural language voice interaction, thanks to the ‘VR Speech Sandbox’  and the many folks on both sides that worked together to get it done.”

Our hope, and the reason we did all of this, is that game developers will see the potential of advanced natural language voice interaction in VR and adapt the “VR Speech Sandbox” for their application. The possibilities are endless.

What can game developers do with Watson in VR?

“‘Star Trek: Bridge Crew’ leverages the advanced chatbot interactive conversation functionality. That’s just one of many potential use cases,” Michael said. “In the ‘VR Speech Sandbox’ sample code, when compiled into a demo, it’s used in more of a voice control interface.

Michael continued, “There are also companies looking into non-gaming use cases as well. Imagine a collaborative productivity tool you can speak to call up different data sets. Or how about being able to use your voice to summon that special brush in a VR design application, rather than always rifling through menu after menu? Maybe you could even ask for a hint in a VR puzzle game with your voice, rather than needing to hit some button and break immersion. The extensible qualities of ‘VR Speech Sandbox’ are something developers are noticing right out of the box. It can scale to support any number of different and inventive use cases we haven’t even thought of yet—but hopefully you will!”

Put on your thinking cap

While you’re busy creating your next VR adventure, if you own an HTC Vive, have a look at the free demo in HTC’s Viveport app store. And drop me a line if you have an idea for a ‘VR Speech Sandbox use case, or a suggestion for a future VR or AR use case-based solution for developers to offer to the community. We’ve got a lot coming up in VR and AR, but the possibilities are endless—so we look forward to hearing your ideas! My Twitter handle is @Michael_Ludden.

Now get out there and explore your very own strange new worlds. Live long, and prosper!

More Watson stories
December 17, 2018

Think Research Accelerates App Delivery with IBM Cloud Kubernetes Service

VirtualCare is architected as microservices. In evaluating cloud platforms for hosting it, we determined that IBM Cloud Kubernetes Service would provide the right combination of distributed resources and automated scaling for developing and updating the app.

Continue reading

December 10, 2018

IBM Cloud Security Advisor Now Integrates to NeuVector’s Kubernetes Security Platform

Today, we are excited to further strengthen the relationship between IBM Cloud and NeuVector by announcing the integration of Security Advisor and the NeuVector container network security solution. We have integrated your IBM Cloud security tools into one dashboard and console to facilitate centralized security management for the security admin.

Continue reading

December 5, 2018

Techwave Helps Businesses Migrate Securely and Non-Disruptively to the Cloud

Cloud computing offers an agile, flexible, and scalable alternative to traditional on-premises infrastructure, but choosing to migrate to a public cloud service can be risky. At Techwave, we help clients migrate to IBM Cloud bare metal servers so they can capitalize on the benefits of cloud computing while retaining complete control of their IT infrastructure.

Continue reading