Events

Bluemix and Twilio mix voice and data to help doctors

Share this post:

I recently had the pleasure of speaking at KrankyGeek 2015, you can watch the replay here. This post summarizes my presentation “Mixing Data and Video – IBM Bluemix, Watson, and Twilio”, which at its core was about improving the doctor/patient experience.

I never heard of this conference before and wasn’t quite sure what I was in for but to my surprise it was a wonderful event. It was an interesting mix of people, you could almost say it was half chocolate ice cream (VoIP people) and vanilla ice cream (web dev’s) and it ended up creating the most utter perfect chocolate twist. I love this analogy because it’s almost exactly what WebRTC is. If you haven’t heard of WebRTC before let’s discuss it real quick…

Quick introduction to WebRTC

WebRTC is one of those things most people depend on and yet they have no clue what it is. If you have ever used Facebook video chat, you have used WebRTC. So you are probably thinking its something related to video chat—well you are partially correct. Video chat and audio chats are incredibly complex things and require very very deep technical skills with technologies such as STUN, TURN, ICE, and the list could go on and on. From my development experience in video and audio conferencing, I can tell you those technologies are not easy to deal and maintain, let alone develop on. If you have deep knowledge in C++ and VoIP those technologies are for you.

However, if you are a web guy like me, you just want to create that next awesome app as soon as possible and have minimal effort on your part consuming services. You want to consume these services through simple API’s or SDK’s. That is where WebRTC comes in…

WebRTC basically abstracts all this really low level and complex tech into simple and reusable API’s. While WebRTC has its issues right now (most maturity and adoption) its an amazing technology that aims to make video conferencing a breeze.

With WebRTC you can basically call someone with the following lines of Javascript

[code]myEndpoint = Twilio.Endpoint(token);

myEndpoint.listen()
.then(
endpointConnected,
function(error) {
console.log("Could not connect to Twilio: " + error.message)
}
);
[/code]

Honestly, just that and you can setup a video call! What if we wanted to take it a step further and allow someone to call us? Well, we already did that, the function above endpointConnected gets called when someone calls us to a video chat. So lets look at that:

[code]function endpointConnected() {
myEndpoint.on(‘invite’, function(invite) {
invite.accept().then(conversationActive);
});
}
[/code]

With the above code we are basically waiting for someone to call us; in tech terms, it’s an invite. When the invite comes through, we accept the invite prompt in the browser then the conversation is started. Again, basic WebRTC stuff here.

Next we need handle the event when the conversation actually starts. Below is the code:

[code]function conversationActive(conversation) {
showLocalVideo(conversation.localMedia);
conversation.on(‘participantConnected’, showRemoteVideo);
}
[/code]

Wow, only 3 lines to actually connect the video call, crazy! If you tried doing this with a VoIP infrastructure you would literally hate your life and have lots of gray hair. In the above function there are 2 function calls, showLocalVideo and showRemoteVideo. These functions basically attach the video and audio streams to a DOM element in your webpage.

Attaching the local stream (if you called someone then you are the local stream) is pretty simple. See below:

[code]function showLocalVideo(localStream) {
var videoElement = $("#aunquiediv")[0];
localStream.attach(videoElement);
}
[/code]

Yes, I know the above selector uses an id and it’s incredibly bad but it exemplifies a point that you need to attach the video to a specific element in the DOM.

OK, so cool we have the local video showing, let’s show the remote video in the browser as well. To do this we can use the following code:

[code]function showRemoteVideo(participant) {
var videoElement = $("#anotheruniquediv")[0];
participant.media.attach(videoElement);
var mediaStreamSet = participant.media.mediaStreams;
var firstMediaStream = mediaStreamSet.values().next().value;
}
[/code]

Basically the above code attach’s the remote audio and video to an element in the DOM.

So all in all let me count it out real quick… It’s only 26 lines of code to setup a video call through your web browser. Again, if you did this through conventional VoIP means you probably would not have any hair left at this point, that stuff is really really really hard to setup; let alone maintain.

Ok, so since we have discussed what what WebRTC is lets now discuss Kranky Geek and what my talk was actually about.

Bluemix and Twilio bring doctors closer to patients

The basis of the talk revolved around mixing video with data and providing a delightful user experience. So I know this isn’t fun to talk about but it’s something that applying technology to can provide for a better outcome for people. The topic we are talking about is treating and helping patients that have cancer. We worked with a client that is a leading cancer center in the United States and they basically wanted to find a way to expand the reach of their amazing doctors to allow access to those amazing life savers for people that could not financially afford to travel to the center or could just not physically be able to travel.

They were looking for a way to revolutionize their way they care for their patients while still giving each patient the one-on-one attention they deserve with some of the best oncologists in the world.

Enter our solution…

Our solution allows doctor’s to teleconsult with their patient’s and provide them with the care that they need and deserve without the travel implications and issues. We used Twilio Video as the basis for this. Twilio Video is based on WebRTC and we came up with a solution that allowed doctor’s to video chat with patient’s from their iPad’s. As the patient was talking to the doctor, IBM Watson was listening to the video stream and transcribing the audio on the fly and saving it to the patient’s file. How cool is that??? My wife is a physician’s assistant (PA) and she spends inordinate amounts of time doing charts and going over notes and writing them down. By IBM Watson transcribing these notes on the fly and saving them the patient’s file it saved the doctors an uncountable amount of time and even providing better notes than the doctor could do themselves.

We even took it a step further and applied IBM Watson to analyze the personality of the spoken audio of the patient. Once we had the transcribed audio that has been converted into text we sent the transcribed text to Watson. Watson would come back with the personality of the patient. In the app, the doctor could see key personality traits of how the patient was feeling about their treatment. This could be called smarter health care. But in effect its giving the doctor’s a way to peer into the the mind of the patient and really try to see how they were feeling about their treatment and the mental state at a point in time.So how does all this work. Obviously it’s back by IBM Bluemix. The whole app runs in IBM Bluemix and is a Node.Js app. In Bluemix it uses the Watson Personality Insights service, Watson Speech to text, Twilio service, and Cloudant database service.

Resources and Multimedia

Check below for a diagram of how things flow:

architecture diagram

The diagram above and more are shown in my slides below:

KrankyGeek 2015 – Mixing Data and Video – IBM Bluemix, Watson, and Twilio from Jeff Sloyeri

Below is also the video on my talk:


Additionally I held a Periscope chat with Billy Chia from Twilio on WebRTC and why Bluemix and Twilio are the platform’s you want to use to build your WebRTC app. Replay is below:


I would love to hear your feedback and any suggestions you have, please reach out to me on Twitter @jsloyer.

IBM Cloud Kubernetes Service - Core Dev Lead

More stories
December 19, 2018

Highlights: IBM Cloud @ KubeCon 2018

KubeCon gathered adopters and technologists from leading open source and cloud-native communities in Seattle, WA from December 10-13, 2018, and IBM Cloud was there in full force.

Continue reading

December 13, 2018

Learn About New Retail, Supply Chain, and Mining Tech Trends at IBM Cloud Platform ISV Sessions at THINK 2019

Leaders who seek the latest technology trends in retail, supply chain, and mining will find them at IBM Think 2019, held in San Francisco from February 12–15, 2019. Hear your peers share their business challenges and successes while leveraging the latest technologies.

Continue reading

December 10, 2018

Gain Insights for Enterprise Efficiency with IBM Cloud Platform ISV sessions at THINK 2019

The IBM Think 2019 conference in San Francisco (February 12–15, 2019) is designed with you in mind. Leaders in your industry will discuss use cases where they faced pain points you have likely encountered. They’ll reveal how IBM Cloud helped them solve their challenges and might do the same for your enterprise.

Continue reading