Cognitive Candy is an internet connected candy dispensing machine that uses Watson IoT Lab Dashboard, Blockchain and IoT Real-Time Insights, and is built on Raspberry Pi. Candy gives away treats to people if she “feels” like doing so. If you want to receive candy, all you have to do is engage her in a friendly conversation. In the video below, you’ll meet Candy, and you’ll see she’s not just your average machine — she actually has a sweet personality (and she also won Top Staff Pick at the Maker Faire in Austin, Texas).
We sat down with Cognitive Candy creator, Igor Ramos — Senior Software Engineer focused on the Internet of Things, IBM Watson and Blockchain — to lean more about Candy, and why the emotional interactions and experiences he facilitated through Candy are so special.
Igor is an engineer and a maker, who’s passionate about researching and developing new products, concepts, and experiences. Although, Igor’s background is in safety-critical software for heavy vehicles, he is a generalist with patents ranging from Machine Learning algorithms to Human-Machine Interfaces.
Question: How would you define your day job?
Igor Ramos: I’m a software development leader working on IBM’s exciting Watson IoT Platform and Blockchain technologies.
Question: Igor, when did you first get involved with the Maker community?
Igor Ramos: I became a Maker in 2011 when I moved to Minnesota. To better cope with Minnesota’s sub-zero winters, I created a gadget to remote start my car’s engine using my cell phone, so that by the time I got into the car it would be nice and toasty inside. Fortunately, now that I live in Austin, Texas, that gadget has become obsolete.
Question: Can you tell us who Candy is and how you created her?
Igor Ramos: Have you ever watched a movie that made you laugh out loud, or cry like a baby or empathize with a character? Isn’t it crazy you got so emotional even though you know it’s all fake? In my Cognitive Candy project, I noticed something similar to that phenomena. And I was caught by surprise. I thought what I was putting together was a simple 3D printed candy machine made with a Raspberry Pi circuit board and Watson technologies. But when I took that THING to the Maker Faire Austin, and saw how people interacted with HER. It was much more than that.
I gave Candy sweetest personality. She’s always happy and wants you know “she’s happy to meet you!” She calls you sweetheart, honey, cutie-pie, and she finds you awesome when you answer a question. She loves to chitchat and is thrilled to give out candies. The candy was just a conversation starter. Candy is actually a machine that distributes emotions and positive feelings.
I built a machine that interacts and facilitates emotions. When I was demonstrating Candy, I witnessed joy, interest, cheerfulness, and admiration. Children were overwhelmed with joy. Their interactions were positive, full of delight. Overall, the interaction created memorable experiences. I saw a deep level of amusement as well as awe in kids’ reactions. During the two days I was demonstrating Candy, I was able to craft and tweak Candy based on their experiences. Based on the tips and the interaction of each child, I was able to teach Candy how to be more ‘sweet’. As a result, I used what I was learning to produce more visible results – more candy being dispensed, more interactions, etc.
Question: What was the impetus for creating Cognitive Candy?
Igor Ramos: It all started with an IBM competition to build a fun gadget using Watson IoT Platform and Cognitive services. Being new at IBM, I joined in July 2015, I thought this contest would be a great way to learn about Watson by doing. That’s when I jumped at the chance and signed up for the challenge. Honestly, it’s this kind of initiative — for which John Cohn is the sponsor — that assures me I’ve made the right decision to join IBM!
Originally, the project was conceived in response to a contest — a challenge put forward by the Austin Makers Faire. The contest was truly special as the organizers used fun and engaging design criteria. It took me a few months to pull it all off — learning how to use Watson APIs, and the Watson IoT Platform. Really, it was one the most fulfilling days of my life. I will always think of it as a delightful and memorable experience, and I hope the kids and people who interacted with Candy will remember it that way as well.
My inspiration comes from a software called Dr. Sbaitso, an artificial intelligence speech synthesis program developed by Creative Labs in 1992. That program shipped with a sound card I purchased back then, and this was the first time I ever heard speech coming from a computer! On top of that, I was amazed by the illusion of intelligence and that concept stuck with me.
The next step was to apply that concept. I was adamant about my gadget doing something no phone or tablet could do. So giving away candies was a simple action no phone or tablet can do. It took me just 2 or 3 evenings to confirm my idea was viable and create a low fidelity prototype in Node-RED. Then it took me 4 months or so improve my design. In that time I acquired some excellent knowledge of Node-RED, Watson Cognitive APIs, and 3D modeling, were the hardest parts of the project. The candy dispensing mechanism was hard. It’s one thing to design a static part; it’s another to design a movable part driven by servos. I’m glad to report the IBM Watson IoT Platform was very easy to setup.
Question: Can you tell us a bit about the Maker Fair challenge and your experience there with Cognitive Candy?
Igor Ramos: On a Tuesday, I learned I had been selected to present at the Makers’ Faire that coming Saturday. The project had lots of loose ends I had to wrap up in just four evenings. That was very intense, but my attitude was what do I have to lose?
Question: Were you surprised by how people interacted with Candy at the Makers’ Faire after you unveiled Candy and observed how people interacted with her?
Igor Ramos: The moment I knew I had something powerful was when people started thanking Candy. When you walk out of a restaurant and grab a candy from a bowl, do you thank the bowl? Or when you get a can of soda from a vending machine, do you thank her for that refreshment? When the kids didn’t say thank you the parents quickly intervened. So that shows that both kids and adults ‘fell for it’, and talked as if they were talking to a human. And that was powerful.
As always, with great power comes great responsibility. When I debuted Candy, most of my audience were kids so I kept it positive, but towards the end of the Faire there was a girl in her twenties who approached the booth to know what it was all about. She didn’t want sweets, she just wanted to chat. When Candy learned, the girl didn’t like sweets, Candy expressed her feelings.
“You don’t like candy… so you don’t like me? You make me feel sad, I think I’m going to cry.” Interestingly the girl ended up apologizing for not liking candy and they became friends.
Question: Do you think people understood that Candy was a machine interacting with their emotions?
Igor Ramos: Yes. I was really surprised by the level of emotional interaction the children and adults had with the machine. The combination of Cognition and IoT affected people — something became someone. In my view this project really helped to demonstrate why cognition plus IoT is powerful. It involves ethics (privacy, dignity). It made me wonder if we should start to call the Internet of Things, the Internet of Emotions or The Emotion of Things. What Candy helped me to realize is that through design, we have the opportunity to influence people’s emotions and interactions, and as a result, we have a responsibility to ensure that experience is positive — showing how interaction between man and machine can be bi-directional.
Question: How does Cognitive Candy make use the Watson IoT Platform, Analytics and blockchain together?
Igor Ramos: The Watson IoT Platform provides an easy connection to the “CloudCandy” app running in Bluemix. I use that app to manage Cognitive Candy remotely. For example, for editing its dialog via the Brain Editor page. Watson IoT Platform is critical for inbound connections. Since Candy sits behind a firewall, it would be a great pain to configure a firewall to accept connections and setup a dynamic IP address. By using Watson IoT Platform, I am able to setup Candy to subscribe to an MQTT topic in the Watson IoT Platform which listens for commands to arrive.
The most notable Watson Cognitive API used in Cognitive Candy is the Speech to Text API, which converts people’s speech into words. Words are then sent to the ‘brain’ to process it. The output from the brain goes to another Watson API— the Text to Speech, which produces the speech people will hear.
Cognitive Candy uses Blockchain for vouchers. I call these vouchers ChainChecks (similar to rain checks). In essence, ChainCheck is a digital crypto-currency to demonstrate IBM Blockchain technology. It’s just for play. You’re issued a ChainCheck code when you fill out a survey form asking what’s your favorite color. Then when you meet Candy in person you use that code and redeem it for a candy. ChainChecks states are tracked in the Blockchain. This demo is open for anyone who wants to try it.
Question: What kind of prototype is Cognitive Candy? How do you imagine the principles used to design Candy could be applied in a real world situation – for example, is it supply chain, retail inventory, M2M?
Igor Ramos: The Cognitive Candy prototype is a glimpse into the future, where intelligent conversations become the new User Interface. Note that I say “intelligent conversation” as opposed to voice interactions using keyword or commands. That future is coming fast. On June 16 2016, IBM announced Watson technology being used in Olli, the self-driving car. That’s a real-life example of an organization disrupting a market, creating not only an idea and a prototype, but actually taking orders and manufacturing the final product to deliver to clients – all in an incredibly short span of time.
In the instance of Olli, passengers will be able to interact conversationally with Olli while traveling from point A to point B, discussing topics about how the vehicle works, where they are going, and why Olli is making specific driving decisions. Watson empowers Olli to understand and respond to passengers’ questions as they enter the vehicle, including destinations (Olli, can you take me downtown?) or specific vehicle functions (How does this feature work?), and even, (are we there yet?). Passengers can also ask for recommendations on local destinations such as popular restaurants or historical sites based on analysis of personal preferences. These interactions with Olli are designed to create more pleasant, comfortable, intuitive and interactive experiences for riders as they journey in autonomous vehicles. It’s pretty fascinating, futuristic stuff , and it is happening now.
The news about Olli further highlights that it is a great time to be a developer and a maker. Incredible as it may seem, the same technology IBM uses for Olli — it leverages four Watson developer APIs — Speech to Text, Natural Language Classifier, Entity Extraction and Text to Speech – is available to you and me. I’m delighted to tell you that Cognitive Candy uses the same technology Olli uses to enable seamless interactions between the vehicle and passengers.
Although it sounds like science fiction, it isn’t, it’s real and here and available to developers, across any industry. As a result, we can all start to explore and consider using this new UI alternative when designing solutions that require, or benefit from these sorts of “intelligent conversations” that create seamless interactions.
Question: Are there any plans to update Cognitive Candy for Halloween — to dispense candy in return for a scary story, or a scary sound?
Igor Ramos: Cognitive Candy is way too sweet of a ‘person’ to scare kids. She just doesn’t have the right personality for Halloween. I toyed with the idea of adapting Candy to be a HAL 9000 clone. HAL is an artificial intelligence system running a spaceship in the movie 2001: A Space Odyssey. It’s pretty scary when HAL disobeys human orders and says, “I’m sorry Dave, I’m afraid I can’t do that.” “I know that you and Frank were planning to disconnect me, and I’m afraid that’s something I cannot allow to happen.” But, I reconsidered because I think a HAL clone would play a bit on people’s fears of Artificially Intelligent robots taking over the world.
Question: Hmm – yes, I definitely agree – world domination by rogue robots with Artificial Intelligence isn’t really the image for Candy. I’m sure people will have questions about Candy and the technology you used in creating her. How can people get a hold of you?
Igor Ramos: Questions and comments are welcome and encouraged. I love talking to people about Cognitive Candy and all the technologies I used to create ‘her’. My Twitter ID is: @RamosIgorS.
Learn more about Cognitive Candy
To fully appreciate Candy, you really do need to listen and see her in action. To learn more about how Cognitive Candy was created, you can watch a video which not only shows different kids’ reactions to meeting Candy, but also provides an overview of the technology, the architecture, including how Igor used blockchain for coupons. The video illustrates the connections between user interaction, research, engagement and interaction, in effect giving you the building blocks to enable you to help facilitate a natural M2M interaction.
The video links offer you a different ways to view the project:
Over the past few months, IBM has hosted several Engineering Academy Comes to You events in locations throughout the world. I’m happy to tell you that these local presentations have become extremely popular and well attended learning opportunities. That’s because they provide valuable product information and one-on-one interactions between IoT professionals and their peers. If you’re not keeping up with the changes in engineering, you’re assuredly falling behind. That’s why I urge you to register now for one of these upcoming events. ...read more
IBM IoT Exchange 2020 is an intensive two-day conference that attracts the best and the brightest in key industries, including automotive, utilities, industrial, defense, government, and the Fortune 500. That’s why speaking at IoT Exchange is a win-win for everyone. ...read more
IoT Exchange 2020 has been deliberately planned to address your current challenges. And there are a lot of them. Equipment, buildings, and vehicles you’re responsible for have morphed to become complex, connected software-driven assets. They require a new kind of operating model. They demand a new kind of asset intelligence. This conference is designed to give you the intelligence and inspiration you need. ...read more