Rainbow Octopus – Animate a 3d character with ARKit & IBM Watson

Share this post:

Rainbow Octopus is an open-source project created to show how a developer could use IBM’s Watson services in ARKit and Unity to control and manipulate a 3d animated character. Specifically, via the Watson Unity SDK, we capture and send speech to the IBM Speech To Text, and interpret the results using Tone Analyser and Watson Assistant. The responses from these services and Watson’s interpretation of the data would then impact the animated figure – an Octopus.

Technologies Used

  • Blender – Open-source 3d modelling and animation software
  • Unity – Cross platform games engine
  • ARKit – Software development tools to enable developers to build augmented-reality apps for iOS
  • Watson Unity SDK – Software kit to integrate IBM Watson services into Unity
  • Watson Assistant – An AI assistant designed for business, enabling conversational interfaces into any application, device, or channel.
  • Watson Speech-to-Text – Automatically transcribe audio from 7 languages in real-time with customisation available to improve accuracy for language and content you care most about
  • Watson Tone Analyser – Analyse emotions and tone of text in what people write online, or from audio transcripts. Predict whether they are happy, sad, confident, and more.

The result is an animated octopus, who could sit with you (via Augmented Reality) and respond to you – both by undertaking the actions you ask of it, and by reacting and showing emotions by changing it’s own colour based on the tone of your voice.

Give it a go!

The code is all available on our public GitHub here.

You’ll need a copy of Unity and an IBM Cloud account (available for free) to get it up and running. Unity will enable you to deploy the AR environment to an iPhone/iPad using ARKit. We can talk to the Octopus using IBM Speech To TextTone Analyser, and Watson Assistant, and it will respond in various ways.

As soon as the app is started, ARKit identifies horizontal surfaces within the camera view. When found, the ground is established and the octopus is rendered. As a user, you can then tap on your device’s screen to control the movement of the animated character.

Cognitive understanding in an animated character

The app is constantly listening too. You can try particular commands and “intents” like:

  • “Hello! “ – Octopus will wave
  • “Go for a walk” – Octopus will start walking
  • “Stop!” – Octopus will get scared – changing colour to yellow – and stop in his tracks
  • “Jump for Joy” – Octopus will jump up and down, whilst glowing green
  • “Turn right / left” – Octopus will turn in the indicated direction
  • “Get bigger / smaller” – Octopus will grow / shrink
  • A variety of other emotive sentences that cause the Octopus to change colour. Reinforcing that emotion will make that colour become stronger.
  • “You look happy” will make the octopus go light green. Then, “Jump for Joy” for example, will make the Octopus go a stronger green colour.

The app sends the speech to Watson Speech-To-Text (via the Watson Unity SDK), and receives raw text in response. The app passes the raw text to Watson Assistant and simultaneously to Watson Tone Analyzer – again utilizing the Watson Unity SDK. The results from the Tone Analyzer determine what tone was identified from the stated speech – this may be “joy”, “sadness”, “anger” etc. – and colours the octopus accordingly. The Tone Analyzer only analyzes the content of the words said, rather than the way it was said. Watson Assistant returns an “intent” from the text sent to it. For example if you say “Go forward”, or “Walk”, or “move” – the intent will be “forward” and the Octopus will move forwards. 

The project itself was coded by Kevin Brown, modelled on a similar project by Gwilym Newton (, Joe Pavitt created the 3d octopus and associated animations in Blender, and these were imported into Unity so they could be activated at the right time (via the Animation Controller). Kris Schultz then applied a range of techniques to make the experience slightly smoother. 


Master Inventor & Emerging Technology Specialist

More Cognitive stories

Helping PACE Make The World More Accessible for People with Motor Neural Disorders

On the 21st of November 2019 ETS hosted PACE for a 2 day hack event. The PACE Centre is a UK-registered charity and special school based in Buckinghamshire which specialises in helping children and young people with motor disorders, like cerebral palsy, which is estimated to effect 30,000 children in the UK. PACE transforms the […]

Continue reading

Emerging Technology in full force at THINK!

On 16th October, 8 members of the Emerging Technology team attended IBM’s THINK Conference in London across 3 areas – Shaping the Future, Data & AI and the Developer Zone.   Shaping the Future Zone Over in the Shaping the Future area, we had Rosie Lickorish presenting ‘A Quarter of An Owl with Rosie’ – […]

Continue reading

Analysing Sports in a Virtual Reality Environment Using Conversational Interfaces

Tools in the sports industry today regularly represent 3-dimensional data, e.g. player or ball positions, in a 2-dimensional plane. Whilst this is accessible to most, and can provide an overview of the information recorded for analysis, it does lose a sense of reality and context. It’s easy for an analyst to assess a match from […]

Continue reading