Tis the season - To automate stuff...
Hickmat 100000QA3T Visits (1386)
As we approach Christmas I find that I have a bit more time available to experiment and with this new found freedom I started to think about how I could extend publicly facing conversational assistant to support something very specific to an individual (like controlling smart appliances in a house). Sound like one of the plethora of currently available assistance - well yes but they tend to lock you into a set eco system where as I wanted to allow individuals to create their own management layers (and of course its a good excuse to play).
Personally I have a set of TPLink SmartPugs at home so that was the target I was aiming for but the approach I decided to take should allow for anything else with a suitable API to be controlled.
The first step was to look at how to create a personalised "callout" or "extension point" in an existing Assistant. To do this I took a simple assistant that I had created for a demo around accessing / managing GIT (Scrumbot) as my base and extended it using the "API Calling" capability I created (se previous blog posts) to allow me to pull in a preferences document from a Cloudant database. This document specifies the target endpoint to call for each configured "callout" e.g. "Home automation". The following shows the structure of the JSON document.
To support providing access to the Cloudant database (and also as a place to build out my Home Automation) I opted for my old friend - NodeRED, and the flow I created was:
In this flow the "Get Preferences" REST service is invoked via a GET passing a query parameter id which contains the <unique ID> to lookup. In the true tradition of experimentation as you can this flow is lacking in any form of error checking
In the assistant I'm building on, the <unique ID> is passed from the channel UI and presented to Watson Assistant in the context so it was an easy task to set the dialog flow in Watson Assistant call this API as the first step.
Here you can see I check if I have a passed in username and if I do then I use <callapi> to access the API and pullback the preferences document. I configured the APIC flow supporting the preferences/read API to push the retrieved into a context variable named "preferences".
Next I trained and intent within the Assistant to detect "Home Automation" style questions. This is the key effort I needed to put into expanding the existing Assistant and I used the following questions as examples:
Within the Dialog flow I set up a Dialog node to trigger on the trained "homeAutomation" intent and this node simply copies the input into a context variable and calls a general "Home Automation" API in APIC using the <callapi> command.
and use this as the target endpoint to invoke.
This requires the target endpoint ULR to support a REST POST of a JSON body which contains the element "Action" which is a copy of the input provided by the user to the Assistant and the expected return is a JSON object which contains an element "message" which contains the result of processing the automation action. Upon returning to Watson Assistant the dialog is defined to simply display the message returned from the Automation invoked via the configured URL.
The "Home Automation" flow I created in NodeRED is where the real action happens. The flow looks like:
Here you can see that I am using Watson Assistant (that's what's behind the "Ask Watson" node) to understand the utterance. In this case the dialog flow isn't really important, what I want to get is the intent and the entity that is being requested. To do this I set up a skill which was focused on really understanding the request i.e. is it to switch something on and off, or to list the appliances I can control etc.
and the expansion of #turnOff
Here you can see that I've used the annotation capability in Watson Assistant to mark up where the entity can be found and I stored this in the "@target" intent. With this all in place I was able to interact with my TPLink SmartPlugs in my house as shown below: