More and more we are getting requests from our customers to leverage their existing documentation on a subject they want a chat bot for.
In this blog item we call this a document based chat bot.
Consider the following scenario:
You have a chatbot that can answer questions about IT Security policies.
You also have used historic helpdesk data to train the chatbot for questions like:
- How do I reset my password?
- Why do I need to change my password every 3 months?
And while you can (and should!) improve your chatbot regularly by adding new questions and improved answers there is always this ‘long tail’ of questions. Maintaining your chatbot for all these questions is a lot of work and often there is lots of documentation available on the subject that contains the answers.
This is where the Discovery Service comes in (see https://www.ibm.com/watson/services/discovery/ for a detailed description).
The documents that you already have, can be put into a so-called collection in a Discovery Service instance and once they are processed, they can be queried by using natural language.
For the chatbot this means that when it does not have the answer to the question, it can use the discovery service to go search for an answer.
In a web client it can look like this:
Figure 1 baas chatbot searching for an answer in discovery service
The first answers (to the coffee question) come from the Watson Assistant and the rounded boxes contain text retrieved from the Discovery Service.
These are the so-called passages that the Discovery Service found to be most relevant for this question.
In the next section we will describe the steps we took to get this to work
For this solution we borrowed a lot from these two sources
In fact, to illustrate the architecture, we took the diagram from the medium post and adjusted it to fit our situation.
In our case we have a Liberty Profile Server (we call “man in the middle) that communicates with the Watson Assistant and with the Discovery Service.
First, we discuss the role of the Assistant.
Use the Assistant to find out what the customer is asking
When you work with chatbots like the Watson Assistant the first thing you do, is trying to find out what the customer is looking for. What is the user’s intent?
The customer can start with some small-talk. The customer may want the assistant to perform an action, or the customer is looking for information.
Our Discovery Service collection does not contain all the information in the world, so next to the intent we need to find out if the customer is asking for information about a certain topic.
Those topics are called entities in Watson Assistant. Remember this term, it will return.
We combine the intent and the entities in the condition for the dialog node in Watson Assistant.
The dialog node is the place where the assistant provides an answer.
For our use case this dialog node looks like this (in the IBM Watson Assistant web browser tooling)
Figure 2 Watson Assistant dialog node with action
When you have been working with Watson Assistant before, you can see an extra property called “action”.
It is this property that our “man in the middle” uses to determine if the answer from Assistant should be given to the customer OR if we should ask the Discovery Service for an answer.
Discovery Service has the answer
The Discovery Service needs content that it can search. For this blog item we used publicly available documents on policies for IT security (for our customer project we used their private documents).
Getting your documents in a usable format into the Discovery Service takes a few steps as shown in the diagram below.
Figure 3 Importing documents in Discovery Service
In the enrichment step, the Discovery Service extracts structured data from the unstructured text in the documents.
An example of the out of the box Enrichment the Discovery Service added to our documents is shown below:
Figure 4 Discovery Service enrichments
And if you look at this screenshot, you also see the possibility to get entities from the documents. Remember those from the Watson Assistant section? They will play a role in our final step, but for now what you can already do is the following
- Ask the baas chat bot for information about it security policy
- The chat bot will indicate (action property) that the information is probably available in the Discovery Service
- The “man in the middle” performs a natural query search and the relevant passages from the results are shown to the user.
This is what is shown in figure 1 above.
Add a custom domain model using Knowledge Studio
Although the Discovery Service itself has enrichments that are valid for our purpose. We can improve the enrichments by creating a custom configuration for Discovery Service using Knowledge Studio.
Knowledge Studio allows us to build domain models in a collaborative environment designed for both developers and domain experts, without needing to write code. For more information on Knowledge Studio see watson-knowledge-studio
For this purpose, we teach Watson which terms make up which entity. This process is called “annotation”. Below you can see a screenshot of the Watson Knowledge Studio annotation functionality.
Figure 5 Annotating entities in Watson Knowledge Studio
So now we have three places that contain entities that describe our domain (policy for it security).
We can now apply this newly trained model, to our Discovery Service collection.
The Watson Studio tooling allows you to deploy a model to the Discovery Service, from within the web browser!
Figure 6 Deploy WKS model to Discovery Service
Now that the Discovery Service knows about this model, we need to apply it to our collection. Again, in the web browser it goes like this. We want to extract entities (or enrich our documents with entities) using our own domain model. The custom model ID is available in the Watson Knowledge Studio tooling as result of the step above.
Figure 7 Apply custom model to Discovery Service collection
Now that we our custom entity model we can use this in the Discovery Service
Figure 8 Custom Entity Extraction available in Discovery Service
In the figure above you can recognize the entities we defined in Watson Knowledge Studio (see figure 5).
Now that we can query our collection for documents containing custom entities, we can do the same from our man in the middle. We use the entities that the Watson Assistant provides to extend the query.
Figure 9 Watson Assistant finds entities in question
This is translated into the following query to the Discovery Service (like what you can see in figure 8):
Conclusion and challenges
We have seen that we can easily forward a question as a Natural Language query to the Watson Discovery. This means newly added content can be found even when you don’t have the time to (re-)train your assistant.
We have also seen that we can improve our search results for the customer’s questions by applying the entities as a filter to the query for the Discovery Service. We can achieve this by maintaining the same entities in Watson Assistant and Watson Knowledge Studio.
While this is now a powerful document based chatbot, there are still some challenges to consider / improvements to be made
Where to maintain the entities?
Right now, we have two places where entities are maintained. There are several export and import options that can be used to maintain the entities in one place (for example Knowledge Studio) and import them in the next place (Watson Assistant).
A search result is not always an answer
When you only have a chatbot, you enter the answers in the dialog output and they will look like answers to the customer (they may be wrong, but a when question will get a time/date/situation answer, if it is a correct answer is another issue)
When you search in content, you will get the most relevant result, but not always an answer to your question.
An example would be looking for an expert. This is typically a “who” question, so you expect people in your answer.
For this use-case, we have made a solution, but the more generic challenge remains.
Below a screenshot where Discovery Service query is applying an extra filter to find only documents of the type “profile”. This example is in Dutch, which is another benefit of the IBM Watson Services, the understanding of multiple languages.
Figure 10 Watson Discovery Service finds experts
baas: create your own chat bot
assistant-with-discovery on GitHub: get started with the code and configuration
Integrate Watson Assistant with just about anything: once you known everything you need to know to query ANY back-end system, a man in the middle can perform that action for you.