Regression Testing Conversations
Hickmat 100000QA3T Visits (1649)
Wow its been a busy few months and its good to have a bit of breathing space to update my blog...
Over the course of this year I've been heavily engaged on a number of chatbot implementations with a customer and one of the requirements that has been outstanding has been around the area of regression testing. Early on in this work I created an "Intent" tester to support regression testing of the intents (see Testing Watson Conversation ). This has been really valuable and is used as part of the release cycle but it doesn't address the area of validating the flow of conversations against and expected path. Based on this I set about working with a colleague (Stuart Feasy) to look at how this could be addressed using Rational Integration Tester.
Before getting into the approach that was taken I need to provide an overview of the chatbot architecture which is in place. The architecture aligns with the conversation reference architecture and looks like:
NB: The sharp eyed amongst you will have spotted a box labelled "Controlled English". This is a research area I've been experimenting with along with the IBM Emerging Technology team out of Hursley in the UK. I'll be creating another post soon to discuss that area.
From the above you can see that the "AI" layer where the chat processing happens is fronted by an API presented through API-C and this API provides the full level of detail from the AI processing layer i.e the full response payload from the call to conversation and other supporting services. Once this is passed through the "Conversation bot" layer the payload is simplified to just a response payload to be rendered by the channel. As such I wanted to interact with the API directly so that I could validate the conversation flow via other elements than just the response to the inbound utterance. A key reason for this is that the response text is changed more rapidly to meet business needs but the conversation flow remains the same. The area I wanted to focus on was the conversation context as for the chatbot implementation I am working on this conveys the right information to track that the conversation is flowing correctly.
Using this approach it was clear that Rational Integration Tester (RIT) would need to manage the state of the conversation flow which is fine as that's a standard use case for RIT. To support the creation of the conversations to replay I want to leverage the conversation data that is stored in the "MI" Cloudant database. Again RIT has the capability to access data via a number of mechanisms including REST calls so this was perfectly feasible. Armed with this I worked with Stuart to create a MVP showing the following high-level flow.