The Voice Agent Tester (VAT) is a new tool for running manual, automated, and load tests against Voice Gateway and/or Voice Agent deployments.

It can also be used to run tests against any Interactive Voice Response (IVR) system. The goal of the VAT is to make it easy for end users to perform large sets of regression tests against their voice agent by simulating a real VoIP call into the system under test.

The VAT simulates speech using Watson TTS or WAV files and interprets audio coming from the Voice Agent under test by first transcribing speech using Watson STT and then comparing the transcriptions to expected results to verify the accuracy of your Voice Agents dialog flows.


The Voice Agent Test relies on Voice Gateway to simulate a caller. The following two features of IBM Voice Gateway are used to create the VAT:

  • The ability to add a Service Orchestration Engines to drive the tests
  • The outbound dialing interface for making outbound calls

Voice Agent Tester Service Orchestration

A Service Orchestration Engine (SOE) is a REST-based endpoint that sits between the Voice Gateway and Watson Assistant. The Voice Agent Tester runs as an SOE, enabling it to customize the conversational test flows. This SOE also utilizes the outbound dialing interface for initiating test calls.

Service Orchestration Engine

Voice Gateway outbound dialing interface

Outbound dialing is a new feature of the Voice Gateway 1.0.2 release that exposes a REST API to make outbound calls from the Voice Gateway to a SIP endpoint. You make a request to the REST API with the JSON body (as described in the IBM Knowledge Center), and the Voice Gateway will connect to the endpoint specified.

Outbound Calls Feature

The Voice Agent Tester is a stateless, REST-based microservice which uses:

  • A Cloudant/CouchDB for persistence of all test-related objects
  • A Voice Gateway that starts and drives the outbound calls
  • A SOE to define, drive, and validate the conversation
  • Watson Speech to Text to transcribe audio coming from the Voice Agent under test
  • Watson Text to Speech to simulate a caller’s audio to drive the test dialog flows

Voice Agent Tester

Test resource details

There are four main resources that the VAT uses for testing.  Each test resource is stored and maintained in either Cloudant or CouchDB. These are described below in detail: 

  • Test cases are the primary resource for running the tests. Test cases contain the outline of the conversation you wish to exercise in the Voice Agent under test. The most important part of the test case is the turns array, which is an ordered array of JSON objects that contains what you expect to receive and what you would like to send back. Test cases can be very simple (with only a couple of turns) or very complex (with various turns and various types responses).
  • Workers are the second most important resource when executing tests. Workers define which test cases to run, the number of test iterations, and which Voice Agent to call when executing the test. 
  • Jobs are the actual execution of the workers. When a job is created, the test cases defined by the worker associated with the job are executed in sequential order. Jobs contain all the metadata for the running and finished tests, including the errors, time taken, status, executed test cases with their versions, and other properties.
  • Batch jobs are an easy way to start and manage multiple concurrent jobs. Running multiple jobs can be difficult to monitor and manage. A batch jobs makes it easier to do this.

Running tests

The usual steps to follow for testing would be as follows:

  1. Create your test cases. 
  2. Create a worker.
  3. Create a job or batch job to start the test execution.

When the job starts, the system will grab the first test case listed in the worker and make a request to the Voice Gateway’s outbound REST API to initiate the call for the test. The VAT will follow the conversation flow defined within the test case being executed until it either completes or fails. If there is a failure, the call will end, the failure will be marked on the job, and, if necessary, another call will start to continue to the next test. Once all tests have completed, the results will be stored with the job in the configured noSQL DB.

Learn more

You can learn more about the new Voice Agent Tester by connecting to the related documentation. These helpful videos will also help jumpstart your development using the Voice Agent Tester:


More from Announcements

IBM TechXchange underscores the importance of AI skilling and partner innovation

3 min read - Generative AI and large language models are poised to impact how we all access and use information. But as organizations race to adopt these new technologies for business, it requires a global ecosystem of partners with industry expertise to identify the right enterprise use-cases for AI and the technical skills to implement the technology. During TechXchange, IBM's premier technical learning event in Las Vegas last week, IBM Partner Plus members including our Strategic Partners, resellers, software vendors, distributors and service…

Introducing Inspiring Voices, a podcast exploring the impactful journeys of great leaders

< 1 min read - Learning about other people's careers, life challenges, and successes is a true source of inspiration that can impact our own ambitions as well as life and business choices in great ways. Brought to you by the Executive Search and Integration team at IBM, the Inspiring Voices podcast will showcase great leaders, taking you inside their personal stories about life, career choices and how to make an impact. In this first episode, host David Jones, Executive Search Lead at IBM, brings…

IBM watsonx Assistant and NICE CXone combine capabilities for a new chapter in CCaaS

5 min read - In an age of instant everything, ensuring a positive customer experience has become a top priority for enterprises. When one third of customers (32%) say they will walk away from a brand they love after just one bad experience (source: PWC), organizations are now applying massive investments to this experience, particularly with their live agents and contact centers.  For many enterprises, that investment includes modernizing their call centers by moving to cloud-based Contact Center as a Service (CCaaS) platforms. CCaaS solutions…

See what’s new in SingleStoreDB with IBM 8.0

3 min read - Despite decades of progress in database systems, builders have compromised on at least one of the following: speed, reliability, or ease. They have two options: one, they could get a document database that is fast and easy, but can’t be relied on for mission-critical transactional applications. Or two, they could rely on a cloud data warehouse that is easy to set up, but only allows lagging analytics. Even then, each solution lacks something, forcing builders to deploy other databases for…