Integrate Rational Quality Manager with Slack and Jenkins
Build a custom test launcher to boost automated regression testing and reporting
Recently my team needed to update our Rational Quality Manager test automation framework for regression testing on a cloud-based offering. We also wanted to integrate several new technologies to our test automation workflow: Git for source control, Jenkins for build/deploy, and Slack for team communication and collaboration. In this article I'll show you how I built a custom test launcher to integrate Rational Quality Manager with Slack, Jenkins, and our in-house test framework. You'll also learn a bit about Rational Quality Manager automated testing, the Rational Quality Manager Execution Tool, Node.js, Slack Webhooks, and Elasticsearch. Most important, you'll get tips for customizing Rational Quality Manager for your own needs, using these technologies or others to boost your workflow.
What you'll need
This tutorial assumes you have the following technologies installed and set up in your development environment or testing machine:
- Rational Quality Manager v5.0 or higher
- Elasticsearch v2.0 or higher
- Node.js v0.12.7 or higher
- Java 8 or higher
Note that you will need to have a registered Jazz.net account in order to follow the examples in this tutorial.
Why customize Rational Quality Manager?
It's common for software developers to use a mix of technologies to manage the software development lifecycle. My team uses Rational Quality Manager for quality management and some components of Rational Team Concert for project management. That worked well as a foundation, but adding new requirements recently pushed us to think out of the box.
As our team grew we wanted new team members to be able to easily execute our SVT regressions without having to learn Rational Quality Manager. In order to encourage cross-team involvement, especially on tests that failed, we needed our regression testing results to be visible to the broader team. And we needed our regression bucket to run fast, so that it could run many times throughout the day. We were already using Git for source control and Jenkins for project builds. Our development team also started using Slack for team communication and collaboration. We needed to find a way to tie all these technologies together.
Our regression bucket consisted of roughly 20 Node.js test applications. We were able to easily run our tests using Rational Quality Manager's Command-Line Adapter (CLA). With CLA, any test tool that can run from the command-line can integrate with Rational Quality Manager. To create a regression bucket we used Rational Quality Manager Test Suites, choosing the option of parallel suite execution to help speed-up regressions. Starting with Rational Quality Manager 5.0, it's possible to run multiple automated tests in parallel using a single CLA instance. With these Rational Quality Manager features we were able to run our entire regression bucket using two CLA instances in five minutes.
Building the test launcher
In order to integrate our regression bucket with Jenkins and Slack we wrote a simple Node.js application to launch the regression suite. The launcher application exposes URL endpoints for executing the regressions, getting test results, and posting the results into Slack. You can get the source for our test launcher app below.
The launcher app depends on Elasticsearch, which we run alongside it on one of our test machines. Elasticsearch gives us easy programmatic access to test results. Although Rational Quality Manager provides a URL utility for programmatically accessing test results, we've found this utility cumbersome compared to accessing results via Elasticsearch.
The best way to learn is by doing, so the rest of this tutorial will step you through setting up the integrated regression testing framework, then running a test via the custom test launcher.
Step 1. Set up Elasticsearch
- Download and unzip Elasticsearch.
- Modify the config file,
elasticsearch/config/elasticsearch.yml, as follows:
- Look for the line that starts with:
- To enable this setting, remove the hashtag at the start of the line, then modify the IP address to be the hostname of the machine that is running Elasticsearch. This will enable apps on other machines to access Elasticsearch.
- Look for the line that starts with:
- Start Elasticsearch via
elasticsearch/bin/elasticsearchon Unix or
- Verify that you can access Elasticsearch in a browser using this URL: http://network.host:9200. Be sure to replace "network.host" with the configured hostname. This should return a JSON object showing information about your Elasticsearch instance.
- Create a repository to store your test results. Elasticsearch calls
this an index. You can think of it as a database for storing JSON
objects. Issue the following command to create an index named
$ curl -X PUT -d '' http://network.hosts:9200/test_results
If you’re on Windows and
curlisn’t available, it's easy to issue HTTP requests via the Postman extension for a Chrome browser.
- Tell Elasticsearch about the test result documents that you'll be
storing. To do this, download the TestResult mapping file that is part of the
launcher application. You'll need the mapping to set an expiration
time for your test results and to specify the search behavior of key
$ curl -i -X POST http://network.hosts:9200/test_results/TestResult/_mapping --data @TestResult.map
If you're using Postman, simply use the contents of the
TestResult.mapfile as the body of your
Your Elasticsearch setup is now complete!
Step 2. Setup Slack and Rational Quality Manager
For the examples in this section I assume that you have access to a Rational Quality Manager server and an active Slack account. In order to use Slack with Rational Quality Manager, you’ll need to create an incoming Webhook. A Webhook is essentially a URL that you post to, and the posted data shows up in the Slack channel associated with that Webhook. If you’re the admin for your Slack project, you can use this page to create the Webhook. Otherwise you’ll need to ask your admin to set up a Webhook for your Slack channel. Once you have the Webhook, you'll be able to post automated test results to Slack.
Eventually you’ll want to use your own automated tests, but for this
exercise I’ve provided a sample test implementation that you can use to
insert a test result into Elasticsearch. We’ll take a closer look at the
sample test in a bit. For now, simply download
rqm-test-launcher Git project to the test machine running
your Rational Quality Manager CLA, then go to the
rqm-test-launcher directory and modify the sample test file
test-stub.js. Look for the variable
ESEARCH_URL and modify the server name to match the hostname
of the machine running Elasticsearch. Then, from the directory containing
test-stub.js, run an
npm install. This will
download the Node.js libraries for the sample test.
Create Rational Quality Manager test scripts, cases, and suites
Next you'll create a new test script in the Rational Quality Manager UI.
- Open the Rational Quality Manager UI. Under Construction Menu, select the link to create a new test script. Name the script "Script Example 1."
- In the setup page, select the type Command-Line. In
the Command field enter
nodeand provide the path to
test-stub.js. The setup page will look something like this:
Figure 1. Setup page for the Rational Quality Manager test script
- On the Execution Variables tab, click the checkbox to enable built-in variables of type "Artifact Variables." Create a second test script named "Script Example 2" and enter all the same options as defined for the first script.
- Now create two Rational Quality Manager test cases, one for each test
script. Then create a test suite containing both test cases and
specify that the suite should run the test cases in parallel. The
setup page for the test suite will look something like this:
Figure 2. Setup page for the Rational Quality Manager test suite
- Click Test Suite Execution Records (TSER) in the left navigation bar of the above page. Above the TSER table click Generate New Test Suite Execution Record. In the following dialog you can accept the defaults by clicking Next and Finish. Save the test suite after the Generate TSER dialog closes and be sure to note the ID value shown in the table listing the test suite execution record. You'll need this ID in a bit.
Step 3. Set up and run the test launcher
We're finally ready to jump into using the launcher app! Before running the test launcher, you should know what it does:
- Provides a URL endpoint for launching the Rational Quality Manager test suite via the Rational Quality Manager Execution Tool.
- Implements a task queue so that only a single instance of the suite runs at any given time. With tests so easy to launch, you’ll find more team members initiating them. Implementing a queue helps control the demand placed on Rational Quality Manager test machines running the test adapters.
- After test completion, maps the exit codes of the Rational Quality Manager Execution Tool to meaningful status messages.
- Queries the results stored in Elasticsearch for each individual test and posts a meaningful result to Slack.
- Provides a URL endpoint for querying the status of a test suite run. This is key for integrating the launcher with other tools that need to know when the test suite they launched has completed.
Before you run the launcher, open launcher.js and modify the constants defined at the top of the file to point to your environment. These include your Elasticsearch server name, Slack Webhook, and the Rational Quality Manager information, including the TSER ID from Step 2.
You can run the launcher on any machine that has Node.js v0.12.7 (or higher) and Java 8 (or higher) installed. To start:
- Find the Rational Quality Manager Execution Tool on the All Downloads tab
of the Rational Quality Manager Downloads page. Download the tool and extract it into the
rqm_execution_toolfolder in your
rqm-test-launcherproject directory. (Note that you'll need to be registered and logged in to jazz.net in order to complete this step.)
- From inside the
npm installto download the needed dependencies.
node launcher.jsto start the test launcher.
Thanks to the express Node.js library, the test launcher contains a
webserver that can service HTTP requests. Use your
endpoint to trigger the Rational Quality Manager test suite:
Figure 3. Launching a Rational Quality Manager test suite
Instructions here are for using
curl, but you can use any
framework that can hit a web URL.
curl works well with the
Jenkins script execution build steps. My team schedules the Jenkins
project that launches our morning regression test to run after our
build/deploy has completed.
Test launcher endpoints
The launcher app includes two useful endpoints:
you used above) and
/task, which gives you the status of the
test you've launched. Notice above that the response to
runId. This is used to get the status for the run
you started, as shown here:
Figure 4. Execution status
exitStatus attribute will be set to "pending" to
indicate the run hasn’t completed. Once complete, however, the
exitStatus will be a meaningful string; the summary attribute
will show how many test passed versus failed; and the
resultUrl will contain a link to the associated test suite
result in the the Rational Quality Manager web UI. The
/task endpoints make it trivial to integrate Rational
Quality Manager test suite execution into other frameworks. They also
shield consumers of the test suite from the idiosyncrasies of Rational
Querying and posting results to Slack
The test launcher uses the Node.js superagent library to issue a query
request to your Elasticsearch server for retrieving all individual test
script results associated with the
runId. It then posts an
informative message to your Slack Webhook. Here’s how the messages from
the test launcher should look in your Slack channel:
Figure 5. Basic slack message
Note that the Rational Quality Manager test script needs to upload the test
results to Elasticsearch. In
test-stub.js you’ll see this
being done via a
POST request to Elasticsearch. The body of
this request is a JSON document that will be stored in Elasticsearch. This
document contains your
runId, which is passed from the test
launcher to the test script via Rational Quality Manager execution
variables. This document also contains the test case name and the test
For the purpose of demonstration I’ve simplified the launcher, the test script stub, and the document being stored in Elasticsearch. You can expand the example to store more data in Elasticsearch and provide more informative messages in Slack. My team actually stores the entire test output for our project in Elasticsearch, in JSON format. The links within Slack point back to our test launcher, which has the capability to serve our test suite results in HTML or JSON. We’ve also enhanced the Slack message to indicate the verdict of each individual test script. We even mixed in a little fun to show different emojis based on the overall test verdict!
Figure 6. Enhanced Slack message
These enhancements are highly productive for our team, because they bring immediate attention to the status of each regression test. With Rational Quality Manager you’d have to click through many pages and tabs before seeing the detailed test results, whereas in Slack the detail test output is a single click away.
Building an integrated Rational Quality Manager test launcher with Jenkins, Elasticsearch, and Slack has worked well for our team. HTTP endpoints have made our test launcher very easy to use and the Slack interface has improved the visibility and ease-of-consumption of test results. As a result, developers and devops team members use our regressions daily, without requiring any knowledge of Rational Quality Manager. Hopefully you’ll find this approach useful for your projects as well.
- Web development
- Continuous engineering
- Attaching content and links to Slack messages
- Using Slack Webhooks to send data in realtime
- Integrated testing with the RQM Command-Line Adapter