Contents

Integrate Rational Quality Manager with Slack and Jenkins

Build a custom test launcher to boost automated regression testing and reporting

Kurtis Martin
Kurtis Martin
Published on August 04, 2016
Comments

Recently my team needed to update our Rational Quality Manager test automation framework for regression testing on a cloud-based offering. We also wanted to integrate several new technologies to our test automation workflow: Git for source control, Jenkins for build/deploy, and Slack for team communication and collaboration. In this article I'll show you how I built a custom test launcher to integrate Rational Quality Manager with Slack, Jenkins, and our in-house test framework. You'll also learn a bit about Rational Quality Manager automated testing, the Rational Quality Manager Execution Tool, Node.js, Slack Webhooks, and Elasticsearch. Most important, you'll get tips for customizing Rational Quality Manager for your own needs, using these technologies or others to boost your workflow.

What you'll need

This tutorial assumes you have the following technologies installed and set up in your development environment or testing machine:

Familiarity with basic Rational Quality Manager terminology and test automation using the RQM Command-Line Adapter will help you get the most out of the tutorial. The test launcher is built using Node.js and written in JavaScript. You can apply the techniques discussed here to other technologies and programming languages.

Note that you will need to have a registered Jazz.net account in order to follow the examples in this tutorial.

Why customize Rational Quality Manager?

It's common for software developers to use a mix of technologies to manage the software development lifecycle. My team uses Rational Quality Manager for quality management and some components of Rational Team Concert for project management. That worked well as a foundation, but adding new requirements recently pushed us to think out of the box.

As our team grew we wanted new team members to be able to easily execute our SVT regressions without having to learn Rational Quality Manager. In order to encourage cross-team involvement, especially on tests that failed, we needed our regression testing results to be visible to the broader team. And we needed our regression bucket to run fast, so that it could run many times throughout the day. We were already using Git for source control and Jenkins for project builds. Our development team also started using Slack for team communication and collaboration. We needed to find a way to tie all these technologies together.

Our regression bucket consisted of roughly 20 Node.js test applications. We were able to easily run our tests using Rational Quality Manager's Command-Line Adapter (CLA). With CLA, any test tool that can run from the command-line can integrate with Rational Quality Manager. To create a regression bucket we used Rational Quality Manager Test Suites, choosing the option of parallel suite execution to help speed-up regressions. Starting with Rational Quality Manager 5.0, it's possible to run multiple automated tests in parallel using a single CLA instance. With these Rational Quality Manager features we were able to run our entire regression bucket using two CLA instances in five minutes.

Building the test launcher

In order to integrate our regression bucket with Jenkins and Slack we wrote a simple Node.js application to launch the regression suite. The launcher application exposes URL endpoints for executing the regressions, getting test results, and posting the results into Slack. You can get the source for our test launcher app below.

Get the code on GitHub

The launcher app depends on Elasticsearch, which we run alongside it on one of our test machines. Elasticsearch gives us easy programmatic access to test results. Although Rational Quality Manager provides a URL utility for programmatically accessing test results, we've found this utility cumbersome compared to accessing results via Elasticsearch.

The best way to learn is by doing, so the rest of this tutorial will step you through setting up the integrated regression testing framework, then running a test via the custom test launcher.

Step 1. Set up Elasticsearch

  • Download and unzip Elasticsearch.
  • Modify the config file, elasticsearch/config/elasticsearch.yml, as follows:
    • Look for the line that starts with: # network.host:.
    • To enable this setting, remove the hashtag at the start of the line, then modify the IP address to be the hostname of the machine that is running Elasticsearch. This will enable apps on other machines to access Elasticsearch.
  • Start Elasticsearch via elasticsearch/bin/elasticsearch on Unix or elasticsearch/bin/elasticsearch.bat on Windows.
  • Verify that you can access Elasticsearch in a browser using this URL: http://network.host:9200. Be sure to replace "network.host" with the configured hostname. This should return a JSON object showing information about your Elasticsearch instance.
  • Create a repository to store your test results. Elasticsearch calls this an index. You can think of it as a database for storing JSON objects. Issue the following command to create an index named test_results: 
    $ curl -X PUT -d '' http://network.hosts:9200/test_results

    If you’re on Windows and curl isn’t available, it's easy to issue HTTP requests via the Postman extension for a Chrome browser.
  • Tell Elasticsearch about the test result documents that you'll be storing. To do this, download the TestResult mapping file that is part of the launcher application. You'll need the mapping to set an expiration time for your test results and to specify the search behavior of key attributes. Use curl to POST the downloaded TestResult.map to Elasticsearch: 
    $ curl -i -X POST http://network.hosts:9200/test_results/TestResult/_mapping --data @TestResult.map

    If you're using Postman, simply use the contents of the TestResult.map file as the body of your POST request.

Your Elasticsearch setup is now complete!

Step 2. Setup Slack and Rational Quality Manager

For the examples in this section I assume that you have access to a Rational Quality Manager server and an active Slack account. In order to use Slack with Rational Quality Manager, you’ll need to create an incoming Webhook. A Webhook is essentially a URL that you post to, and the posted data shows up in the Slack channel associated with that Webhook. If you’re the admin for your Slack project, you can use this page to create the Webhook. Otherwise you’ll need to ask your admin to set up a Webhook for your Slack channel. Once you have the Webhook, you'll be able to post automated test results to Slack.

Eventually you’ll want to use your own automated tests, but for this exercise I’ve provided a sample test implementation that you can use to insert a test result into Elasticsearch. We’ll take a closer look at the sample test in a bit. For now, simply download the rqm-test-launcher Git project to the test machine running your Rational Quality Manager CLA, then go to the rqm-test-launcher directory and modify the sample test file named test-stub.js. Look for the variable ESEARCH_URL and modify the server name to match the hostname of the machine running Elasticsearch. Then, from the directory containing test-stub.js, run an npm install. This will download the Node.js libraries for the sample test.

Create Rational Quality Manager test scripts, cases, and suites

Next you'll create a new test script in the Rational Quality Manager UI.

  • Open the Rational Quality Manager UI. Under Construction Menu, select the link to create a new test script. Name the script "Script Example 1."
  • In the setup page, select the type Command-Line. In the Command field enter node and provide the path to test-stub.js. The setup page will look something like this:
    Figure 1. Setup page for the Rational Quality Manager test script

    View image at full size

  • On the Execution Variables tab, click the checkbox to enable built-in variables of type "Artifact Variables." Create a second test script named "Script Example 2" and enter all the same options as defined for the first script.
  • Now create two Rational Quality Manager test cases, one for each test script. Then create a test suite containing both test cases and specify that the suite should run the test cases in parallel. The setup page for the test suite will look something like this:
    Figure 2. Setup page for the Rational Quality Manager test suite

    View image at full size

  • Click Test Suite Execution Records (TSER) in the left navigation bar of the above page. Above the TSER table click Generate New Test Suite Execution Record. In the following dialog you can accept the defaults by clicking Next and Finish. Save the test suite after the Generate TSER dialog closes and be sure to note the ID value shown in the table listing the test suite execution record. You'll need this ID in a bit.

Step 3. Set up and run the test launcher

We're finally ready to jump into using the launcher app! Before running the test launcher, you should know what it does:

  • Provides a URL endpoint for launching the Rational Quality Manager test suite via the Rational Quality Manager Execution Tool.
  • Implements a task queue so that only a single instance of the suite runs at any given time. With tests so easy to launch, you’ll find more team members initiating them. Implementing a queue helps control the demand placed on Rational Quality Manager test machines running the test adapters.
  • After test completion, maps the exit codes of the Rational Quality Manager Execution Tool to meaningful status messages.
  • Queries the results stored in Elasticsearch for each individual test and posts a meaningful result to Slack.
  • Provides a URL endpoint for querying the status of a test suite run. This is key for integrating the launcher with other tools that need to know when the test suite they launched has completed.

Before you run the launcher, open launcher.js and modify the constants defined at the top of the file to point to your environment. These include your Elasticsearch server name, Slack Webhook, and the Rational Quality Manager information, including the TSER ID from Step 2.

You can run the launcher on any machine that has Node.js v0.12.7 (or higher) and Java 8 (or higher) installed. To start:

  • Find the Rational Quality Manager Execution Tool on the All Downloads tab of the Rational Quality Manager Downloads page. Download the tool and extract it into the rqm_execution_tool folder in your rqm-test-launcher project directory. (Note that you'll need to be registered and logged in to jazz.net in order to complete this step.)
  • From inside the rqm-test-launcher directory, run npm install to download the needed dependencies.
  • Run node launcher.js to start the test launcher.

Thanks to the express Node.js library, the test launcher contains a webserver that can service HTTP requests. Use your /run endpoint to trigger the Rational Quality Manager test suite:

Figure 3. Launching a Rational Quality Manager test suite

View image at full size

Instructions here are for using curl, but you can use any framework that can hit a web URL. curl works well with the Jenkins script execution build steps. My team schedules the Jenkins project that launches our morning regression test to run after our build/deploy has completed.

Test launcher endpoints

The launcher app includes two useful endpoints: run/ (which you used above) and /task, which gives you the status of the test you've launched. Notice above that the response to /run contains a runId. This is used to get the status for the run you started, as shown here:

Figure 4. Execution status

View image at full size

Initially the exitStatus attribute will be set to "pending" to indicate the run hasn’t completed. Once complete, however, the exitStatus will be a meaningful string; the summary attribute will show how many test passed versus failed; and the resultUrl will contain a link to the associated test suite result in the the Rational Quality Manager web UI. The /run and /task endpoints make it trivial to integrate Rational Quality Manager test suite execution into other frameworks. They also shield consumers of the test suite from the idiosyncrasies of Rational Quality Manager.

Querying and posting results to Slack

The test launcher uses the Node.js superagent library to issue a query request to your Elasticsearch server for retrieving all individual test script results associated with the runId. It then posts an informative message to your Slack Webhook. Here’s how the messages from the test launcher should look in your Slack channel:

Figure 5. Basic slack message

View image at full size

Note that the Rational Quality Manager test script needs to upload the test results to Elasticsearch. In test-stub.js you’ll see this being done via a POST request to Elasticsearch. The body of this request is a JSON document that will be stored in Elasticsearch. This document contains your runId, which is passed from the test launcher to the test script via Rational Quality Manager execution variables. This document also contains the test case name and the test case verdict.

For the purpose of demonstration I’ve simplified the launcher, the test script stub, and the document being stored in Elasticsearch. You can expand the example to store more data in Elasticsearch and provide more informative messages in Slack. My team actually stores the entire test output for our project in Elasticsearch, in JSON format. The links within Slack point back to our test launcher, which has the capability to serve our test suite results in HTML or JSON. We’ve also enhanced the Slack message to indicate the verdict of each individual test script. We even mixed in a little fun to show different emojis based on the overall test verdict!

Figure 6. Enhanced Slack message

View image at full size

These enhancements are highly productive for our team, because they bring immediate attention to the status of each regression test. With Rational Quality Manager you’d have to click through many pages and tabs before seeing the detailed test results, whereas in Slack the detail test output is a single click away.

Conclusion

Building an integrated Rational Quality Manager test launcher with Jenkins, Elasticsearch, and Slack has worked well for our team. HTTP endpoints have made our test launcher very easy to use and the Slack interface has improved the visibility and ease-of-consumption of test results. As a result, developers and devops team members use our regressions daily, without requiring any knowledge of Rational Quality Manager. Hopefully you’ll find this approach useful for your projects as well.

Downloadable resources

Related topics

Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational, DevOps
ArticleID=1035382
ArticleTitle=Integrate Rational Quality Manager with Slack and Jenkins
publish-date=08042016