Use Rational Quality Manager in a continuous delivery DevOps pipeline

Invoke automated test scripts with the Ant API

Automated testing is an important piece of the DevOps continuous delivery pipeline. It's easy enough to write automation test scripts and have them be triggered after a build is installed. It is more difficult to maintain a group of tests to run, the machines to run them on, and a history of the test results. This set of tasks can be daunting if you are using plain ANT and properties files to maintain the inventory.

Ajay Chebbi (ajay.chebbi@in.ibm.com), Software Architect, Rational Quality Manager , IBM

Photo of Ajay ChebbiAjay is a software architect in product development at IBM. He works on the development of the Rational solution for Collaborative Application Lifecycle Management tools. Ajay develops test run methods in Rational Quality Manager. Ajay's team uses Rational Quality Manager and to author test scripts in Rational Quality Manager.



Thomas Neal (tpneal@us.ibm.com), Advisory Software Engineer

Photo of Tom NealTom is a software engineer at IBM working on the development of internal frameworks and services, used by the Collaborative Application Lifecycle Management development teams, to promote DevOps practices.



15 July 2014

Also available in Chinese

This article describes how to use the ANT API to invoke automated test scripts. This process makes it easier to maintain the tests, the test machines, and a record of test history.

IBM® Rational® Quality Manager is a collaborative hub for test planning and managing test results. Rational Quality Manager, as shown in Figure 1, is part of the Rational solution for Collaborative Application Lifecycle Management built on the Jazz™ platform. By using Rational Quality Manager, you can maintain automated and manual test cases, you can implement traceability of test cases to requirements, and you can manage changes such as defects and development plans. It integrates well with products from other vendors. See the full list of products that integrate with Rational Quality Manager.

Figure 1. Collaborative Lifecycle Management products
CLM manages requirements, quality, and changes

Understand the phases of continuous delivery

Continuous delivery or continuous integration is a practice in which the process between checking in code and deploying the product to a production server is automated. Typically, no manual intervention is required between these two phases. In most cases, various phases exist between coding and deployment, as shown in Figure 2.

  • Developers deliver code to a branch called the main branch or the integration branch. A good practice is to deliver the feature code (the code that goes into the product) and the automation tests required to test this new code as a single delivery.
  • The code is checked into a source control branch or tree and a build is triggered. This build is performed on a fixed time schedule or the build is kicked off after a file change is detected.
  • If the build succeeds, the output of this build is installed on a staging server, which is not the production server. This staging server is now running the change set the developer has delivered.
  • A set of tests are run on this staging server to make sure that the developers' code has not regressed any functionality and that the new tests on the new functions are running fine. This is the first quality gate.
  • If all the tests have been passed (a cumulative verdict results in a "go" or "no-go" decision), the change set is promoted to what is referred to as the production branch or stable branch. This is the source code that is running in production.
  • A build is triggered on the stable branch and the build output is deployed on the production server. Sometimes a set of simple smoke tests are run on the production server to make sure that the new build that contains the developers' changes is running fine. This is the second quality gate.
Figure 2. A simple delivery pipeline
Integration branch, stable branch, production

Add a quality gate

Very often software development teams have well-established build systems. Systems range from manually built ANT and Perl scripts that orchestrate the process to more sophisticated commercial DevOps products. You can introduce the quality gate regardless of the type of system you use. This article targets the more common case in which teams have a build and deploy harness that provides a programmatic hook in ANT, Rake, Jython, plain Java™ code, or some other harness such as Maven or Hudson.


Create the automation script

Although the automation script is not strictly part of the process to add a quality gate, it is a critical aspect of ensuring quality. In most cases you already have scripted tests that the developers are running in their development environments for functional validation. Your tests are usually scripted in a test authoring platform using test practitioner tools such as Selenium, Java JUnit, Rational Functional Tester, HP QTP, or other tools. Each unit of test must be granular enough to provide useful validation of a specific failure. Avoid tests that have such a broad scope that a failure leaves a big part of the product feature under suspicion. Avoid tests with such a narrow focus that it becomes cumbersome to write meaningful test scenarios. This article uses the example of a Java JUnit test script for illustration; however, this can be extended to other test script types.


Define a test script

After the script is authored, start connecting it to Rational Quality Manager so that Rational Quality Manager can run it. Rational Quality Manager uses the concept of a script type. Each script type corresponds to an execution component called a test adapter. In the example in Figure 3, the script type is a JUnit or Selenium script used to point to a Selenium web driver test and to a JUnit test. In a continuous delivery pipeline, the test script JAR file is installed to the test machine. A later section describes how to set up an adapter.

Figure 3. Create the Rational Quality Manager test script
create test script

Define a test case

As shown in Figure 4, you need to add a Rational Quality Manager test script to a Rational Quality Manager test case. The test case is a basic, executable entity in Rational Quality Manager. The results of test executions are maintained with reference to a test case. They are called test case execution results. It is a good practice to maintain a one-to-one relationship between a test case and a test script, and to assign them the same name. You can specify the adapter at run time. You can group the test cases by using categories. Categories can be useful if you want to run reports later for specific products, components, or for any other form of grouping you choose.

Figure 4. Create a Rational Quality Manager test case
Add the test script to the test case

Create test suites

Test suites in Rational Quality Manager, as shown in Figure 5, offer a method to aggregate a set of test cases. With test suites, you can manage the set of automation test scripts that you want to run for a given kind of quality gate. For example, the test suite "Build Verification Tests" might be a collection of test scripts to quickly ascertain whether the green threads of the application are broken. If your application has a user interface, a test suite might include the set of UI automation tests that exercise the major UI functions. You might want to run this test suite on the build before you promote the build to the next phase. Next, you might have another series of test suites that tests each major feature in much greater detail. For example, a set of API tests might test underlying REST interfaces. You add all the test cases you want to run when the test suite is run.

Figure 5. Test suite
Add the test cases to the test suite

Create test suite execution records (TSER)

For the test suites you have created, create a TSER, as shown in Figure 6. Typically, you have a test plan. Add the test suite to the test plan. You create one TSER per test plan.

Figure 6. Create test suite execution records
List of records by ID

Install and deploy

This step is usually done as part of your continuous delivery harness. You might use IBM® uDeploy®, a custom build script, or a tool such as Chef, which installs the build after the source code is successfully built. In this phase, you must install your application and the test assets (such as all the Junit classes and required JARs) to a well-known location on the target machine. This location is used in defining the Rational Quality Manager test script at a later step. If you are using automated test scripts such as IBM® Rational® Functional Tester, you can also use the shared location option supported by the Rational Functional Tester adapter to get the test assets delivered to the target machine.

Rational Quality Manager adapter

Rational Quality Manager has a component called an adapter that is a remote component that is usually installed on the test machine. This adapter communicates with Rational Quality Manager server and invokes the test scripts that Rational Quality Manager instructs it to run. The results of the tests are then passed back to Rational Quality Manager.

Installation of appropriate adapter

In this Install phase you also install the Rational Quality Manager adapter. If you are writing JUnit/Selenium, you install the Selenium JUnit adapter; if you are writing Rational Functional Tester, you install the Rational Functional Tester adapter. When you configure the adapter and must specify the adapter name, use the fully qualified domain name of the machine where the adapter is running. This practice makes it easy to identify which adapter to use when running the test case and the test suite. For more details, see the documentation on running automated tests and test adapters.


Use the Rational Quality Manager ANT API to invoke tests

Your master harness is probably an ANT script. Rational Quality Manager provides an ANT-based API called the Rational Quality Manager execution tool to invoke the running of test case execution records and test suite execution records. The Rational Quality Manager execution tool is invoked from the build.xml file associated with the specific quality gate. The TSER id used is kept as a property in an associated property file. For example, the code in Listing 1 invokes the Rational Quality Manager integration tests.

Listing 1. Invoke the Rational Quality Manager integration tests
<executeTestSuiteExecRecord
   userId="${rqmUser}"
   passwordFile="${root}/${rqmPasswordFile}"
   rqmServerUrl="${rqmServerUri}"
   projectName="${rqmProjectName}"
   testSuiteExecRecordId="${rqmTestSuiteERId}"
   suiteStepAdapterIds="${testClientServer}"
   arguments="-passVariables=true -exitOnComplete=true 
-variables=com.ibm.rqm.oslc.test.serverHost:${deployedServer},server:$
{deployedServer},buildLabel:${buildLabel}" />

You can pass any configuration or execution information to the tests using the -variables option. Learn more about the execution tool and the parameters.


Use the result to move to the next step in the cycle

When the execution is complete, the Rational Quality Manager execution tool ANT task sets two properties to reflect the status of the execution and a link to the execution result. These are rqmExec:verdict and rqmExec:result_url, respectively. You can add a log file and test result to the build record. If you are using IBM® Rational Team Concert™, you can perform the following tasks:

  • Set the status of the Rational Team Concert build result to be the status of the Rational Quality Manager execution using the Rational Team Concert ANT taskdef logPublisher.
  • Add a link in the Rational Team Concert build result pointing to the Rational Quality Manager execution record using the Rational Team Concert ANT taskdef linkPublisher.

Read more about these Ant tasks. Listing 2 shows sample code to write the result and the result URL to a log file.

Listing 2. Write the results to a log file
<echo file="${root}/QMIntegrationTestsLog.txt" >
   Verdict: ${rqmExec:verdict}
   Execution Result: ${rqmExec:result_url}
</echo>

Listing 3 shows sample code to set a property based on execution status.

Listing 3. Set a property based on the execution status
<condition property="QMExecStatus" value="OK" else="ERROR">
   <equals arg1="${rqmExec:verdict}"
   arg2="com.ibm.rqm.execution.common.state.passed"/>
</condition>

Listing 4 shows how to attach the log file to the build record.

Listing 4. Attach the log file to the build record
<logPublisher 
   repositoryAddress="${repositoryAddress}" 
   userId="${jazzUserId}" passwordFile="${jazzPasswordFile}"
   buildResultUUID="${buildResultUUID}" 
   filePath="${root}/QMIntegrationTestsLog.txt" 
   label="QM Integration Test Log" 
   status="${QMExecStatus}" 
/>

Listing 5 shows how to attach the test results link to the build record.

Listing 5. Attach results to the build record
<linkPublisher 
   repositoryAddress="${repositoryAddress}" 
   userId="${jazzUserId}" 
   passwordFile="${jazzPasswordFile}" 
   buildResultUUID="${buildResultUUID}" 
   url="${rqmExec:result_url}" 
   label="Rational Quality Manager Execution Result" 
   componentName="Rational Quality Manager Execution Result"
/>

Analyze the results

After the test cycles have begun, for a particular build you can drill down to the Test Case Result from the Test Suite Result and look at the results, as shown in Figure 7. (The link to the test suite result is added to the build using the linkPublisher.) You get a graphical representation of the results as well.

Figure 7. Test Case Result details
Bar graph shows test passed and failed

After you have accumulated enough results, you can derive more intelligence by mining the result data. This is a capability available only in a quality management tool such as Rational Quality Manager. (Without it, you have only the result files on a file system.) For example, with a quality management tool, you can see execution status by category, as shown in Figure 8. Categories are covered in the section on test case definition.

Figure 8. Result analysis by categories
Bar graph shows status by category

Conclusion

Rational Quality Manager provides a very structured and easy way to manage the tests that need to be run in a continuous delivery pipeline for DevOps. You can use it in an incremental effort to build out your full-blown DevOps offering, with no rework required. Test case management and test result analysis are the two very powerful advantages you get over hand wiring your test scripts in a build harness.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into DevOps on developerWorks


  • Bluemix Developers Community

    Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.

  • DevOps digest

    Updates on continuous delivery of software-driven innovation.

  • DevOps Services

    Software development in the cloud. Register today to create a project.

  • IBM evaluation software

    Evaluate IBM software and solutions, and transform challenges into opportunities.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=DevOps, Rational
ArticleID=977724
ArticleTitle=Use Rational Quality Manager in a continuous delivery DevOps pipeline
publish-date=07152014