Automate tests without compromising coverage of the environment

Using Rational Quality Manager to mitigate the risk of leaking environment-specific defects

Automated testing is beneficial compared to manual testing in many ways, such as accuracy, saving time, resource utilization, and early regression detection. However, it is equally important that automated tests are executed in correct test environments, according to the test plan. Failing to do that could lead environment-specific defects to leak to the stakeholders. Pramod Chandoria describes how can you make sure that automated tests run on the correct machine, based on the test environment coverage planned, and mitigate the risk of leaking platform-specific issues. He also explains how to configure, manage, and run automated tests in Rational Quality Manager.

Share:

Pramod Kumar Chandoria (pchandor@in.ibm.com), Advisory Software Engineer, IBM

Author1 photoPramod Chandoria works at the IBM Rational Software Lab in India. He represents the Test Automation team in defining the first version of the Automation OSLC specification. He has been leading design and development of Rational Quality Manager since Version 1.0. Test execution and integration with test automation tools are his areas of expertise. Pramod has a degree in computer science from the Indian Institute of Technology, Roorkee, and has nearly nine years of product development experience.



10 April 2012

Also available in Chinese

Introduction

Software testing is an important part of the software development lifecycle, because it assures developers and owners that the software works as expected. With the increased complexity of today's software and the software environment, it is becoming difficult to manage testing manually, alone. Organizations are seeking solutions to automate the testing to mitigate the risk of regression defects leaking into the end product.

Fortunately, there are many automation tools available to automate the tests, including IBM® Rational® Functional Tester and Rational® Performance Tester. There are initial problems associated with automating the test and with maintaining the tests as applications change. However, after a well-proven suite of automated tests is created, it proves useful in long run.

After tests has been automated, the next challenge arises while running the automated tests on all supported platforms, such as Microsoft Windows 2008 Server, Android, Firefox 10, IBM® WebSphere® Application Server, Oracle, IBM® DB2, and so forth. Every year, the list of supported platforms changes due to the changes in versions, support for new platforms, and other factors.

As a recent example, Mozilla has released many versions of the Firefox browser since Version 3.0. This means that it is not only important to run the test, but it is equally important to run all tests on all of the supported test environments. This article explains how you can make sure that your automated tests automatically cover the platforms in your test plan, thus ensure maximum test coverage and detection of defects before you release software.

Figure 1. Manual script
Manual script Editor with manual steps

Larger view of Figure 1.

Automated testing using Rational Quality Manager

IBM® Rational® Quality Manager is a web-based test management tool. It integrates with most of the leading automation tools, including Rational Functional Tester, Rational Performance Tester, Rational® Robot, HP Quick Test Professional, and many more. The Rational Quality Manager open RESTful web API makes it easy to integrate with any other tools.

Essentially Rational Quality Manager provides two types of scripts:

  • A native manual script with natural language steps, as shown in Figure 1, above
  • A remote script, which represents an external automated script as shown in Figure 2, below
Figure 2. Remote functional test script
Remote script editor for functional tester script

A manual script captures manual steps in natural language statements, as Figure 1 shows, and is useful for manual testing. Remote scripts, on the other hand, represents placeholders for the actual external automation scripts that can be run by specific external automation tools as shown in Figure 2, above. Each remote script has a type that is based on the type of automation tool. Rational Quality Manager cannot run the remote script; it merely coordinates the execution by delegating it to the automation tool that can actually run the remote script.

Automated tests are typically run on lab machines where the application under test (AUT) and the automation tool are configured to run automated tests. To integrate with Rational Quality Manager, each automation tool needs to have a lightweight utility called an execution adapter to bridge the gap between Rational Quality Manager and the automation tool. Each Rational automation tool has Rational Quality Manager-specific adapter to connect the two.

Planning automated test execution

Rational Quality Manager provides test cases that can be associated with manual or remote scripts to execute the test cases. Planning results in the creation of test case execution records. These records typically cover which test cases to be executed, when to execute them, who will execute them, and — most relevant to this article — which test environment to use. A single test case often has multiple execution records, with each execution record covering a unique test environment.

Registering an execution adapter

When an automation tool-specific adapter is started, it registers to Rational Quality Manager and thereby makes itself available for execution of automated tests. As a result of this registration, Rational Quality Manager creates two resources: adapter and lab resource (test machine). The adapter represents the registered adapter, and it is linked to the lab resource, which captures machine details where the adapter is running. By default, the adapter captures only the machine name, IP address, domain name, and so forth.

To automatically provision full machine details, Rational Quality Manager can integrate with IBM® Tivoli® Provisioning Manager, Tivoli Application Dependency Discovery Manager, and open source StAX (Streaming API for XML) or STAF (Software Testing Automation Framework). The Integration section in Rational Quality Manager online help provides more information (see Resources for a link). Alternatively, you can manually edit the lab resource in Rational Quality Manager to provide details about the machine where the adapter is running.

The command-line adapter (CLA) is included with Rational Quality Manager (see the link in Resources for where to get details). CLA can execute any script that can be run from the command line. It could be as simple as ping command to as complex as a path to a .bat file that contains a complex script. Automation tool-specific adapters are often included when you install the automation tool.

Follow these steps to register a command-line adapter to the Rational Quality Manager server:

  1. Copy the command-line adapter .zip file from the Rational Quality Manager installation directory to test machine:
    ...\JazzTeamServer\server\conf\qm\adapters\ RQMCommandLineAdapter.zip
  2. Extract the file into a folder.
  3. Make sure that Java Runtime Environment (JRE) version 1.5 or later is installed on the test machine
  4. Open the command window and change the directory to the location where you extracted and saved the contents of the file.
  5. Run the following command in command window (italic type indicates variables for you to replace with your information):
    start.bat -repository https://rqmservername:9443/qm -user user ID -password password -adapter uniqueAdapter ID -adapterName your adapter -projectArea name of the Rational Quality Manager project area

Upon successful registration, the adapter console will display a confirmation message, as shown in Figure 3.

Figure 3. Registering the command-line adapter
Command line adapter registered and polling server

Viewing registered execution adapters in the adapter console

In this section, you will learn how to view all execution adapters registered from different machines to Rational Quality Manager server.

  1. Open your web browser, and enter the Rational Quality Manager URL, for example:
    https://server:9443/qm/web/console
  2. Provide valid login information to log into the project area
  3. Click Execution > Adapter Console to open the console
  4. Verify that the adapter that you connected from test machine is listed, as shown in Figure 4, and that the Health column for it shows a green icon. (A red icon means that the adapter is not active for a long time after registration.)
Figure 4. Viewing adapter console
Adapter console shows registered adapters

Larger view of Figure 4.

Updating test environment information

After execution adapters are registered, Rational Quality Manager creates a lab resource that corresponds to each machine where the adapter is running. By using this lab resource, Rational Quality Manager determines the test environment where the adapter is running. As mentioned earlier, registering an adapter to Rational Quality Manager provides merely minimal information about the machine, such as the host name and IP address. Such information is not sufficient. You can update test environment information in two ways. One way is to use provisioning tools from Tivoli that will update Rational Quality Manager automatically (see the link to
"Rational Quality Manager and Tivoli" in the Resources section).

This article covers only how to update the test environment information for lab resources manually:

  1. Follows steps in previous section to open the adapter console.
  2. Click the Machine Name link to open the lab resource associated with the adapter.
  3. Click the Operating System section. Update it with operating system information.
  4. Click the Hardware section and provide any details about the machine,
  5. Click the Software section and add all software that you can to update, such as Browser, Database, Application Server, and so on.
  6. Click Save to save the test environment information.

Repeat these steps for each adapter that you have registered.

Import external automated scripts

You can import automated tests that were created by using external Rational software into Rational Quality Manager. Follow the instructions at "Referencing automated test scripts" and the relevant subsections (see Resources for a link). You should be able to import external automated tests into Rational Quality Manager by creating a proxy script for each automated test.

Configuring custom test environment coverage

This section provides step-by-step guidance to make sure that when Rational Quality Manager selects or proposes an execution adapter, the machine where the adapter is running complies with the test environment specified for the test case execution record.

  1. Log in to Rational Quality Manager as a Project Administrator.
  2. Click Application Administration > Manage Project Properties (see Figure 5).
Figure 5. Manage project properties
Manage Project Properties on the drop-down menu
  1. In the Manage Project Properties view, under the Properties section, click Execution Preference as shown in Figure 6.
Figure 6. Execution preference
Execution Preferences selected under Properties

Larger view of Figure 6.

  1. By default, Rational Quality Manager does not make any attempt to match the test environment when executing automated tests. To change this, select the Custom Search radio button, and then select the check box for the type of test environments (operating system, browsers, and so forth) that you want to make sure are matched for searching for the adapter. If you want to match a specific test environment, choose the Environment radio button instead.
  2. Click Save to update execution preferences.

By following these steps, you have configured Rational Quality Manager to run automated tests on the correct machine. This means that if you create an execution record to execute on Windows XP with the Firefox browser, Rational Quality Manager will verify that the chosen machine adheres to the test environment requirement.

Using a test cell for preparing robust test bed

Often, multiple teams work together on single project. Each team will have its own set of execution adapters connected to a Rational Quality Manager project. You do not want your testing to interfere with someone else's test environment, of course. By using a test cell, you can limit the scope of machines where automated tests will be executed.

To create a test cell:

  1. Click Lab Management > Create > Test Cell.
  2. Provide a logical name for the test cell.
  3. Click the green plus sign icon under the Lab Resource section (see Figure 7).
  4. Add all machines where you want to run your tests.
  5. Click the Save button to save the test cell.
Figure 7. Test cell
Test cell editor associated with lab resources

Larger view of Figure 7.

Running an automated test

After test case execution records are generated, you can run individual records manually by clicking the Run button from the Browse Test Case Execution Records view. This is useful when you want to run individual automated tests on an as needed basis.

Often, you will want to create a suite of automated tests and run the full suite to save time. In some situations, you might want to run automated test regularly, according to a schedule. You might also want to run automated tests whenever a new build is created to make sure that there is no regression in the build and to detect problems early, before the whole test team is engaged in testing. In this section, you learn how to run test suites manually for automated test, as well as more advanced options so you can design a workflow of automated tests that can be scheduled to run many times.

Running an automated test by using a test suite

  1. Create a test suite by following instruction given in the information center (see Resources). Add all of the automated test cases to it, and make sure that you specify the test environment for each test case.
  2. To specify the test environment:
    1. Check rows and then click Change Execution Properties.
    2. Select the test environment.
    3. Click OK, and Save your test suite.
  3. Click Run in the test suite editor.
  4. By default, Rational Quality Manager will select an adapter for each test. To change the default adapter, click Edit Adapter. Alternatively, let Rational Quality Manager distribute tests at run time by selecting the test cell. This will allow selection of adapters to be done by the software within the specified test cell at run time, based on the availability.
  5. Select Run in parallel, in case tests do not depend on each other, to minimize the overall suite execution time.
  6. Click OK.

Scheduling an automated test

  1. Click Execution > Create Execution Schedule.
  2. Provide a name for the test schedule.
  3. Select the scheduling options available (see Figure 8):
    1. Schedule for one time at specified time
    2. Schedule daily for given time
    3. Schedule weekly on specified days
    4. Schedule based on build availability

Note:
The first three options are based on the time schedule. The last option is more useful when you want to run tests each time that a new build is created.

Figure 8. Types of schedules
Different type of scheduling available

Larger view of Figure 8.

  1. In the Steps section, click the + (plus sign) button to add a test suite execution record, as shown in Figure 9. Filter to get the record that you want, and add to the Steps section. You can add as many execution records as you need.
Figure 9. Add a schedule step
Adding a test suite step in the execution schedule

Larger view of Figure 9.

  1. Rational Quality Manager automatically selects available adapters to use for execution. However, to make better use of available adapters at run time, select Choose machines at run time in toolbar. The selected adapter will be set to "Search runtime" as shown in Figure 10.
Figure 10. Search for the adapter at run time
Search Runtime selected

Larger view of Figure 10.

  1. Associate a test cell with the execution schedule if you wish to isolate the machines where Rational Quality Manager should search for available adapters and not interfere with another test environment.
  2. Click Save.

Now all steps added in the Execution Schedule will be executed in sequence whenever the schedule is triggered.

Tips:
To manage a scheduled job, select Next scheduled time. Or to cancel schedules, click Execution > Browse > Scheduled Jobs.


Summary of what you have learned

In this article, you learned how to use Rational Quality Manager to run and schedule automated tests for various types of test automation tools, including command-line scripts. The article explained how you can configure Rational Quality Manager so that automated tests run in the test environment according to your test plan and to ensure that all supported platforms have been properly covered. You also learned how to better manage automated testing by using test cells.


Download

DescriptionNameSize
Code sample for a test cell.TestCell1080

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational, DevOps
ArticleID=809020
ArticleTitle=Automate tests without compromising coverage of the environment
publish-date=04102012