Introduction to IBM Rational ClearQuest test management: Part 2. A real-world example of using the standard features

Including test results reporting options

This second article of this series walks you through how a testing team sets up and customizes the test management features, runs the tests, and then uses the reporting functions. In Part 1 we take the reader through the main features of IBM® Rational® ClearQuest® test management and provide guidelines for implementation that take into account the structure and organization of the test team.

Susan Manupelli (smanupel@us.ibm.com), Advisory Software Engineer, IBM Japan

Susan Manupelli is an Advisory Software Engineer and Quality Engineering lead working at the IBM Rational office in Lexington, Massachusetts. She has fifteen years of industry experience in software testing and test management. While at IBM she has worked predominantly with the Automated Software Quality products including Rational Robot, Rational TestManager, Rational Functional Tester and ClearQuest test management. She was part of the team that helped deploy ClearQuest test management across Rational and has earned three "Bravo!" awards for her contributions to IBM. Recently she has joined the Rational Build Forge team.



27 November 2007

Also available in Chinese

With Part 1 as background information, let's walk through using IBM® Rational® ClearQuest® test management features for planning a typical test.

Planning the test

Scenario: XYZ Corp. has eight software projects in the works. They have set up eight asset registries in ClearQuest test management. The code names for their projects are names of islands, thus the asset registries are named after them (see Figure 1). Two of the projects (Aruba and Bermuda) have geographically distributed teams, so the ClearQuest administrator has set up IBM® Rational® ClearQuest® MultiSite to enable remote teams to access ClearQuest.

Figure 1. Asset registries
image of ClearQuest workspace

Standardize configurations and naming conventions

All of the software products under test are supported on three primary platforms: Microsoft® Windows®, UNIX®, and Linux®. Knowing that the configurations are shared among all asset registries, the project managers get together to decide on a set of configurations that will be suitable for all projects. They use a prefix for these global configurations to indicate the overall priority of that configuration. However, two of the eight projects require unique configurations. For this reason, in addition to the global set of configurations prefixed with P#, several other configurations have been created. To easily identify which projects those configurations apply to, the configuration name is prefixed with the first two letters of the project code name. Figure 2 shows the attributes and values.

Figure 2. Configuration Attributes
image of ClearQuest workspace

Figure 3 shows the configurations.

Figure 3. Configurations
image of ClearQuest workspace

Coordinate use of iteration names

The next task for the project managers is to coordinate on the use of iteration names. Standardizing a naming convention for this will enable reporting across multiple projects, particularly those included in the same releases. Of the eight projects being managed, four of them are in the same software release, while the rest are standalone products. All projects follow similar processes; therefore, the managers devise a naming convention that includes the major release version and minor release version, along with the development and test phase. Because iterations are specific to an asset registry, they need to create the necessary iterations in each asset registry.

The first four projects (Aruba, Bermuda, Fiji, and Hawaii) are all part of Release 4, Version 2. Aruba and Bermuda are still in the construction phases, undergoing functional verification testing. Fiji and Hawaii are further along and at the system verification testing stage. Thus, they create the following iterations for these projects:

  • Aruba: Rel4_V2_C1FVT. Rel4_V2_C2FVT
  • Bermuda: Rel4_V2_C1FVT. Rel4_V2_C2FVT
  • Fiji: Rel4_V2_TSVT1
  • Hawaii: Rel4_V2_TSVT1

As these teams move forward in the release cycle, they can add iterations by following the same naming convention. By doing this, they will be able to run reports based either on the test phase or at a higher level across the specific version. Notice that it is fine to have iterations with the same name in different asset registries. Behind the scenes, ClearQuest test management automatically prefixes the asset registry name to the iteration name, so that the XYZ team can always distinguish one from the other.

The remaining projects are for standalone products that are on separate release schedules. However, for consistency, the managers follow the same naming convention. The iterations for these projects are set up as follows:

  • Iceland: Rel8_V1_C1FVT
  • Maui: Rel5_V7_C1FVT
  • Nantucket: Rel3_V6_TSVT1
  • Puerto Rico: Rel3_V2_TSVT1

Decide where and how to store scripts

While the project managers coordinate on configurations and iterations, the ClearQuest administrator works with the test leads to determine the requirements for storing test scripts. Most teams use IBM® Rational® Manual Tester for scripting end-to-end, system-level tests. Those teams need to establish a local server to house their Rational Manual Tester scripts. In their asset registry, they establish a file location to point to the server that contains scripts.

For the Aruba project, the test team is geographically distributed, but each team is responsible for creation and execution of their own scripts. Scripts do not need to be shared across locations. In this case, they establish two file locations in the Aruba asset registry. One is local to Location 1, and the other is local to Location 2.

The Bermuda project team is also geographically distributed, and they use IBM® Rational® Functional Tester. However, for this team, one site develops the Rational Functional Tester scripts, and a remote site is responsible for running tests. In this case, both sites need access to the same scripts; therefore, they use IBM® Rational® ClearCase® and store the Rational Functional Tester scripts in a multi-sited ClearCase versioned object base (VOB). In their situation, one file location needs to be established to point to a ClearCase VOB.

Establish the test plan hierarchy

Meanwhile, as the configurations, iterations, and file locations are being finalized, the test leads begin to put their test plans for the upcoming release into ClearQuest test management. The test plans for functional testing are categorized by feature area. Some teams also need to do system-level testing, so they organize their longer end-to-end tests into a single test plan folder. There is a Performance team that has test plans for several of the projects, so they may have users accessing more than one asset registry regularly. Because the Performance team wants to keep the same test plan hierarchy in all of the asset registries that they are using, they create a hierarchy in one asset registry, and then use the Duplicate feature to copy the hierarchy to the other asset registries.

The hierarchy of test plans for the Fiji asset registry looks like Figure 4.

Figure 4. Test plan hierarchy
image of ClearQuest workspace

Notice the subplans under the Performance test plan:

  • Performance
    • 01_Optimum Env
    • 02_Stress
    • 03_Load

There was a certain logical order in which the test lead wanted to have these listed. ClearQuest test management lists assets alphanumerically. Therefore, when these test plans were created, they were prefixed with a two-digit numeral so that they appear listed in a specific order.

Add test cases

After the test plan hierarchy is established, test designers can begin to add test cases. As they create test cases, if they know that all configured versions of a certain test case will share exactly the same test script, then they will develop the test script and associate the test script with the test case. By doing it that way, the configured test cases will inherit the script association. This is a big time-saver.

Note:
When the test designers are at this stage, if you are scripting in Rational Manual Tester or Rational Functional Tester, it makes sense to run ClearQuest test management from within the Rational Manual Tester or Rational Functional Tester shell. This way, you can seamlessly move from designing and scripting to test case development in ClearQuest test management. Of course, you can do all of the test planning discussed here so far from within the Rational Manual Tester or Rational Functional Tester shell. However, one of the nicest features of ClearQuest test management is that if your high-level planning is done by a project manager or team lead does not normally have the scripting tools installed, he or she can use the rich client version of ClearQuest test management for all of the steps leading up to scripting without having to install the scripting tools. On the flip side, test designers can enjoy the convenience of having ClearQuest test management run from within the same shell as the test scripting tools.

Create configured test cases

After the test cases are set up, the team can begin to tag their test cases with configurations to create configured test cases (CTCs). If they have multiple test cases that will be run on the same platforms, they can select multiple configurations simultaneously and apply the configurations to several of them in batch mode. This is also a big time-saver.

Figure 5 shows test cases belonging to a scenario subplan that has been configured for two specific platforms. Notice that this screen capture shows a view of ClearQuest test management from within Rational Manual Tester.

Figure 5. Test case hierarchy
image of Manual Tester workspace

The team's final step in test planning involves "tagging" their configured test cases with the iteration for which they intend to run them. This is another step that they can do easily in batch mode by selecting several CTCs at once and associating them with one or more iterations. After this is done, they can create queries and reports to show the configured test cases that they have planned. (This will be covered in the sections that follow.)


Creating queries to show planned tests

When they have finished your planning, the team wants to see all of the CTCs that they have planned for the iteration. They can create a query that displays important information about the CTC:

  1. The record type that they need to query against is TMConfiguredTestCase.
  2. At a minimum, they will filter based on the iteration, but they can choose from a variety of important fields to display in the query.

Figure 6 shows a sample query that returns planned CTCs for the iteration but displays the parent test case, as well as the parent test plan. These are the display fields used:

TestCase.ParentPlan.PlarentPlan.Headline 
TestCase.ParentPlan.Headline
TestCase.Headline
ID
Headline
Configuration
Owner
TestType
Script

The first three field names plus the Headline field are set to sort in ascending order, in that order (1 through 4). The rest are set as No Sort.

Figure 6. Query to show planned tests
image of ClearQuest workspace

Running this query yields the results shown in Figure 7.

Figure 7. Query result
Screen output

You can tell from this query that, of the 21 planned tests, 9 of them are not yet associated with test scripts. This is an indication that someone has additional planning to do.

Tip:
Another ClearQuest feature is the ability to modify records from the Query Results view in batch mode. Based on the screen capture in Figure 7, you can see that, currently, all configured test cases are assigned to Admin. In batch mode, you can now select multiple rows in the query results and modify the CTCs to assign them to their rightful owners. This is a very useful feature if resources are shuffled and test cases need to be reassigned to new owners midstream.


Running the test

When the team has finished planning, tagging tests with their proper iterations, and assigning them to the appropriate owners, it is time to begin testing. The administrator can modify the query used in the previous section to show the planned test cases for the current user. Then each tester can import and run the query to show their own assigned tests for the iteration. The query filters on the iteration under test, and the owner is set to [current user]. This way, the user who is currently logged in can run the query and see what planned tests are assigned.

Let's say that Jane is working on the system tests for Microsoft Windows on the Rel4_V2_TSVT1 iteration for the Fiji project. Her query will return what you see in Figure 8.

Figure 8. Query results by owner
image of Manual Tester workspace

Now Jane can easily see a complete list of what she is responsible for running. Notice that she is viewing all of this ClearQuest data from within Rational Manual Tester. She can execute directly from this query, or she can return to the tree view and run her tests from there. She decides to execute from her My Planned Tests - ClearQuest Query Results view.

  1. She highlights a row and selects Execute from the drop-down menu.
  2. Immediately, the Execute Test dialog displays.

Because this CTC is associated with more than one iteration, Jane needs to pick the correct iteration to associate with the test log. She also needs to identify the build number.

  1. If the build has been created before, she can just select it from the list. Otherwise, she can create it at run time.

Build records also need to follow some kind of naming convention to be consistent. Jane doesn't want to identify a build with a particular iteration or test phase, because there are other teams using the same build stream, but they are running different iterations because of their testing schedules.

  1. Therefore, she names the build based on build number and prefixes it with the major release and version identifiers: Rel4_V2_B2353 (see Figure 9).

Differences between test and script logs

It is important to understand the difference between the test log record and the script log.

  • The test log record is a record in the database that retains info about the execution, including the iteration, the final verdict, the build, and any associated defects.
  • The script log is generated by the scripting tool and includes specifics about which verification points passed and failed. The script log is saved in the file system where the scripts are stored.

The script log is not saved into ClearQuest test management. However, the test log that is saved in ClearQuest test management retains a link to the script log. If Jane ever needs to view the defect or the log, she can easily navigate to it from the original configured test case. From there, she can select the Execution tab. This tab will show all of the test logs that were committed to the database, along with the verdict (see Figure 10).

Figure 9. Specifying iteration and build
image of workspace
  1. Jane clicks Finish in the Execute test window, and the Rational Manual Tester script displays.
  2. She records her verification points and finishes the test.

The result displays in the Test Results view. At this point, Jane needs to decide if the result is valid and if this is the verdict that she wants to commit to the database. In this case, the test has failed. Jane decides she does want to commit the test log with the failed verdict, and she also wants to log a defect.

  1. From the Test Results view, she uses the drop-down menu to click Submit Defect.

This launches the New Defect window. After the new defect is saved, it also commits the test log record. The test log record will be associated with the newly created defect.

Figure 10. Configured test case view
image of workspace
  1. From this view, she can select one of the logs to open.
  2. In the test log, she can then navigate directly to the log file for the script and view any associated defects from the test log.

Tip:
You can also run test scripts directly from the test tools, and then import the results into ClearQuest test management. Using that method enables you to create and commit a test log record by importing a script log. This is another way of integrating ClearQuest test management with a test team that may be executing remotely.

All of the testers in the team go through this process of executing tests, and then committing verdicts and test log records to the database for the given iteration.


Useful reporting with ClearQuest test management

Now that the XYZ Corp. team has begun to run tests, they want to be able to report on progress. The test lead wants to see a list of all CTCs that have been executed, along with their results. ClearQuest test management includes a query in the TM Queries folder, under the Public Folders area, that they can use as a starting point. It is called "Configured Test Cases -- verdict history," and the query looks what Figure 11 shows.

Figure 11. Verdict history query
image of ClearQuest workspace

If the team runs this query with no modifications, it will enable the test lead to choose one or more asset registries and see verdict histories for all tests that were ever executed in the selected asset registries. Figure 12 shows the results of this default query for the Fiji asset registry.

Figure 12. Verdict history query results
image of ClearQuest workspace

Notice that the display shows the verdict history across more than one iteration. Although this query is useful, what the test lead really wants to see is the verdict history for the iteration currently under test. To do this, the test lead modifies the query in two ways:

  • Changes the filter to always query the Fiji asset registry, specifically
  • Changes the filter to add a check for the iteration that was assigned to the test log (TestLogs.Iteration)

The new query looks like the one in Figure 13.

Figure 13. Modified Verdict History query
image of ClearQuest workspace

The new query returns the results shown in Figure 14.

Figure 14. Modified Verdict History query results
image of ClearQuest workspace

Now the test lead can see that only two scenarios have been run against the Rel4_V2_TSVT1 iteration for the Fiji project, so far. Given that the query includes a filter for TestLogs.Latest=True, it will return only the latest verdict that was committed for each test for the given iteration.

Resources

Learn

  • Find out more about the test management capabilities in IBM Rational ClearQuest: Run and analyze software tests with Rational ClearQuest Test Management, an IBM® developerWorks® tutorial by Brian Bryson (October 2006).
  • FAQ: Enterprise Test Management with IBM Rational ClearQuest, Version 7
  • Enroll in RT210: Essentials of Test Management with IBM Rational ClearQuest V7.0. Build test plans, test cases, and test suites that contribute to a successful test effort. The robust reporting capabilities of IBM Rational ClearQuest allow you to easily view the status of your testing and the quality of you product. These capabilities are available in the IBM Rational Software Development Platform via a ClearQuest plugin (test management package). Students can also complete lab exercises to reinforce critical concepts and tool functionality. The focus is on the practical application of IBM Rational ClearQuest to resolve common test management challenges. This is self-directed, self-paced online learning with rich media content, including interactions, quizzes, and virtual labs.
  • Enroll in RS120: Essentials of IBM Rational ClearQuest, Version 7.0. This online course introduces you to using IBM Rational ClearQuest to track change requests. It covers the basic concepts involved in change management and how IBM Rational ClearQuest facilitates creating change records, controlling the change management process, and communicating status to project stakeholders using queries, reports, and charts. This is self-directed, self-paced online learning with rich media content, including interactions, quizzes, and virtual labs.
  • Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
  • Browse the technology bookstore for books on these and other technical topics.

Get products and technologies

Discuss

  • Participate in the discussion forum.
  • Participate in the Rational ClearQuest forum. Get answers, advice, or opinions from other ClearQuest users. in this ClearQuest Community on developerWorks. To participate by e-mail, subscribe with a note to clearquest-subscribe@lists.ca.ibm.com. Subscribers may post by e-mail to clearquest@lists.ca.ibm.com.
  • Join the Test Management forum. Discuss all aspects of managing test assets and Rational TestManager in this forum. Ask questions and share experiences. To participate by e-mail, first subscribe with a note to tm-subscribe@lists.ca.ibm.com. Subscribers may post by e-mail to tm@lists.ca.ibm.com.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=270128
ArticleTitle=Introduction to IBM Rational ClearQuest test management: Part 2. A real-world example of using the standard features
publish-date=11272007