With Part 1 as background information, let's walk through using IBM® Rational® ClearQuest® test management features for planning a typical test.
Planning the test
Scenario: XYZ Corp. has eight software projects in the works. They have set up eight asset registries in ClearQuest test management. The code names for their projects are names of islands, thus the asset registries are named after them (see Figure 1). Two of the projects (Aruba and Bermuda) have geographically distributed teams, so the ClearQuest administrator has set up IBM® Rational® ClearQuest® MultiSite to enable remote teams to access ClearQuest.
Figure 1. Asset registries
Standardize configurations and naming conventions
All of the software products under test are supported on three primary platforms:
Microsoft® Windows®, UNIX®, and Linux®. Knowing that the
configurations are shared among all asset registries, the project managers get
together to decide on a set of configurations that will be suitable for all
projects. They use a prefix for these global configurations to indicate the
overall priority of that configuration. However, two of the eight projects require
unique configurations. For this reason, in addition to the global set of
configurations prefixed with
P#, several other
configurations have been created. To easily identify which projects those
configurations apply to, the configuration name is prefixed with the first two
letters of the project code name. Figure 2 shows the attributes and values.
Figure 2. Configuration Attributes
Figure 3 shows the configurations.
Figure 3. Configurations
Coordinate use of iteration names
The next task for the project managers is to coordinate on the use of iteration names. Standardizing a naming convention for this will enable reporting across multiple projects, particularly those included in the same releases. Of the eight projects being managed, four of them are in the same software release, while the rest are standalone products. All projects follow similar processes; therefore, the managers devise a naming convention that includes the major release version and minor release version, along with the development and test phase. Because iterations are specific to an asset registry, they need to create the necessary iterations in each asset registry.
The first four projects (Aruba, Bermuda, Fiji, and Hawaii) are all part of Release 4, Version 2. Aruba and Bermuda are still in the construction phases, undergoing functional verification testing. Fiji and Hawaii are further along and at the system verification testing stage. Thus, they create the following iterations for these projects:
- Aruba: Rel4_V2_C1FVT. Rel4_V2_C2FVT
- Bermuda: Rel4_V2_C1FVT. Rel4_V2_C2FVT
- Fiji: Rel4_V2_TSVT1
- Hawaii: Rel4_V2_TSVT1
As these teams move forward in the release cycle, they can add iterations by following the same naming convention. By doing this, they will be able to run reports based either on the test phase or at a higher level across the specific version. Notice that it is fine to have iterations with the same name in different asset registries. Behind the scenes, ClearQuest test management automatically prefixes the asset registry name to the iteration name, so that the XYZ team can always distinguish one from the other.
The remaining projects are for standalone products that are on separate release schedules. However, for consistency, the managers follow the same naming convention. The iterations for these projects are set up as follows:
- Iceland: Rel8_V1_C1FVT
- Maui: Rel5_V7_C1FVT
- Nantucket: Rel3_V6_TSVT1
- Puerto Rico: Rel3_V2_TSVT1
Decide where and how to store scripts
While the project managers coordinate on configurations and iterations, the ClearQuest administrator works with the test leads to determine the requirements for storing test scripts. Most teams use IBM® Rational® Manual Tester for scripting end-to-end, system-level tests. Those teams need to establish a local server to house their Rational Manual Tester scripts. In their asset registry, they establish a file location to point to the server that contains scripts.
For the Aruba project, the test team is geographically distributed, but each team is responsible for creation and execution of their own scripts. Scripts do not need to be shared across locations. In this case, they establish two file locations in the Aruba asset registry. One is local to Location 1, and the other is local to Location 2.
The Bermuda project team is also geographically distributed, and they use IBM® Rational® Functional Tester. However, for this team, one site develops the Rational Functional Tester scripts, and a remote site is responsible for running tests. In this case, both sites need access to the same scripts; therefore, they use IBM® Rational® ClearCase® and store the Rational Functional Tester scripts in a multi-sited ClearCase versioned object base (VOB). In their situation, one file location needs to be established to point to a ClearCase VOB.
Establish the test plan hierarchy
Meanwhile, as the configurations, iterations, and file locations are being finalized, the test leads begin to put their test plans for the upcoming release into ClearQuest test management. The test plans for functional testing are categorized by feature area. Some teams also need to do system-level testing, so they organize their longer end-to-end tests into a single test plan folder. There is a Performance team that has test plans for several of the projects, so they may have users accessing more than one asset registry regularly. Because the Performance team wants to keep the same test plan hierarchy in all of the asset registries that they are using, they create a hierarchy in one asset registry, and then use the Duplicate feature to copy the hierarchy to the other asset registries.
The hierarchy of test plans for the Fiji asset registry looks like Figure 4.
Figure 4. Test plan hierarchy
Notice the subplans under the Performance test plan:
- 01_Optimum Env
There was a certain logical order in which the test lead wanted to have these listed. ClearQuest test management lists assets alphanumerically. Therefore, when these test plans were created, they were prefixed with a two-digit numeral so that they appear listed in a specific order.
Add test cases
After the test plan hierarchy is established, test designers can begin to add test cases. As they create test cases, if they know that all configured versions of a certain test case will share exactly the same test script, then they will develop the test script and associate the test script with the test case. By doing it that way, the configured test cases will inherit the script association. This is a big time-saver.
When the test designers are at this stage, if you are scripting in Rational Manual Tester or Rational Functional Tester, it makes sense to run ClearQuest test management from within the Rational Manual Tester or Rational Functional Tester shell. This way, you can seamlessly move from designing and scripting to test case development in ClearQuest test management. Of course, you can do all of the test planning discussed here so far from within the Rational Manual Tester or Rational Functional Tester shell. However, one of the nicest features of ClearQuest test management is that if your high-level planning is done by a project manager or team lead does not normally have the scripting tools installed, he or she can use the rich client version of ClearQuest test management for all of the steps leading up to scripting without having to install the scripting tools. On the flip side, test designers can enjoy the convenience of having ClearQuest test management run from within the same shell as the test scripting tools.
Create configured test cases
After the test cases are set up, the team can begin to tag their test cases with configurations to create configured test cases (CTCs). If they have multiple test cases that will be run on the same platforms, they can select multiple configurations simultaneously and apply the configurations to several of them in batch mode. This is also a big time-saver.
Figure 5 shows test cases belonging to a scenario subplan that has been configured for two specific platforms. Notice that this screen capture shows a view of ClearQuest test management from within Rational Manual Tester.
Figure 5. Test case hierarchy
The team's final step in test planning involves "tagging" their configured test cases with the iteration for which they intend to run them. This is another step that they can do easily in batch mode by selecting several CTCs at once and associating them with one or more iterations. After this is done, they can create queries and reports to show the configured test cases that they have planned. (This will be covered in the sections that follow.)
Creating queries to show planned tests
When they have finished your planning, the team wants to see all of the CTCs that they have planned for the iteration. They can create a query that displays important information about the CTC:
- The record type that they need to query against is TMConfiguredTestCase.
- At a minimum, they will filter based on the iteration, but they can choose from a variety of important fields to display in the query.
Figure 6 shows a sample query that returns planned CTCs for the iteration but displays the parent test case, as well as the parent test plan. These are the display fields used:
TestCase.ParentPlan.PlarentPlan.Headline TestCase.ParentPlan.Headline TestCase.Headline ID Headline Configuration Owner TestType Script
The first three field names plus the Headline field are set to sort in ascending order, in that order (1 through 4). The rest are set as No Sort.
Figure 6. Query to show planned tests
Running this query yields the results shown in Figure 7.
Figure 7. Query result
You can tell from this query that, of the 21 planned tests, 9 of them are not yet associated with test scripts. This is an indication that someone has additional planning to do.
Another ClearQuest feature is the ability to modify records from the Query Results view in batch mode. Based on the screen capture in Figure 7, you can see that, currently, all configured test cases are assigned to Admin. In batch mode, you can now select multiple rows in the query results and modify the CTCs to assign them to their rightful owners. This is a very useful feature if resources are shuffled and test cases need to be reassigned to new owners midstream.
Running the test
When the team has finished planning, tagging tests with their proper iterations,
and assigning them to the appropriate owners, it is time to begin testing. The
administrator can modify the query used in the previous section to show the
planned test cases for the current user. Then each tester can import and run the
query to show their own assigned tests for the iteration. The query filters on the
iteration under test, and the owner is set to
[current user]. This way, the user who is currently
logged in can run the query and see what planned tests are assigned.
Let's say that Jane is working on the system tests for Microsoft Windows on the Rel4_V2_TSVT1 iteration for the Fiji project. Her query will return what you see in Figure 8.
Figure 8. Query results by owner
Now Jane can easily see a complete list of what she is responsible for running. Notice that she is viewing all of this ClearQuest data from within Rational Manual Tester. She can execute directly from this query, or she can return to the tree view and run her tests from there. She decides to execute from her My Planned Tests - ClearQuest Query Results view.
- She highlights a row and selects Execute from the drop-down menu.
- Immediately, the Execute Test dialog displays.
Because this CTC is associated with more than one iteration, Jane needs to pick the correct iteration to associate with the test log. She also needs to identify the build number.
- If the build has been created before, she can just select it from the list. Otherwise, she can create it at run time.
Build records also need to follow some kind of naming convention to be consistent. Jane doesn't want to identify a build with a particular iteration or test phase, because there are other teams using the same build stream, but they are running different iterations because of their testing schedules.
- Therefore, she names the build based on build number and prefixes it with the
major release and version identifiers:
Rel4_V2_B2353(see Figure 9).
Figure 9. Specifying iteration and build
- Jane clicks Finish in the Execute test window, and the Rational Manual Tester script displays.
- She records her verification points and finishes the test.
The result displays in the Test Results view. At this point, Jane needs to decide if the result is valid and if this is the verdict that she wants to commit to the database. In this case, the test has failed. Jane decides she does want to commit the test log with the failed verdict, and she also wants to log a defect.
- From the Test Results view, she uses the drop-down menu to click Submit Defect.
This launches the New Defect window. After the new defect is saved, it also commits the test log record. The test log record will be associated with the newly created defect.
Figure 10. Configured test case view
- From this view, she can select one of the logs to open.
- In the test log, she can then navigate directly to the log file for the script and view any associated defects from the test log.
You can also run test scripts directly from the test tools, and then import the results into ClearQuest test management. Using that method enables you to create and commit a test log record by importing a script log. This is another way of integrating ClearQuest test management with a test team that may be executing remotely.
All of the testers in the team go through this process of executing tests, and then committing verdicts and test log records to the database for the given iteration.
Useful reporting with ClearQuest test management
Now that the XYZ Corp. team has begun to run tests, they want to be able to report on progress. The test lead wants to see a list of all CTCs that have been executed, along with their results. ClearQuest test management includes a query in the TM Queries folder, under the Public Folders area, that they can use as a starting point. It is called "Configured Test Cases -- verdict history," and the query looks what Figure 11 shows.
Figure 11. Verdict history query
If the team runs this query with no modifications, it will enable the test lead to choose one or more asset registries and see verdict histories for all tests that were ever executed in the selected asset registries. Figure 12 shows the results of this default query for the Fiji asset registry.
Figure 12. Verdict history query results
Notice that the display shows the verdict history across more than one iteration. Although this query is useful, what the test lead really wants to see is the verdict history for the iteration currently under test. To do this, the test lead modifies the query in two ways:
- Changes the filter to always query the Fiji asset registry, specifically
- Changes the filter to add a check for the iteration that was assigned to the test log (TestLogs.Iteration)
The new query looks like the one in Figure 13.
Figure 13. Modified Verdict History query
The new query returns the results shown in Figure 14.
Figure 14. Modified Verdict History query results
Now the test lead can see that only two scenarios have been run against the
Rel4_V2_TSVT1 iteration for the Fiji project, so far. Given that the query
includes a filter for
TestLogs.Latest=True, it will
return only the latest verdict that was committed for each test for the given
- Find out more about the test management capabilities in IBM Rational ClearQuest: Run and analyze software tests with Rational ClearQuest Test Management, an IBM® developerWorks® tutorial by Brian Bryson (October 2006).
- FAQ: Enterprise Test Management with IBM Rational ClearQuest, Version 7
- Enroll in RT210: Essentials of Test Management with IBM Rational ClearQuest V7.0. Build test plans, test cases, and test suites that contribute to a successful test effort. The robust reporting capabilities of IBM Rational ClearQuest allow you to easily view the status of your testing and the quality of you product. These capabilities are available in the IBM Rational Software Development Platform via a ClearQuest plugin (test management package). Students can also complete lab exercises to reinforce critical concepts and tool functionality. The focus is on the practical application of IBM Rational ClearQuest to resolve common test management challenges. This is self-directed, self-paced online learning with rich media content, including interactions, quizzes, and virtual labs.
- Enroll in RS120: Essentials of IBM Rational ClearQuest, Version 7.0. This online course introduces you to using IBM Rational ClearQuest to track change requests. It covers the basic concepts involved in change management and how IBM Rational ClearQuest facilitates creating change records, controlling the change management process, and communicating status to project stakeholders using queries, reports, and charts. This is self-directed, self-paced online learning with rich media content, including interactions, quizzes, and virtual labs.
- Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
- Browse the technology bookstore for books on these and other technical topics.
Get products and technologies
- ClearQuest users and administrators can find more resources in the ClearQuest section of the developerWorks Rational zone, including hooks, Eclipse plug-ins, product documentation, articles and whitepapers.
- Download trial versions of IBM Rational software.
- Participate in the discussion forum.
- Participate in the Rational ClearQuest forum. Get answers, advice, or opinions from other ClearQuest users. in this ClearQuest Community on developerWorks. To participate by e-mail, subscribe with a note to firstname.lastname@example.org. Subscribers may post by e-mail to email@example.com.
- Join the Test Management forum. Discuss all aspects of managing test assets and Rational TestManager in this forum. Ask questions and share experiences. To participate by e-mail, first subscribe with a note to firstname.lastname@example.org. Subscribers may post by e-mail to email@example.com.