 | Level: Introductory Susan Manupelli (smanupel@us.ibm.com), Advisory Software Engineer,
IBM
27 Nov 2007 This second article of this series walks you through how a testing
team sets up and customizes the test management features, runs the tests, and then
uses the reporting functions. In Part 1 we take the reader through the main features
of IBM® Rational® ClearQuest® test management and provide
guidelines for implementation that take into account the structure and organization
of the test team.
With Part 1 as background information, let's walk through using
IBM® Rational® ClearQuest® test management features for planning
a typical test.
Planning the test
Scenario: XYZ Corp. has eight software projects in the works. They have
set up eight asset registries in ClearQuest test management. The code names for
their projects are names of islands, thus the asset registries are named after
them (see Figure 1). Two of the projects (Aruba and Bermuda) have geographically
distributed teams, so the ClearQuest administrator has set up IBM®
Rational® ClearQuest® MultiSite to enable remote teams to access
ClearQuest.
Figure 1. Asset registries
Standardize configurations and naming conventions
All of the software products under test are supported on three primary platforms:
Microsoft® Windows®, UNIX®, and Linux®. Knowing that the
configurations are shared among all asset registries, the project managers get
together to decide on a set of configurations that will be suitable for all
projects. They use a prefix for these global configurations to indicate the
overall priority of that configuration. However, two of the eight projects require
unique configurations. For this reason, in addition to the global set of
configurations prefixed with P#, several other
configurations have been created. To easily identify which projects those
configurations apply to, the configuration name is prefixed with the first two
letters of the project code name. Figure 2 shows the attributes and values.
Figure 2. Configuration Attributes
Figure 3 shows the configurations.
Figure 3. Configurations
Coordinate use of iteration names
The next task for the project managers is to coordinate on the use of iteration
names. Standardizing a naming convention for this will enable reporting across
multiple projects, particularly those included in the same releases. Of the eight
projects being managed, four of them are in the same software release, while the
rest are standalone products. All projects follow similar processes; therefore,
the managers devise a naming convention that includes the major release version
and minor release version, along with the development and test phase. Because
iterations are specific to an asset registry, they need to create the necessary
iterations in each asset registry.
The first four projects (Aruba, Bermuda, Fiji, and Hawaii) are all part of
Release 4, Version 2. Aruba and Bermuda are still in the construction phases,
undergoing functional verification testing. Fiji and Hawaii are further along and
at the system verification testing stage. Thus, they create the following
iterations for these projects:
-
Aruba: Rel4_V2_C1FVT. Rel4_V2_C2FVT
-
Bermuda: Rel4_V2_C1FVT. Rel4_V2_C2FVT
-
Fiji: Rel4_V2_TSVT1
-
Hawaii: Rel4_V2_TSVT1
As these teams move forward in the release cycle, they can add iterations by
following the same naming convention. By doing this, they will be able to run
reports based either on the test phase or at a higher level across the specific
version. Notice that it is fine to have iterations with the same name in different
asset registries. Behind the scenes, ClearQuest test management automatically
prefixes the asset registry name to the iteration name, so that the XYZ team can
always distinguish one from the other.
The remaining projects are for standalone products that are on separate release
schedules. However, for consistency, the managers follow the same naming
convention. The iterations for these projects are set up as follows:
-
Iceland: Rel8_V1_C1FVT
-
Maui: Rel5_V7_C1FVT
-
Nantucket: Rel3_V6_TSVT1
-
Puerto Rico: Rel3_V2_TSVT1
Decide where and how to store scripts
While the project managers coordinate on configurations and iterations, the
ClearQuest administrator works with the test leads to determine the requirements
for storing test scripts. Most teams use IBM® Rational® Manual Tester
for scripting end-to-end, system-level tests. Those teams need to establish a
local server to house their Rational Manual Tester scripts. In their asset
registry, they establish a file location to point to the server that contains
scripts.
For the Aruba project, the test team is geographically distributed, but
each team is responsible for creation and execution of their own scripts. Scripts
do not need to be shared across locations. In this case, they establish two file
locations in the Aruba asset registry. One is local to Location 1, and the other
is local to Location 2.
The Bermuda project team is also geographically distributed, and they use
IBM® Rational® Functional Tester. However, for this team, one site
develops the Rational Functional Tester scripts, and a remote site is responsible
for running tests. In this case, both sites need access to the same
scripts; therefore, they use IBM® Rational® ClearCase® and store
the Rational Functional Tester scripts in a multi-sited ClearCase versioned object
base (VOB). In their situation, one file location needs to be established to point
to a ClearCase VOB.
Establish the test plan hierarchy
Meanwhile, as the configurations, iterations, and file locations are being
finalized, the test leads begin to put their test plans for the upcoming release
into ClearQuest test management. The test plans for functional testing are
categorized by feature area. Some teams also need to do system-level testing, so
they organize their longer end-to-end tests into a single test plan folder. There
is a Performance team that has test plans for several of the projects, so they may
have users accessing more than one asset registry regularly. Because the
Performance team wants to keep the same test plan hierarchy in all of the asset
registries that they are using, they create a hierarchy in one asset registry, and
then use the Duplicate feature to copy the hierarchy to the other asset
registries.
The hierarchy of test plans for the Fiji asset registry looks like Figure
4.
Figure 4. Test plan hierarchy
Notice the subplans under the Performance test plan:
- Performance
- 01_Optimum Env
- 02_Stress
- 03_Load
There was a certain logical order in which the test lead wanted to have these
listed. ClearQuest test management lists assets alphanumerically. Therefore, when
these test plans were created, they were prefixed with a two-digit numeral so that
they appear listed in a specific order.
Add test cases
After the test plan hierarchy is established, test designers can begin to add
test cases. As they create test cases, if they know that all configured versions
of a certain test case will share exactly the same test script, then they will
develop the test script and associate the test script with the test case. By doing
it that way, the configured test cases will inherit the script association. This
is a big time-saver.
Note:
When the test designers are at this stage, if you are scripting
in Rational Manual Tester or Rational Functional Tester, it makes sense to run
ClearQuest test management from within the Rational Manual Tester or Rational
Functional Tester shell. This way, you can seamlessly move from designing and
scripting to test case development in ClearQuest test management. Of course, you
can do all of the test planning discussed here so far from within the Rational
Manual Tester or Rational Functional Tester shell. However, one of the nicest
features of ClearQuest test management is that if your high-level planning is done
by a project manager or team lead does not normally have the scripting tools
installed, he or she can use the rich client version of ClearQuest test management
for all of the steps leading up to scripting without having to install the
scripting tools. On the flip side, test designers can enjoy the convenience of
having ClearQuest test management run from within the same shell as the test
scripting tools.
Create configured test cases
After the test cases are set up, the team can begin to tag their test cases with
configurations to create configured test cases (CTCs). If they have multiple test
cases that will be run on the same platforms, they can select multiple
configurations simultaneously and apply the configurations to several of them in
batch mode. This is also a big time-saver.
Figure 5 shows test cases belonging to a scenario subplan that has been
configured for two specific platforms. Notice that this screen capture shows a
view of ClearQuest test management from within Rational Manual Tester.
Figure 5. Test case hierarchy
The team's final step in test planning involves
"tagging" their configured test cases with the iteration for
which they intend to run them. This is another step that they can do easily in
batch mode by selecting several CTCs at once and associating them with one or more
iterations. After this is done, they can create queries and reports to show the
configured test cases that they have planned. (This will be covered in the
sections that follow.)
Creating queries to show planned tests
When they have finished your planning, the team wants to see all of the CTCs that
they have planned for the iteration. They can create a query that displays
important information about the CTC:
- The record type that they need to query against is
TMConfiguredTestCase.
- At a minimum, they will filter based on the iteration, but they can choose
from a variety of important fields to display in the query.
Figure 6 shows a sample query that returns planned CTCs for the iteration but
displays the parent test case, as well as the parent test plan. These are the
display fields used:
TestCase.ParentPlan.PlarentPlan.Headline
TestCase.ParentPlan.Headline
TestCase.Headline
ID
Headline
Configuration
Owner
TestType
Script
|
The first three field names plus the Headline field are set to sort in ascending
order, in that order (1 through 4). The rest are set as No Sort.
Figure 6. Query to show planned
tests
Running this query yields the results shown in Figure 7.
Figure 7. Query result
You can tell from this query that, of the 21 planned tests, 9 of them are not yet
associated with test scripts. This is an indication that someone has additional
planning to do.
Tip:
Another ClearQuest feature is the ability to modify records from
the Query Results view in batch mode. Based on the screen capture in Figure 7, you
can see that, currently, all configured test cases are assigned to Admin.
In batch mode, you can now select multiple rows in the query results and modify
the CTCs to assign them to their rightful owners. This is a very useful feature if
resources are shuffled and test cases need to be reassigned to new owners
midstream.
Running the test
When the team has finished planning, tagging tests with their proper iterations,
and assigning them to the appropriate owners, it is time to begin testing. The
administrator can modify the query used in the previous section to show the
planned test cases for the current user. Then each tester can import and run the
query to show their own assigned tests for the iteration. The query filters on the
iteration under test, and the owner is set to
[current user]. This way, the user who is currently
logged in can run the query and see what planned tests are assigned.
Let's say that Jane is working on the system tests for Microsoft Windows
on the Rel4_V2_TSVT1 iteration for the Fiji project. Her query will return
what you see in Figure 8.
Figure 8. Query results by
owner
Now Jane can easily see a complete list of what she is responsible for running.
Notice that she is viewing all of this ClearQuest data from within Rational Manual
Tester. She can execute directly from this query, or she can return to the tree
view and run her tests from there. She decides to execute from her My Planned
Tests - ClearQuest Query Results view.
- She highlights a row and selects Execute from the drop-down menu.
- Immediately, the Execute Test dialog displays.
Because this CTC is associated with more than one iteration, Jane needs to pick
the correct iteration to associate with the test log. She also needs to identify
the build number.
- If the build has been created before, she can just select it from the list.
Otherwise, she can create it at run time.
Build records also need to follow some kind of naming convention to be
consistent. Jane doesn't want to identify a build with a particular
iteration or test phase, because there are other teams using the same build
stream, but they are running different iterations because of their testing
schedules.
- Therefore, she names the build based on build number and prefixes it with the
major release and version identifiers:
Rel4_V2_B2353
(see Figure 9).
|
Differences between test and script logs
It is important to understand the difference between the test log record
and the script log.
- The test log record is a record in the database that retains info
about the execution, including the iteration, the final verdict, the build,
and any associated defects.
- The script log is generated by the scripting tool and includes
specifics about which verification points passed and failed. The script log is
saved in the file system where the scripts are stored.
The script log is not saved into ClearQuest test management. However, the
test log that is saved in ClearQuest test management retains a link to
the script log. If Jane ever needs to view the defect or the log, she can easily
navigate to it from the original configured test case. From there, she can
select the Execution tab. This tab will show all of the test logs that
were committed to the database, along with the verdict (see Figure 10).
|
|
Figure 9. Specifying iteration
and build
- Jane clicks Finish in the Execute test window, and the Rational
Manual Tester script displays.
- She records her verification points and finishes the test.
The result displays in the Test Results view. At this point, Jane needs to decide
if the result is valid and if this is the verdict that she wants to commit to the
database. In this case, the test has failed. Jane decides she does want to commit
the test log with the failed verdict, and she also wants to log a defect.
- From the Test Results view, she uses the drop-down menu to click
Submit Defect.
This launches the New Defect window. After the new defect is saved, it
also commits the test log record. The test log record will be associated with the
newly created defect.
Figure 10. Configured test
case view
- From this view, she can select one of the logs to open.
- In the test log, she can then navigate directly to the log file for the script
and view any associated defects from the test log.
Tip:
You can also run test scripts directly from the test tools, and
then import the results into ClearQuest test management. Using that method enables
you to create and commit a test log record by importing a script log. This is
another way of integrating ClearQuest test management with a test team that may be
executing remotely.
All of the testers in the team go through this process of executing tests, and
then committing verdicts and test log records to the database for the given
iteration.
Useful reporting with ClearQuest test
management
Now that the XYZ Corp. team has begun to run tests, they want to be able to
report on progress. The test lead wants to see a list of all CTCs that have been
executed, along with their results. ClearQuest test management includes a query in
the TM Queries folder, under the Public Folders area, that they can use as a
starting point. It is called "Configured Test Cases -- verdict
history," and the query looks what Figure 11 shows.
Figure 11. Verdict history
query
If the team runs this query with no modifications, it will enable the test lead
to choose one or more asset registries and see verdict histories for all
tests that were ever executed in the selected asset registries. Figure 12 shows
the results of this default query for the Fiji asset registry.
Figure 12. Verdict history
query results
Notice that the display shows the verdict history across more than one iteration.
Although this query is useful, what the test lead really wants to see is the
verdict history for the iteration currently under test. To do this, the test lead
modifies the query in two ways:
- Changes the filter to always query the Fiji asset registry, specifically
- Changes the filter to add a check for the iteration that was assigned to the
test log (TestLogs.Iteration)
The new query looks like the one in Figure 13.
Figure 13. Modified Verdict
History query
The new query returns the results shown in Figure 14.
Figure 14. Modified Verdict
History query results
Now the test lead can see that only two scenarios have been run against the
Rel4_V2_TSVT1 iteration for the Fiji project, so far. Given that the query
includes a filter for TestLogs.Latest=True, it will
return only the latest verdict that was committed for each test for the given
iteration.
Resources Learn
- Find out more about the test management
capabilities in IBM Rational ClearQuest:
Run and analyze software tests with Rational ClearQuest Test Management,
an IBM® developerWorks® tutorial by Brian Bryson (October 2006).
-
FAQ: Enterprise Test Management with IBM Rational ClearQuest, Version 7
-
Demo: Enterprise test management with IBM Rational ClearQuest
- Enroll in
RT210: Essentials of Test Management with IBM Rational ClearQuest V7.0.
Build test plans, test cases, and test suites that contribute to a successful test
effort. The robust reporting capabilities of IBM Rational ClearQuest allow you to
easily view the status of your testing and the quality of you product. These
capabilities are available in the IBM Rational Software Development Platform via a
ClearQuest plugin (test management package). Students can also complete lab
exercises to reinforce critical concepts and tool functionality. The focus is on
the practical application of IBM Rational ClearQuest to resolve common test
management challenges. This is self-directed, self-paced online learning with rich
media content, including interactions, quizzes, and virtual labs.
- Enroll in
RS120: Essentials of IBM Rational ClearQuest, Version 7.0.
This online course introduces you to using IBM Rational ClearQuest to track change
requests. It covers the basic concepts involved in change management and how IBM
Rational ClearQuest facilitates creating change records, controlling the change
management process, and communicating status to project stakeholders using
queries, reports, and charts. This is self-directed, self-paced online learning
with rich media content, including interactions, quizzes, and virtual labs.
- Visit the
Rational software area on developerWorks
for technical resources and best practices for Rational Software Delivery Platform
products.
- Browse the
technology bookstore
for books on these and other technical topics.
Get products and technologies
Discuss
- Participate in the discussion forum.
- Participate in the
Rational
ClearQuest forum.
Get answers, advice, or opinions from other ClearQuest users. in this ClearQuest
Community on developerWorks. To participate by e-mail, subscribe with a note to
clearquest-subscribe@lists.ca.ibm.com. Subscribers may post by e-mail to
clearquest@lists.ca.ibm.com.
- Join the
Test
Management forum.
Discuss all aspects of managing test assets and Rational TestManager in this
forum. Ask questions and share experiences. To participate by e-mail, first
subscribe with a note to tm-subscribe@lists.ca.ibm.com. Subscribers may post by
e-mail to tm@lists.ca.ibm.com.
About the author | | | Susan Manupelli is an Advisory Software Engineer and Quality Engineering lead working at the IBM Rational office in Lexington, Massachusetts. She has fifteen years of industry experience in software testing and test management. While at IBM she has worked predominantly with the Automated Software Quality products including Rational Robot, Rational TestManager, Rational Functional Tester and ClearQuest test management. She was part of the team that helped deploy ClearQuest test management across Rational and has earned three "Bravo!" awards for her contributions to IBM. Recently she has joined the Rational Build Forge team. |
Rate this page
| |
