IBM® Rational® ClearQuest® offers a test management feature that gives you tools to plan, organize, and define a particular set of tests, including varying configurations that need to be run for a given test cycle. This enables you to closely track testing progress so that you can manage resources and schedules. This feature also provides a mechanism for measuring quality according to metrics reported across several key areas.
If you set up ClearQuest with the Enterprise schema, ClearQuest test management is included. If you set up ClearQuest with any other schema or a custom schema, you can apply the ClearQuest test management package to use the features described in this article.
In this first article we take the reader through the main features of ClearQuest test management and provide guidelines for implementation that take into account the structure and organization of the test team.
In the second article we take the reader through a step-by-step implementation of ClearQuest test management in a real-world environment. Note that this article will not cover product installation, but instead will cover the creation and organization of test assets after ClearQuest test management has been installed. It brings the reader through a full test cycle from creation of the Asset Registry to reviewing Test Logs.
Before deploying ClearQuest test management, think about answers to these questions:
- What is the size of the organization that will be using ClearQuest test management?
- Is the testing organization geographically dispersed? If so, how are the testing responsibilities split or shared among remote sites?
- How many projects or products will you manage?
- What scripting tools (if any) will you use?
- What are the requirements for testing across configurations?
- What is the duration (calendar days) and scope (number of tests) in a typical test cycle?
- What metrics are necessary for status tracking?
As you read through this article, choices that you need to make in terms of how you organize your ClearQuest test management deployment will depend on the answers to these questions.
ClearQuest client software options
ClearQuest offers several different client software options. Not all ClearQuest test management features are available on all clients; therefore, the role of the user determines which client is the best choice. These are the ClearQuest clients that are available, along with notes on usefulness with ClearQuest test management:
The Eclipse-based clients show a hierarchical tree view of the test assets, called the TestManager view. Planning and visualizing the relationship of test cases to test plans is difficult without this view. For this reason, most ClearQuest test management users should choose one of the Eclipse-based clients.
- ClearQuest Eclipse rich client platform (RCP): As all of the Eclipse-based clients do, this contains the TestManager view. You can create test logs from the RCP client, but with the ClearQuest 7.0.1 release and later, you cannot run any of the test scripting tools except IBM® Rational® Robot. This client is a good choice for test managers, who may not need to run scripts, or for test teams who use ClearQuest test management for recording results but do not use any of the scripting tools.
- ClearQuest Eclipse Rational Manual Tester shell: From within Rational Manual Tester, you can connect to a ClearQuest database. If your role requires you to develop test scripts in Rational Manual Tester or to execute Rational Manual Tester scripts, this is the client to use
- ClearQuest Eclipse SDP shell: From within the Rational Software Delivery Platform (SDP), you can connect to a ClearQuest database. Rational Functional Tester and Rational Performance Tester run on the SDP. If you plan to develop or execute using either of these scripting tools, you will need to utilize the SDP platform.
- ClearQuest Web: If your site plans to make the ClearQuest database available via the web by installing the ClearQuest Web service, you will be able to view ClearQuest test management data via the ClearQuest Web client. However, the TestManager view is not available from the web client and you cannot execute scripts from the web client. It is adequate for viewing query results and for recording test results when you aren't using one of Rational's scripting tools.
- ClearQuest native Windows client: As with the ClearQuest Web client, the TestManager view is not available. adequate for viewing query results and for recording test results when you aren't using one of Rational's scripting tools.
Organizing ClearQuest test management assets
When deciding how to organize your test assets (and, more specifically, the collection of test plans and test cases), start by thinking about ways to categorize and group them. You can envision many different ways to group test assets -- one may be by product or project, another may be by tester, another by the type of testing, for instance. There really is no limit to the range of possibilities. Thus, the question becomes what will enable you to make the most out of the features available in ClearQuest test management and to ensure that the tests are managed as efficiently as possible? The sections that follow describe a model that works well, along with alerts about common pitfalls to avoid as you structure the test hierarchy.
Understanding the asset registry
The asset registry is the largest container for grouping test assets in ClearQuest test management. The proven model that works well, based on internal use of ClearQuest test management by IBM test teams is to treat the Asset Registry as the container for all test assets related to a specific project or product. For a particular project or product, several test artifacts and resources are shared, including these:
- A specific set of test plans and test cases
- A specific set of features to cover
- A common release or testing schedule, or both
- A common pool of testers
- A specific set of supported platforms
There are specific features in ClearQuest test management for tracking each of these artifacts or resources. In the sections that follow, you will see how to use existing types of record to cover these common areas.
One common pitfall to avoid is creating separate asset registries for different releases of the same product. When you do this, you immediately set up a hierarchy that requires you to duplicate test assets from one asset registry into another, separate asset registry as you move from one release to another release of the same product. You wind up with a ClearQuest test management environment where the test assets grow exponentially with each new release. You end up not being able to take advantage of tracking execution history of a specific test case over several releases, because the execution log results are not copied when a test case is copied. And finally, you end up losing the efficiencies that ClearQuest test management offers for reusing test assets from one release to another.
If the test hierarchy that you are designing requires you to copy test assets as a normal course of moving from one release to another or one version to another, this should be a red flag that you probably are not using ClearQuest test management as efficiently as possible.
Another pitfall to avoid is creating a separate asset registry for every user. Unless users are operating so independently that they are working on separate projects with separate release schedules, a different asset registry for each user is not advisable. Standard ClearQuest test management forms allow you to assign an owner to each type of test asset. This is a better method for tracking which tests belong to whom, as well as overall testing progress by owner.
Using iterations effectively
An iteration in ClearQuest test management allows you to identify a specific testing cycle, along with start and end dates. When iterations are managed and used properly, you get some of the greatest benefits of ClearQuest test management. Iterations enable you to track progress for multiple testing phases of the same product that may be running along parallel development and testing cycles. Iterations also enable you to make the most of test case reuse.
The purpose of the iteration record is to allow you to "tag" other test assets in a way that associates the asset with a specific test cycle. Test plans, test cases, and configured test cases can all be tagged with one or more iterations. Then you can generate reports that show you which test assets were (or will be) used for a given test cycle. Iterations created under a given asset registry apply only to the assets within the same registry.
You can make using iterations even more powerful if you use a structured naming convention. A meaningful naming convention that identifies not only the specific test cycle but also the minor and major release that the test cycle belongs to will help with metrics reporting across not only the specific test cycle, but across larger releases. Let's follow an example of how to develop a naming convention:
- Start by determining the smallest unit of a test cycle that you would bother to track. Tracking testing progress with an iteration in ClearQuest test management requires you to create an iteration, and then to select some number of configured test cases to associate with the iteration. For this reason, iteration durations are likely to last at least a few days.
If there is a test run or a test cycle that would identify a specific test to be completed in shorter durations or more frequent test phases, consider using test suites to track the effort.
- Name the units in consistent syntax that helps you track them. Let's say that, during the Construction phase of development, you expect to run two test cycles that consist of mostly functional testing. Then, during the Transition phase as you prepare for release, you expect to run two more system-level testing cycles and one final regression cycle. You might come up with labels something like these for each phase:
- Construction: Two functional verification test cycles, labeled as
- Transition: Two system verification test cycles and one final regression cycle, labeled as
- Construction: Two functional verification test cycles, labeled as
This convention, alone, might be sufficient if your product or project is a one-time custom development effort. However, if you expect the product that you're testing to have new major releases or minor maintenance releases, you need to reuse as many test assets as possible.
- To facilitate reuse, prefix your iteration cycle with the something that identifies the major and minor releases to which the test cycle belongs.
As a result, iteration names would resemble this one, which indicates major Release 4, Version 2, Transition System Verification Test cycle 2:
This way, you have one test case that can be tagged and, therefore, tracked for more than one release, even if the releases are running in parallel. It also enables you to open up a configured test case record and view test run result history for test cases across several versions and releases.
Let's say that you have more than one asset registry to track more than one product, and both of these products are participating in a specific major release. Using a naming convention that includes the major release allows you to query across multiple assets registries to view test results for a given major release.
Establishing a naming conventions for iterations is crucial to enabling powerful and effective metrics reporting. It is also the key to facilitating reuse of test assets, which makes using ClearQuest test management more efficient.
Iterations are specific to the asset registry in which they are created. You can have an iteration belonging to one asset registry with the same name as an iteration belonging to a different asset registry. When browsing iteration names for reporting, ClearQuest automatically adds the asset registry name to the beginning of the iteration name to distinguish one from another.
Establishing file locations
File locations are records that hold paths to external files, including test scripts. File Locations point to network shares, and they can include IBM® Rational® ClearCase® versioned object bases (VOBs). There are two types of external files that ClearQuest test management uses:
- Test motivators
- Test scripts
Before setting up file locations, consider these factors, because the answers to these questions have implications for how you establish your file locations.
- What scripting tools are being used?
- Is the environment geographically distributed or not?
- Is versioning of scripts a requirement?
Test motivators are documents that may relate to the test effort, such as a project schedule, a requirements document, or a project plan. If you want to link Test Plan Records, or other test assets with these 'project drivers' then you would establish a file location for your test motivators. This is really an optional feature and ClearQuest test management can be deployed successfully without establishing file locations for test motivators.
ClearQuest test management supports direct execution of test scripts generated from IBM® Rational® Manual Tester, Rational Functional Tester, Rational Performance Tester, and Eclipse TPTP tests. If you plan to use any of these script formats, you will need to establish file locations.
If your environment is not geographically distributed, you can use a local network shared drive or location to store your scripts. If your environment is geographically distributed, but the division of responsibility is such that individual sites develop and execute their own scripts for their own areas, then you can get away with having a separate file location for each site. You could name them to indicate the location. For example, within a given asset registry, you could have a file location named Lexington RFT Scripts and a file location named Bangalore RFT Scripts. As long as people in Lexington do not need to access or run scripts in Bangalore, this will work fine.
If your environment is geographically distributed, and remote sites need to access the same set of scripts or motivator files, then you need to have these files stored in a multi-sited ClearCase VOB. The file location is then established to point to a ClearCase VOB, and that single file location is used by all sites accessing that particular asset registry.
If versioning of scripts is a requirement, then you must store your test scripts in a ClearCase VOB. ClearQuest test management maintains a record of the view profile that you were using when you executed a script, and this gets tied to the iteration. If the script changes from one iteration to the next, you can open the specific version of the script that was executed for a particular iteration. In this way, a history of exactly what was executed throughout the history of that particular test case is maintained. If auditing is a major concern, then consider using Rational ClearCase in conjunction with ClearQuest test management.
File locations are specific to the asset registry for which they belong. This is another reason why it makes sense to build your asset registries based on projects, because you would expect the script files for a given project to be located on a shared resource. Otherwise, you would have to establish the same file location in every single asset registry, rather than specifying it once.
The configuration record allows you to identify a collection of variables that make up a unique instance of a given test environment. Before creating configuration records, you must first establish a set of environmental attributes and define the possible values for those attributes. For example, if you know that you need to test on operating systems of varying languages, you might define a Language attribute. Then you might create values such as English, German, Simplified Chinese, and so forth. If testing across different OS types is important, you might create an OS attribute, and then create values such as Windows, Linux, and UNIX. Specific versions of these operating systems could be built into the existing values, or you could define yet another attribute called OS Version to capture more specific variations of each OS. After you capture the possible permutations using attributes and values, you can select combinations of these and create a configuration record.
As you can tell, you can use the configuration record to capture very basic system attributes, such as OS type and version or memory. You can also use it to define significantly more complex environments that may include client browser type and version, server system, Web server, database type, and database version, as well as integrations with other applications, and so on. It is really important to give a lot of thought to how you establish your configurations. If you have too many attributes and you define them too granularly, then you wind up with such a large number of configurations that maintenance of the list becomes unwieldy. You also diminish the usefulness of categorizing tests with configurations if every single test case in your asset registry is assigned to a different configuration.
If the application under test runs only under a specific controlled environment, you do not need to establish different configurations. Instead, you can use ClearQuest test management with a single default configuration.
One other important thing to note about configurations is that they are shared across all asset registries. If you establish your asset registries based on product or project, and these projects share a similar set of configurations, then having one common set that can be used globally is a very convenient feature of ClearQuest test management. However, if the configurations required to support different asset registries vary greatly, then you need to come up with a naming convention so that you can distinguish which configurations apply to which asset registries. You could prefix the configuration names with abbreviations for the asset registry to which they belong.
Organizing test plans and test cases
Test plan records enable you to categorize and group related test cases. Test plans can contain test cases or other subplans. Assuming that your asset registries are named for products or projects, a good model to follow for categorizing test plans would be to name them for product components or feature areas. If portions of your testing span several feature areas, another hierarchy suggestion is to name test plans for the type of testing being performed (for example, System Test, Performance Testing, Install Testing, and so on). These models can be used together, with some plans named for feature areas and others named for test types. This really depends on the organization. As with the asset registry, avoid creating test plans that are named for or specific to a particular release.
As mentioned previously, test plans can contain test cases or other test plans. There are no specific guidelines for how to arrange test plans. If you can effectively group related tests into subplans, then do so. However, avoid excessive nesting, because that can make reporting more difficult. It also can make browsing in the tree view more cumbersome.
The latest version of ClearQuest test management sorts items in the Asset Browser alphanumerically. This enables you to organize test plans, subplans, and test cases in a logical order by inserting a numeric prefix in the name. As an example, you can then create a hierarchy of plans, such as this one:
- 01_Install Plan
- 02_Security Plan
- 02_01 Login Child Plan 1
- 02_02 Users Group Priv Plan 2 (and so forth)
- 03_Editor Plan (and so on)
A hierarchy such as this also enables you to report the status of test cases belonging to a specific plan, as well as test cases belonging to that plan's child plans.
If ordering of test plans and test cases is not important and using the numbering system seems too cumbersome, another method for reporting across all test cases belonging to a plan and it's child plans is to use one of the custom fields under the Legacy tab to specify a keyword that can be used to query test cases. For example, you could tag all configured test cases under the Security Plan, as well as those under the Security Plan's child plans, with the label
SP in the custom field of the configured test case record (under the Legacy tab).
Test cases and configured test cases
Test case records are where you identify information pertaining to a specific test. The information about the test case that is most important are the headline, which is what will show in the file directory (tree view), and the description, which will identify the purpose of the test case.
Test cases are children of test plans. Each test case record must belong to a single parent test plan record. Test cases are the parents of configured test cases. Configured test cases are created when you relate a specific configuration to a specific test case. You can do this by selecting a test case in the tree view and choosing the Add configuration option. You can also open a test case record and associate configurations with it, if you'd rather do it that way. When you do this, the tree view is immediately populated with specific configured versions of the test cases.
If you are using test scripts, and you expect that the same script can be used for each configured version of a test case, then it is best to add the script to the parent test case before adding the configurations. That way, the configured test case will inherit the script identified in the parent test case. This saves a lot of time when building up your collection of test cases. If you require different scripts for each configured test case belonging to the same parent, then the scripts can be associated after you create the configured test cases.
Reorganizing test assets
As your test cycle evolves, you may need to move test cases from one test plan to another. To do this, you can simply reassign the parent test cases:
- Select several test cases simultaneously.
- Open one, and change the parent plan.
- Then click Apply all.
This will change parents of test cases to the new plan selected. The only thing this changes is the relationship to the parent of the test case record. No new records will be created. The test cases will now show under their new parents in the tree view. This action does not move any associated script nor other external documents.
Duplicating a test case does not retain any reference to the original, nor does it copy references to test logs. Test logs can be associated with only a single test case. Therefore, if you have a test case that has been executed for a given release, and then you copy that test case into a new folder for a new release, the copied test case will not retain any log information. You will be unable to easily answer the question of whether or not that test case has ever been run before and what the previous result may have been. As mentioned early in this article, it is best that you do not use the Duplicate feature to replicate a group of test plans and test cases when you move to a new release or iteration. It is much better to maintain a single set of test plans and test cases for your project, and tag them for each iteration, as described in the previous section, Using iterations effectively. This way, you simply need to open a configured test case and view the associated logs to see the execution results history across all releases.
Duplicating of test assets is a handy feature if, for example, you create a certain folder structure for organizing test plans, and you want to use this as a template of sorts for other test plans in the project or in a different project under a different asset registry. You can create the placeholder once, then duplicate it for each test plan (with the idea that the headline and other metadata will be modified as the plan is actually developed).
Ownership of test assets
Test plan, test case, and configured test case (CTC) records all have an Owner field. By default, the person who created the asset is the owner. If you use the ownership field, by assigning users ownership of test assets, it is easy to create queries that allow you to track status. Because configured test cases are what you will actually run, it is a good practice to set ownership on CTCs appropriately. Users can easily create queries to view their own CTCs, thus they can easily see what needs to be executed. Likewise, managers can run queries and reports to check status by owner for an entire team. If there is a change in resources, it is very easy to multiselect records from a query result and change ownership of several CTCs to a new owner in batch mode.
States of test assets
Test plans, test cases, and configured test cases each have a State field. The available states differ, depending on the asset.
For test plans, there are three possible states:
- Draft (default)
- For Review
For test cases, there are two states:
- Draft (default)
Configured test cases can have three states:
- Draft (default)
It is important to understand that there can be only one state setting for a given test asset. This is a limitation of the test management schema. You cannot define a different state for an asset in each iteration. If you work on only a single iteration at a time, this is not an issue. However, if you have overlapping or parallel test efforts, where you might be executing an approved test for one iteration while planning the next iteration, then you may want a way to identify more than one state. If your planning requires you to modify the CTC to account for new functionality, you might want to change the state of the CTC from Implemented back to Draft. If you do this, the test will show as Draft even though it clearly was implemented for the prior iteration.
It is important to understand that the State setting has no effect in ClearQuest test management other than as a way to label a test asset with a certain state. There is no special functionality that is related to the State field. For this reason, you may not need to bother setting states at all. Alternatively, you can use the Custom Fields option under the Legacy tab to identify different states for different iterations.
One area where the State field comes in handy is if you want to identify a set of test cases as being blocked. If there are blocked tests, you can select several CTCs simultaneously and mark them all as blocked at one time. You can then use the Blocked field to easily query on which CTCs are currently blocked.
Understanding the Latest flag
As described earlier, ClearQuest test management allows you to associate multiple iterations with a single configured test case. As you run tests and record results, you identify the iteration to which the result should be applied. During the course of an iteration, you may rerun a single CTC several times with different results. To track the final verdict, the test log record has a field called Latest, with a value of either True or False. When you commit a new test log to the database, ClearQuest runs a search to determine whether there is an existing log for that same configured test case, for that same iteration. If it finds one, it sets the Latest field to False on the old test log record and sets the new test log record to True. For a given CTC and iteration, there should be only one test log that has a Latest flag set to True. Likewise, each iteration can have a verdict for a given CTC.
As you will see in the section on reporting in Part 2, it is important to understand the Latest flag. When you report on results, you need to make sure that you include only test logs where the Latest field is set to True, so that you are certain to always get the latest test results for the iteration that you are reporting against.
Closing one test cycle and starting a new one
Part 2 walks you through how a test team can set up ClearQuest test management using the standard schema and complete a full test cycle, from planning through execution. With the infrastructure, hierarchy, and naming conventions in place, moving to the next iteration or test phase becomes very straightforward.
- First, identify the next phase by creating a new iteration, following your established naming conventions.
- If test cases that were run before will need to be rerun for this iteration, just tag these test cases with the new iteration to identify them as Planned.
- Typically, a few new test cases may need to be developed for new features, or existing test cases may need to be reworked somewhat. You can make these changes and additions, and then tag the new or modified tests with the new iteration. (Note: If an existing test that needs to be modified for a new iteration also needs to be retained for a different iteration that might be running in parallel, you can create a new test for the new iteration. If this is a common occurrence, where you have to retain more than one version of a script for the same CTC, then consider storing your scripts in a ClearCase VOB.)
- Modify your existing queries to filter on the new iteration.
- Begin testing and reporting in the new iteration.
As you can see, moving from one iteration to the next is very simple. With careful thought and planning of your ClearQuest deployment, including the practice of establishing meaningful naming conventions, you will truly be able to unleash the full benefits of ClearQuest test management.
- Find out more about the test management capabilities in IBM Rational ClearQuest: Run and analyze software tests with Rational ClearQuest Test Management, an IBM® developerWorks® tutorial by Brian Bryson (October 2006).
- FAQ: Enterprise Test Management with IBM Rational ClearQuest, Version 7
- Enroll in RT210: Essentials of Test Management with IBM Rational ClearQuest V7.0. Build test plans, test cases, and test suites that contribute to a successful test effort. The robust reporting capabilities of IBM Rational ClearQuest allow you to easily view the status of your testing and the quality of you product. These capabilities are available in the IBM Rational Software Development Platform via a ClearQuest plug-in (test management package). Students can also complete lab exercises to reinforce critical concepts and tool functionality. The focus is on the practical application of IBM Rational ClearQuest to resolve common test management challenges. This is self-directed, self-paced online learning with rich media content, including interactions, quizzes, and virtual labs. Fees for Web-based training (WBT) courses are per user, and each user gets access to the entire collection of courses for one year.
- Enroll in RS120: Essentials of IBM Rational ClearQuest, Version 7.0. This online course introduces you to using IBM Rational ClearQuest to track change requests. It covers the basic concepts involved in change management and how IBM Rational ClearQuest facilitates creating change records, controlling the change management process, and communicating status to project stakeholders using queries, reports, and charts. This is self-directed, self-paced online learning with rich media content, including interactions, quizzes, and virtual labs. Fees for Web-based training (WBT) courses are per user, and each user gets access to the entire collection of courses for one year.
- Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
- IBM Rational tester eKit to download the ClearQuest test management tool
- Browse the technology bookstore for books on these and other technical topics.
Get products and technologies
- ClearQuest users and administrators can find more resources in the ClearQuest section of the developerWorks Rational zone, including hooks, Eclipse plug-ins, product documentation, articles and whitepapers.
- Download trial versions of IBM Rational software.
- Participate in the discussion forum.
- Participate in the Rational ClearQuest forum. Get answers, advice, or opinions from other ClearQuest users. in this ClearQuest Community on developerWorks. To participate by e-mail, subscribe with a note to firstname.lastname@example.org. Subscribers may post by e-mail to email@example.com.
- Join the Test Management forum. Discuss all aspects of managing test assets and Rational TestManager in this forum. Ask questions and share experiences. To participate by e-mail, first subscribe with a note to firstname.lastname@example.org. Subscribers may post by e-mail to email@example.com.