IBM® Rational® Quality Manager is designed to help teams collaborate by synchronizing teamwork throughout the lifecycle and automating labor-intensive activities. Using it, teams can also better govern their projects by providing reliable and timely metrics. Rational Quality Manager is built on the Jazz platform, a collaborative, role-based, business-driven environment that provides tools for workflow control, tracking, and metrics reporting. It is a collaborative, Web-based, quality management solution that offers comprehensive test planning, manual testing, and integration with automated test tools.

Test planning is about taking your test strategy and putting it in action, often for a specific period like an iteration, sprint, or small project. This article examines the test planning process, and looks at how Rational Quality Manager supports that process. You can give Rational Quality Manager as much or as little of the planning documentation as you want. It provides tools to simplify the process where it can. In each area of planning, the goal of using Rational Quality Manager instead of a basic document or project plan is that it integrates with your reporting and metrics after the project is underway.

Thinking about test planning

When you think about your test planning process, it should not start with a document. It is a process. The first thing you need to do is understand the context for your specific company and project. Understanding the context is another way of saying understanding the values, processes, practices, philosophies, politics, and personalities of those that you will be working with. It is more than the business objectives and project requirements; it is about understanding how and why the company and team work.

Once you understand the context, start developing a test strategy. Karen N. Johnson recently gave a talk on the topic of creating a test strategy at the Pacific Northwest Software Quality Conference in Portland, Oregon. During the talk, she had a fantastic quote: "The great thing about a test strategy is that if you don't write one, one will write itself." Karen continued, pointing out that if you don't write a test plan, it will be instead formed by all of the assumptions that people have about what they think you are going to be testing. Only later will you find that you probably would have saved yourself a lot of time and energy by writing something down up front.
That is what a test strategy is all about: it is your way of telling the rest of the project team what you will and will not be testing, and how you plan to do that testing. It is a high-level communication that conveys intent. Another idea that came out of Karen's talk is that you can also view the test strategy as a testing bill of goods or statement of work. It is your way of telling people what you plan to deliver. With your test strategy, aim to answer these questions:

• What are we testing?
• What approach will we take?
• What other information do I need to plan effectively?

It is only after you know what you intend to deliver that you start planning. A test plan is what focuses on the specifics of your testing. It is logistics, test cases or scenarios, and resources, and it contains all of the dependencies and risks that you need to focus on while testing. When you plan, you estimate, find out that you cannot do everything that you want to do, negotiate scope, put dates to deliverables, and assign work.

When you plan, ask questions like:

• How will we execute our tests?
• Where will we execute them?
• When will we execute them?
• How will we manage the issues we find?
• And so on

The purpose of this is to outline and communicate the details of the testing effort for a specific period. Most likely, if you are documenting a test plan (and it is not just for regulatory or process purposes), you will use it to help direct and guide the test effort. That means that you want the information to be as correct and thorough as possible.

Following are topics that you can address with your test plan:

• Preparations
• Staffing
• Test coverage
• Any testing requirements (technical or otherwise)
• Test environments
• Entry criteria
• Exit criteria
• Delegation of responsibilities
• Facility acquisition
• Task planning
• Scheduling
• Documentation on coordination and collaboration with other teams
• Risks and issues that may impact testing
• Specific deliverables of the test project

Often when you plan, the project will begin to execute before your planning is complete. That forces you to plan and execute at the same time. When you use tools like Rational Quality Manager, you can track progress and also remember to close out any open issues from your planning process.

When you are done planning, you should have:

• Information about your context
• Information about the problem (or project)
• Ideas about your testing
• Ideas about your test coverage
• Ideas about the risks to the project
• Ideas about the details of execution
• Documents or artifacts that attempt to share the ideas you have, which are useful in challenging assumptions and understanding
• Documents or artifacts that might be required to move forward in the process (depending on the context)

Back to top

Test planning in Rational Quality Manager

This section discusses how you can use Rational Quality Manager to support your planning process. Rational Quality Manager has objects called test plans, provides customizable templates for test planning, and can provide metrics and visibility into your process against that plan. Here are a couple of ways that you can use Rational Quality Manager’s features on your projects.

Planning within your development process

You can use a number of features in Rational Quality Manager to integrate it into your development environment. Rational Quality Manager utilizes the concept of roles and workflows and a number of them are included in the product. The goal here is not to get you to do things "the Rational way", but instead to give you something you can use without modification. This way, you can learn about some of the capabilities so you know what is possible, and gain some insights into what others in the industry may be doing.

From a planning perspective, this is great because it allows you to do a couple of things. First, you can set up a test planning review process within the tool. This can include reviewers, artifact states, signoffs, and so on. In Figure 1, you can see an example of how that is implemented within the included default workflow. The state of the test plan is currently set to Draft, and you can transition the test plan to Ready for Review status. When you set up Roles, you can define who should review test plans in the Ready for Review status.

In addition to setting up simple workflows for reviews, you can assign sections of the test plan for others to work on by creating work items, as shown in Figure 2 below.

Those items then show up automatically in to-do lists and dashboards for the individual to whom they are assigned. Each work item then has its own state and possible approval process, as shown in Figure 3. For that work item, the state is new, and you can submit it for approval, review, or verification.

On the other end of the spectrum, if you do not need that level of approval and review, then you can remove it or simply not use it. You can create or remove roles as needed, map roles to various aspects of the workflows, and change the workflows. You control the process, and set it up to support your planning process.

In addition to roles, workflows, and reviews, within the test plan there are entry and exit criteria sections that can give you visibility into where you are in your development process. Many teams use entry criteria to specify when they can start testing, and exit criteria to specify when they are done. These can be tightly monitored stage gates, or heuristic indicators of when the product might be ready for more rigorous testing or when testing might be done. However you use them, they can be very handy, because you can track them in one central place and report on them as necessary using automatically generated reports. An example of tracking entry criteria is shown in Figure 4.

Note that even for criteria items you can create work items. In the previous example, you might create work items for the three outstanding test environment configurations that need to be set up. Then you could potentially track the status of those activities more closely.

Regardless of the section of the test plan, you can track individual outstanding items from the plan for Rational Quality Manager (if you want to). The advantage of using tools like this over something like a Microsoft® Project plan is that it keeps you and your team in the tool where the rest of the work is done. It also integrates your project tracking and reporting.

Planning for coverage

One of the key tools for tracking and reporting testing progress in Rational Quality Manager is the test plan. Rational Quality Manger has several requirements features that help you manage requirements coverage. In the test plan, there is a Requirements section (shown in Figure 5) for managing all of the requirements that you will cover in a given test plan. If you want to track all of your requirements from Rational Quality Manager, you can. If you want to import them from another tool, you can. Alternatively, if you want to just create and track some generic testing requirements, that is also possible.

Many projects have great functional requirements coverage (the application should do X, it should not do Y, and so on), but they rarely have requirements for para-functional requirements. That does not mean that you don't test them: you do. However, it is always difficult to keep track of where that testing is in terms of status and coverage. If you create your own requirements, you can add requirements for performance, security, usability, and other often-overlooked areas. You can then tie test cases back to those requirements to track coverage and status at the test plan level.

In Rational Quality Manager, you can explicitly define your quality objectives in the Quality Objectives section of the test plan, shown in Figure 6. This section lists, in table format, your quality objectives for a release. You can free-form edit the Quality Objectives Description, the Current Value, and the Comment field (not shown), allowing you to specify just about any objective that you would like.

Some possible objectives include measures around the following areas:

• Code complexity
• Unit testing success
• Code coverage
• Requirements coverage
• Test case completion (percent complete, percent pass, and so on) by area
• Load, performance, or scalability
• Open issue or defect severity, volume, or status
• Defect arrival rates or testing velocity
• Test case or requirement priority or severity
• Compliance to a standard (section 508, W3C, and so on)
• Documentation or evidence requirements

The quality criteria you choose will depend heavily on what you are trying to accomplish with the project, and what development context you are working in. Whatever you choose, the Quality Objectives section provides an excellent snapshot of where the project is from a quality perspective.

One of the great features in Rational Quality Manager is the Test Environments planning section within the test plan. When you first open that section, you are prompted to define the platform requirements that need to be covered. As shown in Figure 7 below, all you need to do is define which types of platform components you need to cover, and what versions or attributes you need to test. You just create a simple list of what needs to be tested.

From there, you can move on to defining coverage automatically based on a couple of different coverage models. If you switch tabs to the Test Environment tab (as shown in Figure 8), you see a different view, which will eventually contain each environment that you want to cover.

After you save your test plan, if you click the Generate New Test Environments icon, you launch a wizard that walks you through generating the initial coverage list. The first step of that wizard, shown in Figure 9, is to define which elements you want to cover, along with what generation method you want to use.

There are a couple of coverage methods provided, including one-way, pair-wise, and three-wise interactions, and all permutations. The choice you make here determines how many environments you will have to cover. Few teams have the resources to test on all permutations, so it is a question of what level of risk is acceptable for you and your project team. If necessary, you can enhance your generation process in the future: change advanced properties for environment elements, and add explicit inclusions, exclusions, and weightings.

After you select the coverage method that you prefer and click Next, you get a chance to review the generated environments before accepting them. That way, you can make changes if needed. Figure 10 shows the environments generated using pair-wise coverage, sorted by browser.

When you accept the environments, they are added to the Test Environments table in the test plan (as shown in Figure 11). From here, you can remove any of them if they are no longer required, or manually add records if you have a new configuration that needs to be added. You can also edit any of the specific environments as needed.

Planning for execution

One of the trickier aspects of the planning process is planning for execution. You need to take into account these things (some of which you’ll know up front, and others that you won’t):

• The number of testers
• The level of coverage needed for each area of the application, the environment and configuration, or quality criteria
• The size and scope of the initial pool of tests that you have to execute
• The period of time that you believe you have to execute your tests
• Some estimate for how many issues you feel that you will encounter and will need to work through
• Some estimate of how many new tests that you will uncover and will need to run
• Some estimate of how many tests you initially planned that you may not need to run

To make things even more difficult, as a test manager you do not get to do your planning in a vacuum. You have to account for the dependencies with other teams and managers. On past projects, planning has taken place over a number of documents (planning documents, project plans, estimation spreadsheets, and so on), with rounds of meetings and reviews. On current projects, the planning tends to be faster and involves fewer people, but it still has to account for both what you know and what you don’t know.

Some of the features that really make Rational Quality Manager test plans stand out are the Test Schedules, Test Estimation, and Test Team sections. Those three sections tie together all of the other sections (Entry and Exit Criteria, Test and Quality Objectives, Requirements, and Test Cases) in a way that helps you paint a picture of what execution might look like.

Within Rational Quality Manager’s administration screens, you can set up and manage different test teams. You can do this using one-to-many assignments, meaning that the same person can be on multiple teams (which is often the case). Once you have that set up, if you select a test team within a plan, you can see which team members are assigned to this project (as shown in Figure 12). Those team members are then available for task assignments, test case assignments, and other actions within the plan.

As part of the planning process, you can create high-level estimates of the size of the test planning and test execution efforts. You can also provide detailed estimates of the time or effort required to run each individual test case. These estimates help to measure your progress, and they provide input to several reports.

In the early stages of a test project, you can provide high-level estimates of the time required to complete your test planning activities, and the time or effort required to run all of your tests. These estimates are often based on what you know about the project requirements. Figure 13 shows an example of defining high-level estimates within the test plan. This method of top-down planning can be helpful early on.

Later in the plan, you can provide a more detailed estimate of the test execution effort by adding a weight value to each test case. In Rational Quality Manager, test execution records inherit the weight of their associated test case. A test case that is assigned a weight of 10, for example, might take twice as long to run as a test case that has an assigned weight of five. Typically, the weight is a numerical value for whatever unit of measurement your test team is accustomed to using.

Some test teams might think of weight in terms of points, while others might think in terms of hours, minutes, or other measurements. This detailed sizing information is used as input to several execution status reports. By assigning various weights to each test case, you can run accurate reports that take into consideration both the absolute number of test cases that are run and the required time or level of effort to run each test case.

After high-level estimates, you can define a high-level schedule in the Test Schedule section of the test plan (shown in Figure 14). For each milestone or iteration, you can create high-level schedules that list key project dates like final release, code freeze, UI freeze, beta entry, beta exit, and other dates.

Planning for automation

From a test planning perspective, the last perspective to consider is how Rational Quality Manager can help you with your planning around the use of automation. You can set clear quality objectives for things like performance and security testing, and defining environments for your automated test coverage. You can also do some up-front planning for which tools you plan to use, and where you plan to use them.

Within your test plan and all of the testing elements that tie to it in Rational Quality Manager, you have several opportunities to plan and manage your automation efforts. First, you can add custom tags to your requirements. This gives you the ability to do some up-front planning and organization around the use of automation. For example, if you are reviewing a requirement, you could tag it with the following:

• A regression keyword for a requirement for which you want to build out an automated regression test case
• A performance keyword for a requirement for which you need to develop a performance test, or which might tie into a performance test based on some aspect of the requirement
• A configuration keyword for a requirement that might drive automated testing across multiple environments
• An SOA keyword for a requirement that you might want to test at the Web service interface level
• A security keyword for a requirement that you might use IBM® Rational® AppScan® (or some other tool) to test

The idea is to provide some simple tags that you can report on later when planning those automation efforts.

After that, you can look at tying specific automation scripts and tests to test case artifacts. Alternatively, you can assign test cases to categories for some of the different types of automation you might have on your project. Aside from tagging and traceability, consider adding the different automation, performance, security testing, and Web service testing tasks to your schedule and estimates. In this way, you can integrate them fully with the rest of the testing project.

Back to top

Next steps

Think about your test planning process today, and the tools that you use to support it. If you use more than one or two, you might consider moving some of that planning into a tool like IBM Rational Quality Manager. Also, think about how information is shared across your team. Do you use a complicated set of SharePoint folders, space on a network share, or e-mail files back and forth? Those are also signs that you might want to move to a tool that helps distribute some of that information more naturally. Use Rational Quality Manager to develop a couple of plans, work with the data, and see if you can think of changes you can make up front to make reporting easier later.

The next logical step is to use the information that you have captured in your test plan while executing your project. The article "Test analysis and reporting using IBM Rational Quality Manager" picks up right where this article leaves off. It takes an in-depth look at test analysis and reporting using Rational Quality Manager.

Resources

Learn

• For more on getting started with Rational Quality Manager, check out "Managing your first project with IBM Rational Quality Manager" by Michael Kelly
  
  
• For more about Lab Management in Rational Quality Manager and how you can use that to drive reporting visibility, check out "Lab management using IBM Rational Quality Manager" by Michael Kelly
  
  
• You can find more information about IBM Rational Quality Manager and related products here.
  
  
• For small tips, tricks, and happenings with IBM Rational Quality Manager you can track the Rational Quality Manager blog.
  
  
• And as always, you can find more articles like this one at Rational on developerWorks.
  
  
• Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
  
  
• Subscribe to the IBM developerWorks newsletter, a weekly update on the best of developerWorks tutorials, articles, downloads, community activities, webcasts and events.
  
  
• Subscribe to the developerWorks Rational zone newsletter. Keep up with developerWorks Rational content. Every other week, you'll receive updates on the latest technical resources and best practices for the Rational Software Delivery Platform.
  
  
• Subscribe to the Rational Edge newsletter for articles on the concepts behind effective software development.
  
  
• Browse the technology bookstore for books on these and other technical topics.
  
  

Get products and technologies

• Download trial versions of IBM Rational software.
  
  

Discuss

• Participate in the discussion forum.
  
  
• Join the developerWorks Community in forums, blogs, podcasts, wikis, and more.
  
  

About the author

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Thinking about test planning
• Test planning in Rational Quality Manager
• Next steps
• Resources
• About the author
• Comments

Next steps from IBM

• Try: Rational Quality Manager
• Demo: Video: Quality management in 3 minutes
• Webcast: Reduce cost and speed time to market with quality management and asset management
• Briefing: Quality driven software delivery
• Buy: Rational Quality Manager

My developerWorks community

Dig deeper into Rational on developerWorks

Rate a product. Write a review.

Write a review for Rational products today!

Special offers