Since 2008, the Rome Lab for IBM® Tivioli® software has used Rational® Quality Manager to support the test process. Adopting it and integrating it with the Rational Team Concert™ has improved the implementation of the agile development.
The combination gives us several advantages:
- Project lifecycle management with a test plan-centric approach
- Collaborative and adaptive test plan management
- Collaborative and adaptive test cases design
- Easy link between Rational Team Concert epics or stories and requirements and test cases in Rational Quality Manager
- Execution path optimization
- Extensible and open architecture
- Test assets and test environments management
A clear and detailed description of product requirements is one of the most effective elements to have a product release starting with high-level quality. Moreover, traceability of requirements through the software development lifecycle is a key factor to easily identify whether all requirements are developed and have a proper level of test coverage. Product managers were using a Lotus Notes database repository, giving access to a restricted list of people to discuss and refine product requirements list.
This process had several side effects:
- A limited list of empowered people
- A time-consuming process to define, refine, and maintain a list of requirement artifacts
- Information kept isolated and not linked to other product artifacts
So we moved requirements definition and management to Rational Quality Manager, because it includes a tab that the business analyst can use to add to a database by simply completing a web form.
Figure 1. Requirement definition using Rational Quality Manager
This adoption makes the entire team empowered to contribute to discussion on requirements design and development. The fast access to the web interface makes access to the tool uninterrupted, and it is easy to associate the requirements record with other artifacts.
Design your test plan
The product test plan is the central artifact in our process workflow. It describes the scope of the test and the strategy adopted to ensure product quality. It contains items such as assumptions, risk assessment, lists of platform coverage and software dependencies, test phase entry and exit criteria, quality objectives, and more. This was usually accomplished by a one person writing a text document that was stored on a Lotus Notes® database for review by a small list of reviewers and then saved, untouched, after it was approved. But as we adopted agile development methods, several of the test plan items change rapidly, so we needed to adopt a new tool to respond to the changes.
When you create a test plan in Rational Quality Manager, you can design the plan by using a template with sections that you can create, based on your needs. You can also create several templates suitable for different teams working on different aspects of the product, such as the component verification tests, the scalability and performance tests, the globalization verification tests, and the integration test. All of those teams have their own customized test plans, but those plans are all centrally managed and linked to the other product artifacts.
Figure 2 shows details of our quality assurance phase. Meaningful section for our process are now easily designed and updated through web interface access.
Figure 2. Objectives defined in the Tivoli Workload Scheduler QA test plan
Test case definition
Test cases are fundamental to maintain the desired level of quality in the developed product. Test cases are the declaration of the objective for the test team to state that the product works as expected. In the past, each test team member was asked to design test cases in several artifacts, such as a text document or a document that would be aggregated in a document called Test Design. Again, after a review and approval phase, the document would be added on a Notes database as a record for the quality testing process. There were several problems with this:
- Different granularity in designing test cases because of different author styles or skills
- Redundancy of test cases
- Lack of clarity about expected results, because the developed artifact could differ from the initial concept
Rational Quality Manager provides a built-in test case template with predefined sections that you can modify to make your own template. You can reference and link your requirements with the test case, and you can link a test script to the environment where you run it.
Figure 3. Test cases listed in Rational Quality Manager
Test execution records
Information about the physical execution of a test script, associated with the environment where it will run, is consolidated into execution records. By tracking that list of test script executions and by knowing the weight (the effort) of each one of those, the product management and the product team get the execution progress of the overall product.
In the past, you obtained those tracking purposes from manual insertion and modification of entries in an IBM® Lotus Notes® database, plus the association of the spreadsheet document. This produced an isolated level of project control, with the entire team not having the chance to verify in real time the product status. The sense of feeling part of the global project lifecycle was totally missing.
With Rational Quality Manager and Rational Team Concert, we come to the opposite: a daily scrum meeting attended by the team, where each team member reports daily duties or problems to be addressed. This makes the entire team responsible for inserting and updating any kind of information that goes to product tracking.
You can even design your test script manually, so to standardize different author's style in a template manner, and you can manually or automatically execute a test script.
Figure 4. Test script execution
When a designed test script execution encounters a problem on the developed software, you need to open a defect report to manage this event.In the past, we manually opened and managed defects in an additional tool called CMVC (IBM Configuration Management Version Control), This tool was a shared point for the tester, developer, and product manager to count and track all of the defects encountered and resolved on the project. Then, the tester manually inserted in the defect ID extracted from CMVC in the Notes database called Test Tracking Tool (TTTand associated it with the test script. Also, defect management was a problem, because any change in CMVC has to be manually reflected in TTT. With Rational Quality Manager, you simply go View Execution Results, select the records, and add the defect with a web form.
We decided to set up unified database access between Rational Quality Manager and Rational Team Concert so you can create and track defects in Rational Team Concert, where both the developing activities are listed. This makes it easier to communicate between developers and verification team members.
We already talked about our tracking process. We moved from a stylesheet document to the reporting capabilities of Rational Quality Manager and Rational Team Concert. This meant a process evolution from static business analysis to dynamic evaluation of project indicators based on data warehouse query.
Rational Quality Manager includes several predefined reports, such as reports on defects, requirements, and test cases. You can use them to analyze your test runs and identify trends in your project.
Figure 5 shows the Execution Trend report, which graphs and compares the trends of completed, attempted, and planned executions of tests. This example is based on the trend of our product quality assurance phase.
Figure 5. Sample of a report in Rational Quality Manager
We also benefit from the report in pie chart form, shown in Figure 6, that links requirements to test cases.
Figure 6. Plan Requirements Coverage by Test Cases report
Depending on analyst interest, you can look at information by requirements, tester, owner, test plan, or environment.
Test asset and test environment management
In a software development lab, test asset management gains a key role, because to ensure successful software deployment, it is crucial to have a stable environment dedicated to testing.
The Verification team in Rome has a machine fleet that comprises hundreds of physical and virtual machines. Each team has tens of assigned machines. These are the main difficulties in managing a lab of that size:
- Take inventory, both technical and business information
Each machine (physical or virtual) has to be inventoried. For each machine, we must collect technical information (hardware and software data) and business information (for example, the machine owner).
- Keep track of all updates to machines
All of the collected information needs to be up to date. Each change must be documented (for example, an operative system change or owner change).
- Share all of the machines' data with stakeholders
All of the machines' data needs to be shared with the stakeholders.
- Reserve machines for the tests
The resources should not be reserved for any longer than is strictly necessary. We identified two reservation levels: project level reservation and team level reservation. The first is related to reservation for a project: when a project starts, a group of machines is reserved for it. The second one is related to machine reservation within the same team: individual testers reserve machines for their own tests.
- Define, configure, and install the test environment
During the test plan preparation, the test leader defines the test environments. A test environment is a logical description of the environment required for a specific test or group of tests. When testers need to run tests, they must configure and install physical environments. It is very important to have an environment ready for the test in the shortest possible time.
- Share the test environment data with stakeholders
When an environment is ready for the test, it is important to share all of the information about this environment with the other team components.
Rational Quality Manager provides a way to meet each and all of those requirements. In the next section, we compare our processes for these before and after using Rational Quality Manager.
In the past, we used several tools to collect the information about the IT assets: a tool collected the technical data for physical machine, a tool collected the business data, a tool collected the data for virtual machines, a tool managed the virtual machines request, and so forth. Mainly, we manually updated the data.
The Lab Management section of Rational Quality Manager provides an interface to extend and customize the inventory capabilities. IBM® Tivoli® Provisioning Manager (TPM) provides an extension of Rational Quality Manager inventory capabilities (TPM Adapter). The adapter uses the TPM discovery to find the machines (for example, all of the machines contained within an IP range). The collected data is automatically imported into Rational Quality Manager. In this way, the information is always up to date. The physical and virtual machines are described in Rational Quality Manager by using the Lab Resource artifact.
In the past, each tool managed the data update in different way. Usually, the data was manually updated.
Using the Lab Management extension, for example the TPM Adapter, the information can be automatically updated. The update request can be triggered manually or automatically.
As mentioned previously, the stakeholders could be different. Different stakeholders are interested to different information. So another problem with using several tools was that a single user was classified in different roles in different tools.
Using Rational Quality Manager as a central information access point, we assign different roles to different users. This is important because the role associated with each account determines the options that are available and the functions that can be accessed in the Lab Manager editors.
In the past, the project-level reservation was managed by a specific tool. The team-level reservation was managed with document or email. The problem with this reservation system was static management. Each team kept machines the entire duration of the project. Even if the hardware machines were not required for the test or no longer needed, people who were designated as the owners of the machines kept the machines.
Again, Rational Quality Manager enables us to manage information in a centralized way. To reserve the machine, team members need to specify the time interval. The reservation management is transparent, so all users can view the machine reservation status. It is also possible to integrate Rational Quality Manager with tools so you know the effective use of a machine during a time interval.
Test environment creation
These steps are necessary to have a ready-to-use test environment:
- Define the test environment (components and prerequisites).
- Configure the test environment.
- Retrieve the machines.
- Reserve the machines.
- Install the middleware software.
- Install the product build.
In the past, each team leader was asked to design the test environment associated with each test case in several artifacts, such as text documents or spreadsheets. Again, after a review and approval phase, such documents would be added to a Lotus Notes database. The problems were that this was a time-consuming process to define, refine, and maintain a list of requirements artifacts, and information was isolated and unlinked to other product artifacts. For example, there wasn't a link between a logical environment (the logical description of an environment) and an instance of the test environment (a list of real machines and configurations that compose an environment). The retrieval of the machines to instantiate a test environment was done by using a spreadsheet, which was updated manually. The problem was keeping the information about the various test configurations up to date and keeping track of who was running what tests on which machines. And the environment was almost always installed manually.
Now, the test automation environments are installed using automation, but they are predefined and preconfigured. All test environment lifecycles can be managed using basic Rational Quality Manager features and extensions. The test environment can be defined by using the test environment artifacts. The tester defines all of the environment components. For each component, the tester can define the prerequisite for the component (for example, a specific operating system or required middleware).
Figure 7 shows a sample of test environment definition.
Figure 7. Sample of a test environment definition
When a tester needs to instantiate a test environment, he defines a test cell associated with the specific environment. In the test cell, Rational Quality Manager associates a lab resource for each component defined in the associated test environment, if it's available.
Figure 8. Sample of a test cell
The machine can also be reserved by starting from a test cell definition.
As Figure 9 shows, you can reference and link your test environment and test cell with a test case or a specific test execution record.
Figure 9. Test case, test environment, and test cell relationships
You can link a specific test cell with a test script, so you can run the script on the specific environment. The environment configuration (for example, middleware installation) can be done by using the integration between Rational Quality Manager and the Tivoli Provisioning Manager, called TPM Adapter.
The lab management extensibility allows us to integrate Rational Quality Manager with the test automation framework. In the Rome lab, we are running a pilot project to install the daily product build by using the integration between Rational Quality Manager and the test installation suite automation, called Build Provisioning Adapter. The Build Provisioning adapter (BP Adapter) uses the Rational Quality Manager API to perform several operations programmatically with the data in the Rational Quality Manager repository (create, read, update, and delete data).
Figure 10. List of BP Adapter custom operations
Test environment data
Formerly, the information sharing about the installed test environment was done using a spreadsheet and manually updated. The problems were keeping the information about the various test configurations up to date and keeping track of who was running what tests on which machines.
As mentioned previously, Rational Quality Manager API enables several operations with the data in Rational Quality Manager repository. The BP Adapter uses the API to automatically update the test environment information after an install operation. For example, if we use the BP Adapter to install a specific Tivoli Workload Scheduler component on a lab resource, the adapter updates the Lab Resource information automatically (see Figure 11).
Figure 11. Sample of software information
In this article, we have shown how the Rome Lab takes advantage of Rational Quality Manager and Rational Team Concert capabilities that have made it possible to design fully integrated architecture for testing and development. Rational Quality Manager gives the Verification team a useful tool to cover all test phases, using an integrative and collaborative approach.
This change is not immediate, but the Rome Lab is changing little by little. Several points discussed in this article are under development or in pilot project stages. The next steps are related to the integration of Rational Quality Manager with other IBM software, as well as internally developed software, such as a test automation suite).
- Check the Rational Quality Manager and Rational Test Lab Manager pages on developerWorks for links to product documentation, articles, tutorials, courses, downloads, and other useful areas. Also, explore the Rational Quality Manager Information Center for technical details, check the Rational Quality Manager Roadmap for links to more resources, and browse the IBM Quality Management page to learn more about what else is available. For how-to tips, read these developerWorks articles by Michael Kelly:
- Lab Management using IBM Rational Test Lab Manager: Extending IBM Rational Quality Manager for collaborative, comprehensive test planning and test asset management (developerWorks, October 2008)
- Getting started with IBM Rational Quality Manager (developerWorks, December 2008)
- Managing your first project with IBM Rational Quality Manager (developerWorks, October 2008)
- More about Rational Team Concert:
- Find Rational Team Concert articles and links to many other resources on IBM developerWorks, and check the product overview page, features and benefits, system requirements, and the user information center.
- Check the Rational Team Concert page on Jazz.net.
- Watch the Using Rational Team Concert in a globally distributed team webcast or a demonstration of the Dashboards and reports, or listen to the podcast about IBM Rational Team Concert and Jazz.
- Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
- Subscribe to the developerWorks weekly email newsletter, and choose the topics to follow. Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
- Improve your skills. Check the Rational training and certification catalog, which includes many types of courses on a wide range of topics. You can take some of themanywhere, any time, and many of the "Getting Started" ones are free.
Get products and technologies
- Download the free trial version of Rational Quality Manager, which includes Rational Test Lab Manager.
- Download Rational Team Concert from Jazz.net and try it free on up to 10 projects for as long as you want (requires registration).
- Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.
- Rate or review Rational software. It's quick and easy.
- Share your knowledge and help others who use Rational software by writing a developerWorks article. Find out what makes a good developerWorks article and how to proceed.
- Follow Rational software on Facebook, Twitter (@ibmrational), and YouTube, and add your comments and requests.
Ask and answer questions and increase your expertise when you get involved in the Rational forums, cafés, and wikis.
- Get connected. Join the Rational community to share your Rational software expertise and get connected with your peers.