Rational tools make testing much easier for IBM Tivoli lab in Rome

Improving product quality is a matter of both team process and the tools that the team uses. In this article, we describe how the IBM Tivoli software lab in Rome migrated from TTT, a Lotus Notes database, and text documents to the software development lifecycle approach by integrating artifacts from the Rational solution to Collaborative Lifecycle Management (CLM). We describe how we used IBM Rational Team Concert Version 2.0.0.2 and IBM Rational Quality Manager Version 3.0.1.1 to achieve a new day-by-day process. Moreover, adopting those tools made architects, project managers, developers, and testers able to feed and maintain the product artifacts linked together in a centralized way. Product requirements, developed user stories, and test cases are easily traced and tracked within a single point of entry.

Alessandro De Micco (alessandrodemicco@it.ibm.com), Software Engineer, IBM

Author1 photoAlessandro De Micco has been working as a software engineer on the Tivoli Rome Lab's Verification team since 2004. He has been on the Tivoli Workload Scheduler quality assurance team since 2006, and he has been working on integrating Tivoli Workload Schedule with Rational products for software lifecycle management.



Ilaria Gorga (ilaria.gorga@it.ibm.com), Software Engineer , IBM

Author1 photoIlaria Gorga has worked as a software engineer in the IBM Tivoli Rome Lab since 2008. She has been on the Tivoli Workload Scheduler quality assurance team since 2010 and focuses on test automation optimization. Ilaria has written several other articles and presented at conferences, such as Eclipse-IT.



07 August 2012

Introduction

Since 2008, the Rome Lab for IBM® Tivioli® software has used Rational® Quality Manager to support the test process. Adopting it and integrating it with the Rational Team Concert™ has improved the implementation of the agile development.

The combination gives us several advantages:

  • Project lifecycle management with a test plan-centric approach
  • Collaborative and adaptive test plan management
  • Collaborative and adaptive test cases design
  • Easy link between Rational Team Concert epics or stories and requirements and test cases in Rational Quality Manager
  • Execution path optimization
  • Extensible and open architecture
  • Test assets and test environments management

Requirements definition

A clear and detailed description of product requirements is one of the most effective elements to have a product release starting with high-level quality. Moreover, traceability of requirements through the software development lifecycle is a key factor to easily identify whether all requirements are developed and have a proper level of test coverage. Product managers were using a Lotus Notes database repository, giving access to a restricted list of people to discuss and refine product requirements list.

This process had several side effects:

  • A limited list of empowered people
  • A time-consuming process to define, refine, and maintain a list of requirement artifacts
  • Information kept isolated and not linked to other product artifacts

So we moved requirements definition and management to Rational Quality Manager, because it includes a tab that the business analyst can use to add to a database by simply completing a web form.

Figure 1. Requirement definition using Rational Quality Manager
Requirement definition dialog window

Larger view of Figure 1.

This adoption makes the entire team empowered to contribute to discussion on requirements design and development. The fast access to the web interface makes access to the tool uninterrupted, and it is easy to associate the requirements record with other artifacts.


Design your test plan

The product test plan is the central artifact in our process workflow. It describes the scope of the test and the strategy adopted to ensure product quality. It contains items such as assumptions, risk assessment, lists of platform coverage and software dependencies, test phase entry and exit criteria, quality objectives, and more. This was usually accomplished by a one person writing a text document that was stored on a Lotus Notes® database for review by a small list of reviewers and then saved, untouched, after it was approved. But as we adopted agile development methods, several of the test plan items change rapidly, so we needed to adopt a new tool to respond to the changes.

When you create a test plan in Rational Quality Manager, you can design the plan by using a template with sections that you can create, based on your needs. You can also create several templates suitable for different teams working on different aspects of the product, such as the component verification tests, the scalability and performance tests, the globalization verification tests, and the integration test. All of those teams have their own customized test plans, but those plans are all centrally managed and linked to the other product artifacts.

Figure 2 shows details of our quality assurance phase. Meaningful section for our process are now easily designed and updated through web interface access.

Figure 2. Objectives defined in the Tivoli Workload Scheduler QA test plan
Screen captures shows test objectives

Test case definition

Test cases are fundamental to maintain the desired level of quality in the developed product. Test cases are the declaration of the objective for the test team to state that the product works as expected. In the past, each test team member was asked to design test cases in several artifacts, such as a text document or a document that would be aggregated in a document called Test Design. Again, after a review and approval phase, the document would be added on a Notes database as a record for the quality testing process. There were several problems with this:

  • Different granularity in designing test cases because of different author styles or skills
  • Redundancy of test cases
  • Lack of clarity about expected results, because the developed artifact could differ from the initial concept

Rational Quality Manager provides a built-in test case template with predefined sections that you can modify to make your own template. You can reference and link your requirements with the test case, and you can link a test script to the environment where you run it.

Figure 3. Test cases listed in Rational Quality Manager
List shows name, number, weight, date modified

Larger view of Figure 3.


Test execution records

Information about the physical execution of a test script, associated with the environment where it will run, is consolidated into execution records. By tracking that list of test script executions and by knowing the weight (the effort) of each one of those, the product management and the product team get the execution progress of the overall product.

In the past, you obtained those tracking purposes from manual insertion and modification of entries in an IBM® Lotus Notes® database, plus the association of the spreadsheet document. This produced an isolated level of project control, with the entire team not having the chance to verify in real time the product status. The sense of feeling part of the global project lifecycle was totally missing.

With Rational Quality Manager and Rational Team Concert, we come to the opposite: a daily scrum meeting attended by the team, where each team member reports daily duties or problems to be addressed. This makes the entire team responsible for inserting and updating any kind of information that goes to product tracking.

You can even design your test script manually, so to standardize different author's style in a template manner, and you can manually or automatically execute a test script.

Figure 4. Test script execution
Installation Execution, Manual Script view

Larger view of Figure 4.


Defect management

When a designed test script execution encounters a problem on the developed software, you need to open a defect report to manage this event.In the past, we manually opened and managed defects in an additional tool called CMVC (IBM Configuration Management Version Control), This tool was a shared point for the tester, developer, and product manager to count and track all of the defects encountered and resolved on the project. Then, the tester manually inserted in the defect ID extracted from CMVC in the Notes database called Test Tracking Tool (TTTand associated it with the test script. Also, defect management was a problem, because any change in CMVC has to be manually reflected in TTT. With Rational Quality Manager, you simply go View Execution Results, select the records, and add the defect with a web form.

We decided to set up unified database access between Rational Quality Manager and Rational Team Concert so you can create and track defects in Rational Team Concert, where both the developing activities are listed. This makes it easier to communicate between developers and verification team members.


Reporting capabilities

We already talked about our tracking process. We moved from a stylesheet document to the reporting capabilities of Rational Quality Manager and Rational Team Concert. This meant a process evolution from static business analysis to dynamic evaluation of project indicators based on data warehouse query.

Rational Quality Manager includes several predefined reports, such as reports on defects, requirements, and test cases. You can use them to analyze your test runs and identify trends in your project.

Figure 5 shows the Execution Trend report, which graphs and compares the trends of completed, attempted, and planned executions of tests. This example is based on the trend of our product quality assurance phase.

Figure 5. Sample of a report in Rational Quality Manager
Completed and planned execution curves compared

We also benefit from the report in pie chart form, shown in Figure 6, that links requirements to test cases.

Figure 6. Plan Requirements Coverage by Test Cases report
Pie chart from Rational Quality Manager

Depending on analyst interest, you can look at information by requirements, tester, owner, test plan, or environment.


Test asset and test environment management

In a software development lab, test asset management gains a key role, because to ensure successful software deployment, it is crucial to have a stable environment dedicated to testing.

The Verification team in Rome has a machine fleet that comprises hundreds of physical and virtual machines. Each team has tens of assigned machines. These are the main difficulties in managing a lab of that size:

Take inventory, both technical and business information
Each machine (physical or virtual) has to be inventoried. For each machine, we must collect technical information (hardware and software data) and business information (for example, the machine owner).
 
Keep track of all updates to machines
All of the collected information needs to be up to date. Each change must be documented (for example, an operative system change or owner change).
 
Share all of the machines' data with stakeholders
All of the machines' data needs to be shared with the stakeholders.
 
Reserve machines for the tests
The resources should not be reserved for any longer than is strictly necessary. We identified two reservation levels: project level reservation and team level reservation. The first is related to reservation for a project: when a project starts, a group of machines is reserved for it. The second one is related to machine reservation within the same team: individual testers reserve machines for their own tests.
 
Define, configure, and install the test environment
During the test plan preparation, the test leader defines the test environments. A test environment is a logical description of the environment required for a specific test or group of tests. When testers need to run tests, they must configure and install physical environments. It is very important to have an environment ready for the test in the shortest possible time.
 
Share the test environment data with stakeholders
When an environment is ready for the test, it is important to share all of the information about this environment with the other team components.
 

Rational Quality Manager provides a way to meet each and all of those requirements. In the next section, we compare our processes for these before and after using Rational Quality Manager.

Inventory

In the past, we used several tools to collect the information about the IT assets: a tool collected the technical data for physical machine, a tool collected the business data, a tool collected the data for virtual machines, a tool managed the virtual machines request, and so forth. Mainly, we manually updated the data.

The Lab Management section of Rational Quality Manager provides an interface to extend and customize the inventory capabilities. IBM® Tivoli® Provisioning Manager (TPM) provides an extension of Rational Quality Manager inventory capabilities (TPM Adapter). The adapter uses the TPM discovery to find the machines (for example, all of the machines contained within an IP range). The collected data is automatically imported into Rational Quality Manager. In this way, the information is always up to date. The physical and virtual machines are described in Rational Quality Manager by using the Lab Resource artifact.

Machine updates

In the past, each tool managed the data update in different way. Usually, the data was manually updated.

Using the Lab Management extension, for example the TPM Adapter, the information can be automatically updated. The update request can be triggered manually or automatically.

Machine data

As mentioned previously, the stakeholders could be different. Different stakeholders are interested to different information. So another problem with using several tools was that a single user was classified in different roles in different tools.

Using Rational Quality Manager as a central information access point, we assign different roles to different users. This is important because the role associated with each account determines the options that are available and the functions that can be accessed in the Lab Manager editors.

Machine reservations

In the past, the project-level reservation was managed by a specific tool. The team-level reservation was managed with document or email. The problem with this reservation system was static management. Each team kept machines the entire duration of the project. Even if the hardware machines were not required for the test or no longer needed, people who were designated as the owners of the machines kept the machines.

Again, Rational Quality Manager enables us to manage information in a centralized way. To reserve the machine, team members need to specify the time interval. The reservation management is transparent, so all users can view the machine reservation status. It is also possible to integrate Rational Quality Manager with tools so you know the effective use of a machine during a time interval.

Test environment creation

These steps are necessary to have a ready-to-use test environment:

  1. Define the test environment (components and prerequisites).
  2. Configure the test environment.
  3. Retrieve the machines.
  4. Reserve the machines.
  5. Install the middleware software.
  6. Install the product build.

In the past, each team leader was asked to design the test environment associated with each test case in several artifacts, such as text documents or spreadsheets. Again, after a review and approval phase, such documents would be added to a Lotus Notes database. The problems were that this was a time-consuming process to define, refine, and maintain a list of requirements artifacts, and information was isolated and unlinked to other product artifacts. For example, there wasn't a link between a logical environment (the logical description of an environment) and an instance of the test environment (a list of real machines and configurations that compose an environment). The retrieval of the machines to instantiate a test environment was done by using a spreadsheet, which was updated manually. The problem was keeping the information about the various test configurations up to date and keeping track of who was running what tests on which machines. And the environment was almost always installed manually.

Now, the test automation environments are installed using automation, but they are predefined and preconfigured. All test environment lifecycles can be managed using basic Rational Quality Manager features and extensions. The test environment can be defined by using the test environment artifacts. The tester defines all of the environment components. For each component, the tester can define the prerequisite for the component (for example, a specific operating system or required middleware).

Figure 7 shows a sample of test environment definition.

Figure 7. Sample of a test environment definition
Screen capture includes a workflow diagram

Larger view of Figure 7.

When a tester needs to instantiate a test environment, he defines a test cell associated with the specific environment. In the test cell, Rational Quality Manager associates a lab resource for each component defined in the associated test environment, if it's available.

Figure 8. Sample of a test cell
Definition screen: description and lab resources

Larger view of Figure 8.

The machine can also be reserved by starting from a test cell definition.

As Figure 9 shows, you can reference and link your test environment and test cell with a test case or a specific test execution record.

Figure 9. Test case, test environment, and test cell relationships
Screen capture of test environment relationships

Larger view of Figure 9.

You can link a specific test cell with a test script, so you can run the script on the specific environment. The environment configuration (for example, middleware installation) can be done by using the integration between Rational Quality Manager and the Tivoli Provisioning Manager, called TPM Adapter.

The lab management extensibility allows us to integrate Rational Quality Manager with the test automation framework. In the Rome lab, we are running a pilot project to install the daily product build by using the integration between Rational Quality Manager and the test installation suite automation, called Build Provisioning Adapter. The Build Provisioning adapter (BP Adapter) uses the Rational Quality Manager API to perform several operations programmatically with the data in the Rational Quality Manager repository (create, read, update, and delete data).

Figure 10. List of BP Adapter custom operations
Automations screen

Larger view of Figure 10.

Test environment data

Formerly, the information sharing about the installed test environment was done using a spreadsheet and manually updated. The problems were keeping the information about the various test configurations up to date and keeping track of who was running what tests on which machines.

As mentioned previously, Rational Quality Manager API enables several operations with the data in Rational Quality Manager repository. The BP Adapter uses the API to automatically update the test environment information after an install operation. For example, if we use the BP Adapter to install a specific Tivoli Workload Scheduler component on a lab resource, the adapter updates the Lab Resource information automatically (see Figure 11).

Figure 11. Sample of software information
Type, Name, Product Name, Location, other fields

Summary

In this article, we have shown how the Rome Lab takes advantage of Rational Quality Manager and Rational Team Concert capabilities that have made it possible to design fully integrated architecture for testing and development. Rational Quality Manager gives the Verification team a useful tool to cover all test phases, using an integrative and collaborative approach.

This change is not immediate, but the Rome Lab is changing little by little. Several points discussed in this article are under development or in pilot project stages. The next steps are related to the integration of Rational Quality Manager with other IBM software, as well as internally developed software, such as a test automation suite).

Resources

Learn

Get products and technologies

  • Download the free trial version of Rational Quality Manager, which includes Rational Test Lab Manager.
  • Download Rational Team Concert from Jazz.net and try it free on up to 10 projects for as long as you want (requires registration).
  • Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational, DevOps
ArticleID=829585
ArticleTitle=Rational tools make testing much easier for IBM Tivoli lab in Rome
publish-date=08072012