IBM's Test Automation Strategy: Build your test automation architecture around IBM Rational Quality Manager

A robust and usable test automation solution

Over the past years, the aim of improving the development life cycle and the quality of hardware and software products has led to test automation. One major challenge has been that each team has unique automation needs. These unique automation needs have led many teams to create their own automation solutions. Most of these automation solutions neither create components that can be reused by other automation deployments, nor use any existing software to standardize and minimize their effort. This leads to duplication of work and the proliferation of non-standard products. IBM® Systems and Technology Group has created a test automation strategy that encourages reuse, building a flexible solution that uses IBM® Rational® Quality Manager as a hub to coordinate other products with various automated test capabilities.

David W. Mehaffy (dmehaffy@us.ibm.com), System Assurance Release Architect, IBM Systems and Technology Group

David MehaffyDr. David Mehaffy is a Systems Assurance Release Architect in the IBM Systems and Technology Group (STG) systems test area. He works on improving the automation and efficiency of testing across all STG products. David has been with IBM for 26 years, and has worked in many areas in AIX development as well as technical marketing support.



William W. Owen (billowen@us.ibm.com), Systems/Product Assurance Engineering Professional, IBM Systems and Technology Group

Bill OwenBill Owen is a Systems and Product Assurance Engineering Professional. He is working as a senior lead for test tools and infrastructure development in The IBM Systems and Technology Group. He has more than 20 years of experience, and has been a part of major IBM forums and groups like Quality Software Engineering (QSE).



Ashish Aggarwal (ashisagg@in.ibm.com), System Software Engineer, IBM Systems and Technology Group

Ashish AggarwalAshish Aggarwal is a Software Engineer who has worked with IBM for more than three years. In that time, he has been working for the development of automation tools. He is involved with many IBM forums like Quality Software Engineering (QSE) and so on, and plays an active role in the developerWorks forums.



31 August 2010

Also available in Chinese

Introduction

IBM® has a line of products that can meet the automation needs of any organization and team. The challenge is to bring them together to create a robust solution. Now IBM has documented an architecture that not only uses existing standard products, but also provides reusable components to create a robust, customizable, and usable automation architecture. The aim of this architecture is not to reinvent the wheel, but rather utilize existing components and infrastructure. The architecture explained in this article provides a complete picture of test automation. Organizations and teams can use parts of the solution according to their needs and priorities.


Test automation myths and facts

Before explaining the architecture, it is important for test teams to understand some common myths about test automation. Today, most test teams understand the need for test automation; test automation is becoming more prevalent with each passing day. With the dynamic and unpredictable nature of the testing process and the need to minimize test schedules, Quality Assurance (QA) managers are also realizing a greater need for test automation. But most test teams believe some common myths, including:

  • Test automation is a sequence of actions: Test teams consider test automation to be a sequence of events to be executed one after another. However, in actual environments, executing a test constitutes a complex and inter-related set of events on which, many times, the nature of test execution depends. Manual testing can easily detect this complex set of events and determine the direction of test execution. Rather than "boiling the ocean" (trying to do everything at once), you should initially focus on automating only those test steps that lend themselves to simple and straight-forward automation procedures. Apply automation to a smaller set of test steps instead of to all of the test steps.
  • Every manual test can be automated: It is a myth to think every manual test can and should be converted into automated steps. Before developing an automation solution, it is important to estimate how much cost can be saved with each test step being converted into an automated step. If the automated step does not lead to significant cost savings, then it will not be pragmatic to convert the manual step to automated ones. As a QA manager, you must make a trade off between which steps are to be automated and which steps are to be executed manually.
  • Test automation always leads to cost savings: Most teams consider the cost of developing, maintaining, and operating an automation tool as the total cost of an automation solution. But what looks good on paper is not always true. While evaluating cost, you might overlook the dynamic and unpredictable nature of testing procedures. A small change in the testing procedures many times leads to significant changes in the automation tool, and hence more cost. Other important factors (like the cost involved in training test teams, documenting test cases, and testing the test automation itself) are rarely considered.
  • Test automation is just a matter of purchasing the "right" box: Many times, teams buy a set of tools to accomplish their automation requirements and consider this to be the end of the story. But this is rarely the case. Purchasing a tool is just a starting step towards automating the testing procedures. Teams need to segregate which tests should be automated and which are to be manually completed, document testing procedures, and train the test teams. Also, it is rare that an off the shelf tool meets all of the requirements of the purchasing team. Missing features must somehow be accommodated before the tool can be fully deployed. This gap between the expectations and real delivered value leads to dissatisfaction and, many times, the failure of the test automation project.

Common errors

Despite being aware of the myths explained above, some test teams and QA managers proceed to design and develop a test team-specific automation tool. In most cases, there are some elementary errors in architecture that lead to the improper use of resources, escalation of development and maintenance costs and, potentially, test automation failure. The common mistakes made by test automation development teams are listed below:

  • Non-extensible automation architecture: This is one of the most common mistakes an automation development team makes while designing a tool. In the majority of scenarios, automation development teams design the tool which satisfies the requirements of only one test team. Even if some of the features of the tool can be reused by other test teams, the non-extensible architecture deters the new test teams from adopting the tools. Such a situation leads to other teams developing a similar tool again.
  • Duplication of efforts: Duplicating automation effort is another major culprit that deters the success of test automation solutions across organizations. As explained above, most of the test automation solutions have a non-extensible architecture that creates non-reusable components. You should look for existing solutions and products to leverage before going ahead with a new development project.
  • Proliferation of custom tools: Many test automation tools are developed from scratch. In most cases, however, the same work flow could have been accomplished directly by an existing product or an existing product with some extensions. IBM has a line of test automation products that you can use directly, or extend to satisfy the requirements of most test automation teams. This approach not only reduces the work effort required to automate testing, but also promotes standardization among test teams. You should spend time up front doing research about existing tools and find an architecture that fits your environment.
  • Not enough communication or sharing between teams: Even if teams avoid the previously mentioned mistakes, lack of communication among the test teams deters the effectiveness of an extensible and reusable architecture. It is important for you to spread the word about any existing automation tools, to increase the knowledge and understanding about them, and ultimately minimize the testing effort and reduce testing costs.

Test automation architecture

The products described in this section provide the components of a flexible, extensible automation architecture. IBM® Rational® Quality Manager is a web-based, centralized test management environment that provides a collaborative and customizable solution for test planning, workflow control, test tracking, and metrics reporting. It is the hub of the architecture, and provides the central integration point with the other components.

Figure 1 shows a complete view of the proposed test automation architecture, and of the interaction among different products.

Figure 1. IBM's test automation architecture
IBM's Test Automation Architecture

Next, this article lists the different test automation functions, and the IBM products that are used to meet the requirements of those functions.

Note: There are a few instances of duplicated functions in this architecture where two or more products have implemented the same feature. The most obvious choice is highlighted, and the reasons for use of the selected function will be explained.

Test management: Rational Quality Manager

Rational Quality Manager is the central hub of test activity. It provides a set of features and integrations to other tools to help streamline and automate the following functions:

  • Planning the test effort
  • Managing requirements
  • Developing test cases, test suites, and test execution records
  • Developing manual test scripts
  • Managing test assets
  • Managing lab resources
  • Managing product builds
  • Executing and monitoring automated test scripts
  • Submitting and tracking defects

Using a Representational State Transfer (RESTful) interface architecture, Rational Quality Manager provides straightforward integrations with external products (such as IBM® Rational® Requirements Composer for requirements management), as well as the integrations with other products described in the following sections.

Rational Quality Manager also leverages the RESTful interface in such a way that you can write custom adapters that read and write Rational Quality Manager resources for a variety of purposes. One example is that you can integrate existing test automation frameworks with the Rational Quality Manager server, so that your deployment of Rational Quality Manager can be a gradual, phased process, not “rip and replace” (all at once).

Reporting in Rational Quality Manager

With Rational Quality Manager, you also get an enterprise class reporting solution with the IBM® Rational® Insight product. Many standard data items and test reports are provided ready to use with Rational Insight. Using Rational Insight development tools, you can specify additional data to collect in the Rational Insight Data Warehouse component, publish specific data warehouse metadata to simplify report authoring, and author a wide range of custom reports and dashboards. Rational Insight also allows reports using data from more than one Rational Quality Manager project area or server.

Setting up defect integration with Rational Team Concert

You can integrate Rational Quality Manager and IBM® Rational Team Concert™ allowing you to use Rational Team Concert as your defect provider and to synchronize defects and other work items with Rational Quality Manager. After establishing this integration, you can track work items (including defects) in Rational Quality Manager, even though the work items are maintained in Rational Team Concert.

The ClearQuest bridge

You can integrate Rational Quality Manager and IBM® Rational® ClearQuest® defect tracking system using the ClearQuest bridge. The ClearQuest bridge lets you work with ClearQuest records directly from the Rational Quality Manager environment. You can create, view, and modify ClearQuest records, run ClearQuest queries, and associate ClearQuest records with Rational Quality Manager execution results. The ClearQuest bridge does not require the creation and maintenance of synchronization rules or the duplication of data in multiple repositories; it links directly to the ClearQuest data.

Test resource provisioning: Tivoli Provisioning Manager

IBM Tivoli Provisioning Manager is IBM's enterprise class component for automating data center provisioning activities. It provides a broad range of capabilities, including:

  • Discovery of IT resources
  • Provisioning of operating systems, software packages, storage, and network resources
  • A detailed datacenter model that can represent all of the IT resources in a test lab
  • A powerful automation package development environment to create or customize automation packages
  • An automation engine for executing and monitoring automation packages on remote computers
  • A library of automation packages that can be used as is, or as examples for your custom workflows
  • Software compliance management and license management

Tivoli Provisioning Manager provides a SOAP (simple open access protocol) interface that facilitates integration with other software components. Using this interface, external components can request operations (for example, the execution of a discovery or provisioning workflow), or retrieve information from the datacenter model.

Within the test automation architecture, Rational Quality Manager delegates provisioning tasks to Tivoli Provisioning Manager through its integration interface. This integration makes Tivoli Provisioning Manager resources and automation packages visible in Rational Quality Manager's Lab Management subsystem. As the test engineer, you can request the execution of an automation project on one or more lab resources. The Rational Quality Manager interface simply requests that Tivoli Provisioning Manager execute an automation project and then displays the results of the automation project when the results become available.

Tivoli Provisioning Manager's broad collection of features can create complexity, because you can use it for every automation task imaginable. Instead, our strategy has been to focus on the features that Tivoli Provisioning Manager does uniquely well: that is, managing the complex tasks in the test lab infrastructure. For example, use it to manage the provisioning of operating systems on all of your required platforms. Simpler automation tasks can be handled easily by the general purpose workflow engine, IBM® Rational® Build Forge®.

Each site will typically have a single production instance of Tivoli Provisioning Manager, though there may be special circumstances where more than one production Tivoli Provisioning Manager server is justified.

General purpose automation engine: Rational Build Forge

Rational Build Forge is the workflow engine in the architecture. It is a straightforward product that enables you to specify projects that contain steps to be executed. The steps may be one or more commands that will be executed on a selected server in the test infrastructure. Rational Build Forge automation projects can be written in a way so that they can be embedded into larger projects. This provides a simple mechanism for you to deploy very complex test automation processes using a collection of well-tested, lower-level building block projects.

Rational Build Forge provides built-in mechanisms for error detection and recovery when a project step fails. You can specify that another project be run following a failure. That project can clean up after the failure before continuing with the automation process. There is also the concept of environments, which enables the automation logic to bind values in run time.

The Rational Build Forge API, with Java™ and Perl bindings, exposes most of the features available from the user interface (UI). With this, you can create extensions, such as dynamically constructing test projects from an external specification. For example, you can create a regression test project that consists of just the regression tests that are relevant to a particular release of code. This enables better separation of concerns, so that the responsibility of determining what tests to execute lies outside of Rational Build Forge, which is only responsible for preparing test environments and executing tests against those environments.

Within the test automation architecture, Rational Quality Manager delegates automation tasks to Rational Build Forge through its integration interface. With this integration, Rational Build Forge servers and projects are made visible in Rational Quality Manager's Lab Management subsystem. You can select to execute an automation project on one or more lab resources. Furthermore, complex test scenarios can be constructed in Rational Quality Manager that include a setup phase, an automated test suite execution phase, and a tear down phase. The test stand setup and tear down can be automation projects within Rational Build Forge. The Rational Quality Manager interface simply requests that Rational Build Forge execute an automation project and wait for the result before going on to the next step.

There may be several production instances of Rational Build Forge: one per product test area is expected.

UI automation tool: Rational Functional Tester

Your team can use IBM® Rational® Functional Tester to perform your UI automation testing. The types of UI applications that you can test with Rational Functional Tester include:

  • Web-based applications
  • Microsoft® .Net applications
  • Java applications (for example, SWT (Eclipse) or Swing)
  • Applications for 3270 (IBM® zSeries®) and 5250 (IBM® iSeries®)
  • Siebel applications
  • SAP applications
  • Flash applications

Some of the prominent Rational Functional Tester features are:

  • Lifecycle traceability:Rational Functional Tester provides IBM® Jazz™ integration to support collaborative application lifecycle management. Jazz Eclipse Client Version 2.0 integration provides Rational Functional Tester access to work items within Rational Team Concert and Rational Quality Manager. Additionally enhanced SCM (software configuration manager) integration with Rational Team Concert supports the management and sharing of test assets.
  • Keyword testing to streamline GUI test automation:Rational Functional Tester provides the ability to define and automate keywords that can be reused in both functional and manual tests. This promotes script reuse and enables manual testers to easily and selectively leverage the power of automation within manual test cycles.
  • Choice of test editing language between Java or Visual Basic .NET:Test script customization is mandatory in order to perform anything but the most basic tests. Rational Functional Tester gives you a choice of powerful, mainstream scripting languages to make this possible. Both options can be used with all the supported user interface technologies.
  • Support for Microsoft® Internet Explorer, Firefox, and Adobe® PDF:Functional testing support for HTML applications in Internet Explorer V8.0, Firefox V3.0 and Adobe PDF V7.0 and V8.0. You can also test HTML pages that are loaded in multiple tabs in Internet Explorer.

With Rational Functional Tester, you need to determine if Rational Functional Tester's record-playback capability is appropriate for your automation testing. If the application under test (AUT) is a short-term project (single release) or its GUI changes are limited across releases, then it may be appropriate to use record-playback. Otherwise, you should implement the GUI automation testing programmatically, and only use record-playback as an aid to scripting.

You should write your test scripts in terms of "what" needs to be tested in your AUT, and encapsulate "how" it is achieved within the UI in separate class methods. That is, your test scripts should consist of a set of method calls, where each method serves to implement a given task or function in the AUT. This isolation of operations in the UI is key to reducing maintenance cost across releases.

Lab resource management: Tivoli Application Dependency Manager and Tivoli Change and Configuration Management Database

You can use IBM® Tivoli® Application Dependency Discovery Manager (TADDM) to discover IT resources, and the dependencies between those resources. Use the TADDM agentless discovery technology to identify hosts, storage, and network components like routers, switches, and firewalls in the data center, as well as applications, middleware, and databases. TADDM discovers relationships between components, as well as how each is configured, and can identify how they change over time. TADDM can also discover resources defined in IBM or other Storage Resource Manager products, for example, IBM TotalStorage® Productivity Center.

TADDM has a Java API that enables integration with other software components. With this API, external components can request discoveries and retrieve information about discovered resources.

Within the test automation architecture, Rational Quality Manager integrates with TADDM to provide a richer way to populate resources in the Lab Management subsystem. You can populate Rational Quality Manager Lab Manager with computer resources discovered using TADDM, as well as from Tivoli Provisioning Manager and Rational Build Forge. Additionally, TADDM can automatically maintain a detailed inventory of all resources available for test, and simplify the sharing of high-value lab resources.

Each site will typically have a single production instance of TADDM. These site instances of TADDM can feed into an organization-wide CCMDB instance to provide an enterprise view of lab resources.


Benefits of IBM's test automation strategy

Leveraging the described architecture, test automation teams can reduce test expenses, improve the level and quality of testing, and reduce the effort required to deploy new test automation capabilities. The productivity gains delivered by using this architecture can enable and encourage your organization to test more often and more completely. In addition to eliminating the problems mentioned in the Common errors section, IBM's test automation strategy brings the following advantages to test teams:

  • Time factor:Many test automation efforts are custom developed, which leads to high development and maintenance costs. Using existing products as the basis for new test automation changes the focus to integration rather than development. Test automation teams do not have to reinvent the wheel. This will mean shorter development iteration times. The time before the product is ready to use will be reduced, and ultimately the product can be released to the market more quickly.
  • High skills availability:Using common test automation components and standard products means an increased availability of skilled test automation developers. Test automation development teams will be more aware of the standard products and their usage. On the other hand, test teams can focus more on test development, developing deeper and more comprehensive test scripts for testing the product. By freeing the manual testers from having to execute repetitive mundane tasks, test automation can enable them to focus on using their creativity, knowledge, and instincts to test more effectively.
  • Productivity:Many test teams spend most of their time developing automation tools instead of testing. By reusing existing products and by creating dedicated test automation development teams, testers can focus on testing. They need not worry about developing and maintaining test automation components. Again, this leads to better tested and stable products.
  • Low cost:Integration rather than development will reduce the development cycle. By using the same set of products and automation components across different test teams, development costs can be further reduced.
  • Standardization:One major problem that test teams face is non-integratable, non-sharable test automation components. Lack of standards used in test automation efforts is a major contributor to this problem. Using existing products and thinking about test automation in terms of reusable components leads to test automation tools that can be more easily integrated and reused.

What you have learned

This article has presented an architecture for test automation. Using this architecture, you are encouraged to design and develop automation tools that either directly use or leverage the capabilities of existing automation products. You can create tool designs that are extensible and can be reused by other test teams, and that ultimately promote standardization. This way you can more fully realize the power of test automation.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational, Tivoli, DevOps
ArticleID=513252
ArticleTitle=IBM's Test Automation Strategy: Build your test automation architecture around IBM Rational Quality Manager
publish-date=08312010