Multi-platform test automation in Registry Services

Registry Services currently supports 17 different platforms, such as Red Hat Enterprise Linux (RHEL) 5.0 Update 6 x86-64, Microsoft Windows Server 2008 Enterprise Edition x86-64, Linux on IBM® System z with Red Hat Enterprise Linux 5.8 and AIX® 6. It supports IBM DB2 9.7 and IBM DB2 Enterprise Server Edition 10.1. The supported application servers are IBM WebSphere® Application Server 8.0 Fix Pack 2, and IBM WebSphere Application Server 8.5.

CVT (Component Verification Tests) plays an important role to guarantee the quality of an application such as Registry Services, that may comprise hundreds or thousands of requirements, and must support such a variety of platforms.

A scenario where we must continue supporting previous versions of a specific middleware software, or different platforms, sharpen the probability that we introduce problems during maintenance or development. A fix or function that works fine, for example, in one version of the database middleware may have a disastrous outcome in a different version. Also, when adding a new function, we may be relying on an API that is available only for newer versions of the middleware, and will break clients that still use previous versions. When supporting different platforms, functions that somehow depend on command line interfaces, or scripts, must be thoroughly tested, otherwise we may end up having different behaviors in different operating systems like Windows and Linux. Regression testing intends to ensure that changes do not introduce new problems, or failures, in a software. Also, it ensures that previously fixed bugs are not reintroduced due to changes.

Registry Services uses an internal framework called CCVT (Continuous Component Verification Tests), which is based on STAF, to execute regression tests across a set of predefined test machines on a nightly basis. The Software Testing Automation Framework (STAF) is an open-source framework that contains a set of built-in services specially suitable for the construction of an automation solution. Services are reusable components that provide all the capability in STAF. Each STAF service provides a specific set of functionality such as logging, file system operations, security and process invocation, and defines a set of requests that it will accept. Additional services, such as Email Service, FTP Service and Timer Service, among many others, are not built-in, but can also be easily plugged into the framework.

STAX service is a special STAF service, or execution engine, implemented in Java, that allows the invocation of other STAF services through XML. CCVT contains a set of STAX and python scripts that allow developers and testers to easily add new test cases. In most cases, a new test case is comprised of a STAX script that invokes a JUnit test class. The framework contains utility functions that make sure the environment on each machine is prepared before each execution. For example, the CCVT framework ensures that the database is created, the application is installed with the latest production build, and the application server is ready for requests. The invocation of the JUnit class is performed by auxiliary scripts that collect test results (JUnit output and logs) and code coverage information.

CCVT is executed on a nightly basis, and runs all test cases across the regression machines in parallel. After all test cases are executed on all machines, CCVT generates an HTML report, accessible via an internal Web server, where developers and testers can verify the test results per machine. During the day, the execution of individual test cases, called incremental test, can also be performed through a request page where users can choose the test case, the environment and the images (production or uploaded images) to be used.

 

 

The report generated for each environment contains a list of test cases that were executed in that environment. For each testcase, you can access the log generated by STAX script and also a link to the compressed logs generated by the testcase and all processes that were created during the execution. In the report summary there is a link for the JUnit report that contains the results grouped by JUnit test classes. In the following figure, you can see an example of a report generated for the environment Windows2012DB2Websphere85:

 

One advantage of using STAF/STAX to build both the test cases and the infrastructure for continuous execution is the flexibility we have to grow the system as we need. Also, working in a project the adopts agile development, the capability of having all tests automated by STAF/STAX and integrated in a continuous system gives the ability to test new functions and fix defects in a very timely fashion.

 

• For a complete and up-to-date list of operating systems and middleware support, please refer to Operating System & Middleware Support
• General procedures for the Quality Assurance for Registry Services
• Find more information about STAF and STAX at the STAF Home Page
• Download and find documentation for JUnit testing framework
• A developerWorks article about CCVT: Test automation and continuous integration with STAF/STAX