IBM z/OS application continuous integration: Part 2. Continuous testing at all levels

Continuous integration improves software quality by continuously applying a quality control process. It has become an important practice of agile software development. This article describes how continuous integration concepts that are used in distributed development map to the IBM System z domain. The first article of this two-part series was about how to use Rational Team Concert to automate builds and deployment of mainframe applications. This article describes how to add automated unit tests, interface tests, and UI tests to the continuous integration process.


Zhang Hong Chen (, Enterprise Modernization Solution Architect, IBM

Tony (Chen Zhang Hong) is a Senior Software Engineer within the IBM Rational Enterprise Modernization organization. He is the development lead for continuous integration for IBM System z software, responsible for creation and validation.

Rosalind Radcliffe (, Enterprise Modernization Solution Architect, IBM

Author photoRosalind Radcliffe is an IBM Distinguished Engineer within the Rational Enterprise Modernization organization. She is the Enterprise Modernization Solution Architect responsible for Collaborative Lifecycle Management products for IBM System z and Power Systems. This includes Rational Team Concert support for standard mainframe development activities, as well as the support required within the IDEs to provide a complete development and test environment. Prior to Rational, she was in the Tivoli software division, responsible for the SOA management strategy for IBM. Rosalind is the Chair of NC TEC, the North Carolina IBM Academy of Technology Affiliate, and she is a member of the IBM Academy of Technology and a Master Inventor.

13 November 2012

Also available in Portuguese Spanish

Continuous quality feedback

The power of continuous integration comes from the immediate feedback to the team on the quality of the system being developed. The first level of validation is the automated build process, which can catch basic quality issues such as compilation errors and interface mismatches. When a build is successful, tests can be run on the executable file or the deployed application.

Modernized mainframe applications

Over the years, many mainframe shops have modernize their application user interfaces with Web 2.0 or mobile technology to improve UIs and increase access. The business logic running on the mainframe might have been turned into services that can be used easily by the front-end application. Figure 1 shows the configuration of a modernized mainframe application.

Figure 1. Modernized mainframe application
modernized mainframe application configuration

Levels of testing

As Figure 2 illustrates, testing can be performed at all levels with appropriate tools.

Figure 2. Levels of testing
Rational software by type of test

Unit tests

Unit tests are typically written and run by software developers to ensure that code meets its design and behaves as intended. Unit tests often test the smallest testable artifacts in the programming language (for example, a method in a Java class). Unit tests use technologies such as mock objects to isolate tests from the larger system so that the tests can run without having to integrate the entire system.

The zUnit unit testing framework is an adaptation of the JUnit framework for writing code to run repeatable, self-checking unit tests. The ideas and framework developed in JUnit for unit testing object-oriented code have been adapted by zUnit for testing Enterprise COBOL and PL/I code. The zUnit test framework, runner, and tool support are available with Rational Developer for Systems z from Version 8.5 onward.

Interface tests

Complex systems are usually composed of multiple components, which provide application functions through interfaces or APIs. For example, a bank account component provides a web service for opening an account that can be used by multiple front-end applications.

API tests or interface tests are to test the components based on the specification. The specifications are usually built on top of standard protocols, such as SOAP, XML, JMS, and MQ.

IBM® Rational® Integration Tester, powered by Green Hat technology, was created to address the problems inherent in testing systems at the interface level. It has an extensive range of enterprise standard capabilities, and it supports more than 70 protocols.

GUI tests

GUI tests check a running system through the user interfaces as end users would use the system. A tester normally performs GUI tests. Many of the GUI tests can be automated with tools, such as Rational Functional Tester.

Rational Functional Tester supports GUI testing for Java, Web, Web 2.0, and Microsoft Visual Studio .NET Windows Forms applications. It also support automation testing for IBM® zSeries® and iSeries® applications, which use 3270 and 5250 terminals.

The following sections show how to automate these tests and include them in the continuous integration process.

Automate zUnit tests

You can run zUnit tests by calling the Test Runner API as Figure 3 shows.

Figure 3. zUnit test framework
zUnit test framework

You can call the Test Runner from JCL, from a TSO command shell, or from an IBM® z/OS® UNIX System Services command shell. To automate the execution of zUnit in continuous integration, use the script execution feature of the Rational Team Concert enterprise build. There are multiple places that you can insert a custom script. Figure 4 shows an example of using a Post-Build Command Line of a dependency build to run zUnit tests.

Figure 4. Run zUnit from a post-build command line
Configuration page for Post-Build Command Line

The commands in Figure 4 write test results into the JKERS001.azures file in the USS file system. You can open the resulting file in Rational Developer for System z. It is also possible to publish the results into the continuous integration build result by using the junitResultPublisher Apache Ant task in Rational Team Concert.

zUnit test results are not compatible with the formats expected by the junitResultPublisher task, but with a simple XSL to do the transformation, it can be used. The Ant script that follows shows how to convert the result and publish it to the continuous integration build result. See Resources for where you can download the XSL.

Listing 1. Script to convert results and publish them to the CI build result
<target description="Publish zunit test results" name="publish_zunit">
<xslt in="${zunit.result}" out="${transfomresult}" style="${zunit2junit.xsl}" />
<junitResultPublisher buildResultUUID="${buildResultUUID}" 
    repositoryAddress="${repositoryAddress}" userId="${userId}" 
    passwordFile="${passwordFile}" filePath="${transfomresult}" 
    componentName="Mortgage" verbose="true" mayFailPattern="pattern1" 
    failOnError="false" />

The mayFailPattern allows control of whether test failures cause the entire build to fail. Many projects we have seen allow unit tests failures in their continuous integration. Figure 5 shows the zUnit test result after it is published to the continuous integration build result.

Figure 5. zUnit test results published in the build result
Failure detail section shows code

Automate interface tests

Automated interface tests created in IBM® Rational® Integration Tester can be exported to Rational® Quality Manager. After being exported, you can manage the execution of the tests and suites through the Rational Quality Manager centralized test management environment.

Configure the build and the integration

To run tests automatically for a build, integrations must be set up to use IBM® Rational Team Concert™ as a build provider and to synchronize build information with Rational Quality Manager. Figure 6 shows the configuration.

Figure 6. Configure Rational Team Concert as the build provider
Project dashboard, Manage Project Properties view

The default interval for build synchronization is 500 seconds. In Figure 7, the interval is changed to 60 seconds to run tests more promptly.

Figure 7. Configure build integration
Build integration framework details

Schedule the automated tests

The next step is to create an execution schedule in Rational Quality Manager.

An execution schedule contains steps that run sequentially at a scheduled time or when they are triggered by a build completion. When a build triggers an execution schedule, all test case results and test suite results are associated with the corresponding build record. Figure 8 shows an execution schedule that is triggered by the completion of build and runs the build verification test, BVT JKE Bank test suite, which contains two test cases.

Figure 8. Execution schedule
Details of the schedule and steps

Now the automated interface test for a build is in place. Whenever a build is completed, Rational Quality Manager will trigger the execution schedule that uses Rational Integration Tester agents to run the interface tests. Figure 9 shows the Rational Quality Manager Build Records view that lists build results with the build verification test (BVT) status.

Figure 9. Build results and BVT status
screen capture of details in table format

Key to abbreviations in Figure 10

RD&T: Rational Development and Testing Environment for System z
RFT: Rational Functional Tester
RIT: Rational Integration Tester
RQM: Rational Quality Manager
RTC: Rational Team Concert
See Resources for links to more information about each product.

Figure 10 shows how the products are put together to make an automated BVT. Rational Development and Test Environment for System z, as the name implies, is a manageable IBM® System z® development and test environment running on x86 machines. In the last section of this article, we explain how this Rational environment improves testing.

Figure 10. Automated BVT architecture
Flow between the products for executing a test

Automate GUI tests

You can create automated GUI tests by using Rational Functional Tester. The tool provides both a Record and Replay way and a programming way to create the GUI tests for testing web, Web 2.0, and desktop applications. Rational Functional Tester tests can be imported into Rational Quality Manager as test scripts and associated with new or existing test cases. After the automated GUI tests are in Rational Quality Manager, you can use the same method that you use to execute interface tests to execute GUI tests.

GUI tests usually take longer to execute than unit tests or interface tests. This is not only because GUI tests run the entire transaction but also because they simulate user actions with concepts such as think time. One best practice for continuous integration is that the build must be fast. This means both the build and the tests related to the build. Therefore, you need to assess carefully whether GUI tests should be part of the build tests for a project and which GUI tests you select to run for each build.

Flexible z/OS testing environment

z/OS tests often have resource constraints. Figure 11 shows a typical z/OS testing architecture with multiple teams and projects sharing the same test LPAR and database. This resource sharing model creates conflicts, impedes innovation, and slows code delivery. The coordination of environmental changes and releases causes bottlenecks, delays, and additional overhead. Shared test data is difficult to manage and can lead to over-testing or incorrect test results.

Figure 11. Typical z/OS testing architecture with shared resources
Diagram that shows the flow

With the Rational Development and Test Environment for System z, each team has its own isolated, manageable z/OS environment for development and testing, as shown in Figure 12. This provides an ideal environment for running continuous builds and tests.

Figure 12. z/OS testing architecture in a Rational Development and Test Environment
Separate resources


In this second and last article of this z/OS continuous integration series, we discussed the modernized mainframe application architecture and tests at different levels. We also showed how to automate these tests and run these tests in the continuous integration context and how Rational Development and Test Environment for System z changes the z/OS testing architecture by providing a flexible test environment.


Sample filezunit2junit.zip1KB



Get products and technologies



developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into Rational software on developerWorks

ArticleTitle=IBM z/OS application continuous integration: Part 2. Continuous testing at all levels