Enterprise DevOps pattern

IBM Z® should be an integral part of the Enterprise DevOps strategy. DevOps is a way of working that drives a culture of continuous learning, continuous improvement, transparency, visibility, and trust. The goal is to deliver “Better Value, Sooner, Safer, Happier (BVSSH)”, as Jonathan Smart said. Large enterprises have to adopt new practices and develop a new culture to truly transform. The most effective organizations are embracing DevOps enterprise-wide, including their IBM Z applications alongside cloud native applications. According to 451 Research, “Digital transformation leaders, those executing on their plans, lead the way with 65% identifying improved quality of user experience as the top-sought DevOps outcome, compared to 36% of digital transformation laggards, which have no formal transformation plans. In other words, IBM Z is most effectively brought into the standard organizational DevOps transformation as another set of teams and platforms to integrate. Tools and assets exist that provide access to the breadth of IBM Z assets for handling within the pipeline irrespective of your pipeline tools of choice.

A single heterogeneous pipeline can be used to orchestrate the development, integration, and deployment of an application across multiple target platforms and environments with the right security rules in place. This single pipeline should handle all stages of the process from build tasks such as compilation and binds through to artifact packaging, deployment, and larger scale Integration testing. The pipeline can be used to enforce gating mechanisms at each stage, ensuring governance and quality.

DevOps for z/OS® applications

IBM Z now supports a wide variety of programming languages, toolchains, and development practices. This enables traditional mainframe or cloud or distributed developers to analyze and innovate for the mainframe using the languages and tools they are comfortable and familiar with. Applications can be written in popular languages like COBOL, Java and Node.js with automation scripts written in Python. They can be built, tested, and deployed into a z/OS environment or a Linux on z/OS environment as with any other platform in a continuously integrated pipeline using the same set of tools. Database and middleware changes can be stored as code and changed and integrated into the same pipeline.

Continuous Integration

The concept of continuous integration is about being able to develop in parallel and incrementally and continuously integrate code and test it, and not have siloed phases of manual code integration and testing. A modern Source Code Manager (SCM) forms the crux of any continuous integration pipeline. Having z/OS applications written using COBOL, PL/I, HLASM, JCL, Java, Python etc. managed within a modern SCM like Git enables developers to be able to branch and develop feature/function in parallel, and work in a process like what cloud developers or college graduates expect. z/OS applications can be tested in various stages with early test capabilities to reduce integration and production pipeline impacts.

A modern SCM like Git also brings configuration, automation and control of all technologies comprising an Enterprise application into the same SCM, enabling auditability and standardization of pipeline practices across the organization. z/OS applications using both traditional and new languages, the databases and middleware configurations can be first-class participants in a Git-based pipeline, with deployable binaries stored in an artifact repository. This enables applications to be built once and deployed many times, bringing in traceability of source code and deployed versions across multiple environments.

Continuous Test 

Continuous Integration requires new code to be automatically tested as part of the pipeline to ensure that the built product is free from regressions. For this to be of maximum benefit to a modern enterprise this testing should not only be automated, efficient, scalable and reliable it must also span both pre and post deployment of the built application.

Pre-Deployment testing should be executed as part of the build phase ideally in a stubbed or mocked environment to both eliminate potential defect vectors but also to expediate the execution of the tests. Use of a stubbed or mocked environment allows pre-deployment testing to focus on the updates to the code rather than any external impacts or changes. Modern languages such as Java or Python will often have unit testing frameworks available as either part of the core language or as an easily accessible framework. For z/OS applications there are often API calls into CICS, IMS or Databases that can also be stubbed and mocked either programmatically or automatically through a record and playback technique. These tools allow all languages and applications running on z/OS to have a set of automated unit tests that can be executed as part of the pipeline without needing to deploy the whole application. This allows introduced regressions to be isolated and fixed quickly.

Post-Deployment testing can be used to find regressions in elements of the application that cannot be isolated through a pre-deployment test. Code-based configuration as well as interactions with other elements of the enterprise application cannot be solely tested as a pre-deployment test, neither can test activities such as load, failover or SRE testing. The requirements of a test framework for post deployment testing move away from language specific requirements and move toward the need to integrate the external interfaces of the application together. This is not IBM Z specific as an enterprise application will likely scale across platforms so the test framework must be able to connect to the application regardless of platform. It is important to create this test automation suite that can address various levels of post deployment testing – from basic functional testing to end to end integration spanning multiple end points and applications. Basic functional testing may be scoped to a single element of the application running on a single platform; however, this must scale to encompass the entire application, both driving the system of engagement whilst also validating those changes in the system of record. The test framework must thus be able to access all elements of the enterprise application.

Post-Deployment testing requires a version of the application to be deployed, however it is important that the location of this deployment shouldn’t be fixed into a single location or platform. The set of automated tests should be flexible to bind to any environment either virtual or full scale. As the corpus of tests increases so must the ability for these tests to be executed from any pipeline at scale. This requires the test framework to scale horizontally so that new application deployments can be assured efficiently. It also requires the pipeline to be able to select the right set of tests from the corpus to be executed.

Most clients today will already have a set of post deployment tests that they are already running manually. Automating these tests not only speeds up application deployment but allows test teams to drive further quality improvements by deriving further tests.

Test Environments and Infrastructure automation

In today’s world, everybody is a developer. Whether you are developing applications, or automation scripts for infrastructure automation or pipeline orchestration – it is important for everything to be version controlled within a source control manager. Automation technologies and languages such as Ansible and Python can be used consistently across IBM Z and other platforms for automation and infrastructure provisioning within a pipeline.

On demand test environments are an integral part of continuous testing. Agile z/OS test environments can now be spun up and down on demand in OpenShift, in IBM public Cloud or on-prem based on demand. Ansible-based test automation also enables on demand middleware environments to be spun up and down for testing. The test environments and the infrastructure automation can all be a part of a CI pipeline or invoked via OpenShift Kubernetes operators. All these capabilities make sure that continuous integration is a reality for z/OS applications bringing z/OS development close to any other cloud native development experience.

Continuous Delivery

Continuous Delivery – Audited, controlled deployments are an important part of a CI/CD process. Urbancode Deploy and Velocity as well as scripted deployment frameworks based on Ansible are equally applicable to z/OS applications. The deployments can be managed from cloud and Kubernetes-based environments bringing IBM Z into the modern deployment ecosystem.

An Example:

The diagram below shows a sample application which involves a chatbot for an insurance application. This chatbot in turn talks to an API exposed from a CICS application.

The chatbot allows a customer to ask questions like the premium value, expiry dates, new deals etc. Let us assume that there is a business need to expose a new field into the chatbot – for example, the customer’s status. With an Enterprise DevOps pipeline in place, the COBOL program, table definitions can be checked out from Git using a modern IDE. The changes, including the addition of a new column can be made by the developer and committed into their own personal branch. A git push can automatically run a pipeline that implements and validates the Db2 change as well as the COBOL change – including running automated tests. Once verified, a pull request can run a Continuous Integration pipeline that can build the code from git, as well as the API changes and deploy the changes into a given environment, run automated tests with necessary quality gates in place. If a new CICS program is to be defined in this process, it can be done as code in Git and run as part of the same Enterprise pipeline.