![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
In this paper, I will review two implementations of IBM Rational ClearCase and the integration of ClearCase z/OS Extensions (z/OS is one of the most-used operating systems for the System z mainframe). I will also describe a data-driven approach to build management for the z/OS artifacts contained within the ClearCase repository. Before describing this approach I will first describe the business problems that ClearCase and the ClearCase z/OS Extensions solved for these two clients.
Both clients' business software had evolved over time -- by design or through acquisitions -- into multiplatform environments, meaning parts of their business applications ran under Windows, Unix, System z ( z/OS), and System i, and most all of these applications communicated with each other either by feeding information to one another or as an interface to the end user. This is a condition common in mature IT systems, and it is known as "cross-platform dependency." Configuration management solutions must be able to coordinate these types of applications across platforms.
Each platform solves a true business need and performs its tasks as directed, so the solution is not to consolidate applications onto one platform. Rather, the goal is to understand a business's working technology, how it is evolving in the modern work place, and how the business uses this technology.
In both business scenarios, each platform had a different process in place for the migration of code to through its lifecycle and onto its production environment, and each had its own configuration management systems in place to aide with this migration. The coordination of these different configuration management systems is difficult at best, and this was true in both of these business scenarios.
It is as fact of life that mature IT systems exist on multiple hardware and software platforms, and that business software continues to be developed on each unique platform. What are the reasons for this complicated state of affairs?
The first reason has to do with a mantra from about fifteen years ago: "The mainframe is dead." This theme throughout the IT industry led to many efforts to move software off the mainframe onto other platforms, including UNIX, System i (formerly i Series), and Windows. Certainly, the popularity of these platforms was growing naturally without "The mainframe is dead" movement, but this movement did contribute to the greater utilization of other platforms. This led to new software being developed on multiple platforms leaving, for a time, the tried-and-true mainframe applications in a state of maintenance only.
The second reason was the dot.com boom, which both nurtured and lived off of the maturing of the Internet. At the beginning, everyone wanted their bit of Web real-estate, from Fortune 500 companies to the mom and pop businesses in our local communities. Websites at first seemed to be a whim, but soon businesses realized how effective the Internet could be as a way to interact with customers, business partners, and internal users. Our on demand world had been born. This led to the rapid development of new software and tools that could utilize the Internet and the platforms that supported it.
The advance of the Internet and the way information was being delivered led to the advances in the architecture with the existing platforms, such as the mainframe and the AS400. And with the majority of the raw data still residing on these platforms, the need to interact with other platforms became paramount.
This rebirth and birth of technologies on all platforms has led us to this multiplatform reality; each platform has it strength, and businesses need to exploit each platform to its maximum advantage.
Configuration management solutions
As the above software development dynamic was evolving, so too were configuration management (CM) solutions. In fact, each CM solution evolved completely independent of others, according to the needs of its unique platform and the perceived way software was being developed and deployed on that platform.
Many companies created home-grown CM tools to meet their own specific needs. Software vendors also responded with a plethora of CM solutions, most with a niche in a single platform. Businesses responded by selecting one of these solutions or creating their own to meet the needs of the moment for their current architecture.
Meanwhile, either by acquisitions or responding to business needs, companies were busy adding platforms to their software architecture. This meant that companies inherited additional CM tools, needed to acquire an additional tool to meet the needs of the new platform, or had to customize their home grown system to meet their new requirements. So as business grew across platforms, so did the number of configuration management solutions that are part of company's software portfolios.
Naturally, many companies experienced a crisis of multiple CM solutions. In some cases not only are there different CM tools for each platform, but even multiple CM tools for a single platform, combinations of vendor packages, and home grown solutions. This particular phenomenon usually occurred after acquisition of other companies and when the cost of migration and retraining of development teams was perceived as too great.
In addition, some companies also faced a customization factor with their CM tooling. For example, a vendor package had been selected and the CM administrators -- in response to both business drivers and usability issues -- customized and enhanced the solution to their specific usage model over time. This ties the company and staff resources to the unique tooling for the foreseeable future. Yet, as staff attrition occurs and as the business model accelerates, the tool cannot be modified to keep pace with the business needs. Development teams tied to the CM tool cannot keep up with the changing business landscape.
So, those are the most common historical reasons for development teams following different processes, with cross-platform dependencies. Now, add today's IT needs -- e.g., compliance mandates such as Sarbanes-Oxley, governance, and management of geographically distributed development teams -- and we have a very complicated software development process as well as a complex business model to navigate.
As noted earlier, I will describe two actual business scenarios using complex CM solutions for managing IBM Z/OS artifacts.
Example 1: Native Transport
This first example comes from the transportation industry; I will refer to this company as Native Transport. Their current configuration management system was in-house designed. Although it had served them well for years, compliance issues had arisen pertaining specifically to Sarbanes Oxley. Their home-grown system did not actively include version control, so they could not accurately reproduce artifacts that had run against their production system. They also could not accurately track the deployment of their artifacts through their development life cycle. Their business applications were also evolving into this cross platform architecture.
There were also environment changes that were also a factor, after acquiring another transportation company south of the US border that had is own development staff, the desire was to outsource or near-shore some of their System z software development. This meant that development would occur at two separate locations and they needed to maintain the same process in both locations, and coordinate the releases of these cross platform applications.
Example 2: Department C
The second example comes from state government, which I will refer to as Department C. Their current configuration management system was a hybrid, consisting of a third party vendor product as the core, with customized REXX exec's, Clist, and ISPF panels as an interface. Although with this customized interface and the third party product gave them version control and a way to track what had been deployed, the architecture had been created many years ago. The employees that created and maintained this configuration management system had long since left the state agency.
In addition, recent legislative changes to the state's retirement policy meant that they would be losing close to a third of their IT workforce in the next two years. This left the state in a poor position for two reasons. One, due to the customizations made to the third party vendor product they could not apply regular maintenance; therefore they were running an unsupported release of CA-Librarian. Two, any knowledge about the current system in the existing workforce would soon be lost because those workers were steadily retiring.
The state's development environment was also changing. Created many years ago, the architecture needed to be modified to reflect how the state needs to develop their software now and into the future. The existing architecture was fixed to a serial -- what IBM Rational might call "waterfall" -- development process. They used paperwork and project management to prevent developers from overlap and code regression to their applications.
Their State legislation actually drives the majority of their projects, which means that how the state agency actually runs its business changes as laws change. And legislation changes from one new administration to the next, as well as within the governing years of any given administrations. When I was involved with Department C, new legislative changes regarding compliance had recently been mandated, and the deadlines that had passed. So their application development teams had make large scale changes at varying times to the entire system.
The problem was, their technical architecture was preventing them from developing their applications at a rate that could meet these statutory requirements. Plus, the effort to maintain their current configuration system was too costly -- i.e., they did not have the skill sets needed to modify their current configuration management architecture for the long term.
Introduction to IBM Rational ClearCase z/OS extensions
Both of these business scenarios needed a way to manage their build, compile, and deploy process for their z/OS artifacts. The z/OS Extensions feature of ClearCase offers a way to interface with the mainframe while utilizing ClearCase as the single repository for both the mainframe and distributed artifacts. This allows for a single point of control and a single process of development for both platforms.
Describing the single repository approach could be an article on its own. Instead, I will simply describe how build management can be obtained by utilizing native ClearCase with the z/OS extension feature.
The architecture is simple. A z/OS agent or Started task runs on the target z/OS system, and it communicates with the ClearCase repository via TCP/IP and a port address. The ClearCase host system also has an agent that points to the target z/OS system. The z/OS artifacts are stored in the ClearCase repository called a VOB (versioned object base).
The build and deployment interface between the ClearCase host and the target z/OS system is a combination of PERL, Remote build commands, and PJCL.1 The combination of these three components can essentially run any mainframe job or utility from the centralized location of the ClearCase repository.
PERL is the driving language it controls the build or deployment flow, checks return codes and based on the logic executes through a designed build and deployment path. The PJCL is the actual sequence of JCL steps that get sent to the ClearCase agent that then runs under the umbrella of the started task on the target z system. The remote build commands actually do the sending of the source and the PJCL. This setup is just like many other configuration management systems; with CA-Endevor this would be comparable to Processors, and with Serena's Changeman this would be comparable to ISPF skeletons.
What do we mean by a data-driven approach?
A data-driven approach to configuration management utilizes metadata within the repository that is associated to the individual source module, so in the case of z/OS artifacts these would be build components. A build component would be what language is the source module, does it include components such as DB2, CICS, IMS, MQseries, etc. This type of information is stored in ClearCase element-level attributes -- by "element," I mean the source module. With this information about the element stored, the build engine just queries the data and can derive the proper PJCL to build the element dynamically.
In addition to storing metadata about the element and how to build it, ClearCase project level attributes are also stored; these are similar to element level attributes, except that these relate to an entire project and the z/OS infrastructure. Depending on the infrastructure, these attributes would be related to syslib concatenation relating to copybooks and load modules, DB2 subsystem, CICS regions, IMS regions, etc. -- anything that is related to the deployment of artifacts to the proper z/OS environment. For example, in a simple lifecycle based on Test, QA, and Production environments, the project level attributes would have stored the metadata to know each of the target libraries, CICS regions, DB2 subsystems, etc., all needed when building or moving to any of above environments.
ClearCase environment variables are also used in this approach. ClearCase environment variables store information about the designed life cycle. For example, if I am building in the Test environment, the ClearCase variables can be queried to know where the next mapped environment is located, for either compilation or deployment, depending on methodology.
With this approach the developers do not need to concern themselves with how to build or how to deploy; the configuration management system (in this example, ClearCase) has the intelligence built into the build engine. It knows where the developers are within the lifecycle, knows what is being built and where, and how to deploy it in to the proper z/OS environments.
The migration or assignment of the element-level attributes is not complicated. If the migration is from an existing configuration management tool -- whether it is a home grown tool or third party tool -- the data should be available within the tooling, usually via a simple report. As part of the migration, the element level attributes can be assigned as the source files are imported into the ClearCase repository. In the case of the two implementations described earlier -- Native Transport and Department C -- the information was delivered in spreadsheet form, then read in to the import process as a text file, and the commands were generated to assign the attributes.
If there are no metadata or element level build attributes as part of the existing system, an inventory analysis must be done, usually on the mainframe, using the amblist utility. From that output the components can be determined.
The design of the ClearCase architecture relates to the current z/OS infrastructure or current life cycle design. It can be architected within ClearCase, either as a one-to-one relationship identical to what exists, or modified to suit client needs. In the case of Native Transport, the ClearCase architecture was implemented as a one-to-one relationship to their current z/OS architecture. In our Department C state government example, it needed to be redesigned to provide for parallelism in their development process.
Once the design is laid out, and as projects begin, the ClearCase administrator can enter the project attributes, essentially defining the migration path of the individual project as it relates to the z/OS supporting infrastructure and lifecycle. The lifecycle can be defined in a serial fashion, meaning every project follows the same lifecycle, or that projects can have the flexibility to point to different supporting environment infrastructures based on requirements of the individual projects.
The following diagrams will illustrate the processes of 1) assigning element-level attributes, 2) assigning project level attributes, and 3) determining the Build Engine and how it interacts with ClearCase as it relates to the Z/OS infrastructure. A small summary section will follow each diagram.
Assigning element-level attributes
In our Department C example, the migration process used a file that held all the element names and their associated attribute values from a DB2 table. The data was then imported into an Excel spreadsheet, then converted to a CSV file that was tab delimited. It then was read into a PERL script import_attr.pl that used the data to create a batch file of Cleartool commands to assign to the elements their attribute values.
This input file, although a DB2 table in this case, could also be a report that can be generated by other mainframe CM tools, such as CA-Endevor or Serena's ChangeMan. Most all CM tools -- even the home-grown ones -- will have this functionality. This import_attr.pl script will read in the file and create the Cleartool commands as long as it is tab delimited.
In our Native Transport example this task was accomplished by a REXX exec that analyzed the data on the mainframe and created a similar batch file of the Cleartool commands, which was then ported to the ClearCase host and run as a batch file.
The import_attr.pl script would be delivered as part of field service to assist clients with the migration process.
Assigning project-level attributes
The SetZCCEnvAttr.pl utility is used to update a ZCCENV attribute that is defined at the project level. Its function is to define and map the z/OS infrastructure and lifecycle on the mainframe. This data is then queried by the build process and used to determine where and how the project will migrate through the life cycle. If a change is needed or a different migration path is necessary as development proceeds, the ClearCase administrator need only change the values with the ZCCENV definition for the project utilizing the SetZCCEnvAttr.pl script. The build engine will use the new values and continue on in the new lifecycle, or it will move to the new z/OS infrastructure for testing and QA.
The SetZCCEnvAttr.pl utility is delivered as part of field service assisting clients with this process. This utility was used in both the Native Transport and Department C implementations.
In the above usage model, the build script first extracts element-level build properties. Then it determines where in the lifecycle it will be building into by extracting project-level attributes assigned when the project was created. These attributes include the proper target load libraries, SYSLIB concatenations, DB2 subsystem, etc. The lifecycle is also queried using ClearCase environment variables to determine where the build is taking place within the lifecycle.
The next phase determines the actual PJCL needed to perform the build in the proper stage in the lifecycle by creating PJCL and substituting all the proper metadata that where part of the queries from the repository. It creates the PJCL necessary and executes a build a step at a time. Progress regarding the build process is displayed to the developer via STDOUT. At the end of the build process, the PJCL that was created and run along with all output is stored in the user view where the build was initiated. If errors occurred, all the data necessary to determine the issues are stored in the view.
This build engine is delivered as part of field service to assist clients with this process. The only true modification is the task of adapting each client's unique JCL and library standards into the build engine. This build engine was part of both Native Transport and Department C implementations.
For Native Transport, the ClearCase Z/OS Extensions solution positioned them to be able to exploit other rational solutions like ClearCase Multisite. Native Transport was planning to off-shore some of their z/OS development while maintaining their process of development in both locations. It also allowed for very low administration with regard to their build process, because once a build engine is in place, the only maintenance is providing for new target libraries should the z/OS infrastructure be changed, such as an upgrade to the compilers occurs, in which case new libraries would also need to be added. Should a new element type be added to the build process, the build engine would only need to be modified if a new development language was to be added to their z/OS development process.
Both implementations saw additional value by allowing z/OS development teams to utilize GUI tools to assist in their development process, utilizing ClearCase diff/merge utilities and code regression testing as they moved through their lifecycle.
The use of Project level attributes also allowed flexibility within the lifecycle. Although in Native Transport's case the desire was to have a fixed life cycle, the project level attributes will allow them to add z/OS infrastructure to their lifecycle if desired at some point in the future -- for example, if they need to add QA capabilities, they could easily change their migration path dynamically with project level attributes. The only restriction would be in creating the z/OS infrastructure to support it. Department C saw the same value as Native transport, with the additional benefit of utilizing ClearCase for parallel development, along with the flexibility to change their migration path dynamically as warranted.
Change is not viewed lightly within z/OS environments, but ClearCase makes the process manageable. The best way to understand the new process is to think of ClearCase as an interface to z/OS. Just as Interactive System Productivity Facility (ISPF) is an interface, ClearCase serves as a delivery mechanism to the environment. It allows the z/OS development teams the benefits of ClearCase and the use of GUI tooling to assist them with their processes. The single repository solution allows for all development to use the same processes for both distributed and z/OS platforms. This helps consolidate multiple CM solutions that may be in place, and lessons the complexity of development in today's mainframe-based environments.
1 PJCL stands for "Pseudo Job Control Language," which is proprietary to ClearCase z/OS Extensions. It is similar to normal Job Control Language with some additional features specific to ClearCase. It is also similar to other proprietary languages with other CM solutions like CA-Endevor's SCL Software Control Language, and Serana's Change Man ISPF skeletons.
Brandt Onorato has been a z/OS systems programmer for sixteen years, focused on configuration management and data security. He joined the IBM Rational organization two years ago. He has created technical manuals, instructional technical classes, executive briefs, and functional specifications during his career.
Table of contents
Local resources
Tags
Use the slider bar to see more or fewer tags.
For articles in technology zones (such as Java technology, Linux, Open source, XML), Popular tags shows the top tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), Popular tags shows the top tags for just that product zone.
For articles in technology zones (such as Java technology, Linux, Open source, XML), My tags shows your tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), My tags shows your tags for just that product zone.
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
