Last year I wrote the first blog in this series about our approach in getting started with adopting DevOps for z/OS. Lets take a closer look at some of the DevOps challenges for large operating system products and how we are addressing them.
How does operating system level code get to a DevOps continuous delivery paradigm? Most successful continuous delivery implementation examples are SaaS-type products. Operating system level code has significant differences from SaaS products/services:
- Not easily decomposable into smaller deliverables
- Customers value stability over having “latest and greatest feature set on a rapid delivery cycle"
- Multiple releases in support, each with its own service stream, complicates delivery
How to discourage yourself. Comparing “canonical” DevOps principles and examples to operating system level code is discouraging. We're so different from a SaaS product. This is z/OS, not Gmail! We can't decompose large, monolithic code into small enough customer deliverables. We have to maintain service streams on three releases in the field. We don't have any resources to create new automation. Our customers are very conservative about putting changes into production, and most of them won't accept continuous deliveries. There is so much control and process over what we can disclose to which customer. etc ...
The approach we used. Stop comparing ourselves to canonical SaaS continuous delivery products (like Gmail or Facebook) and finding ourselves wanting. Instead, work on improving internal processes across the organization and moving them toward continuous delivery through automation. Even if it's just to an internal test group, this would be a prerequisite to achieving continuous delivery anyway. Internal improvements are something that the development group controls and can show positive results early, even if small, to help get the right buy-in from the team.
How we moved forward. Thoroughly document existing delivery pipelines with an emphasis on identifying hand-offs and pain points. This information is often not fully documented in one place. This is across the organization, not just development. Formally documenting this has the following benefits:
- Requires the teams to think end to end about how they are delivering to customers today
- Provides opportunity to question “why are we doing it this way”
- Identifies pain-points and bottlenecks that can be addressed
- Identifies opportunities for automation and removing the waste out of the system
- Will help the team answer the question "What problem are we trying to solve?"
Make it graphical and clear. Pipeline documentation should be graphical and clear, which will make it easier to identify bottlenecks and pain points. The next two diagrams show parts of the z/OS Communications Server pipeline as an example. This is not the only way to do it, but it shows what’s meant by graphical and clear, and shows the value gained by doing it that way. We started with a very high level overview of the release cycle shown in diagram 1.
Then we zoomed in on specific phases for the more detailed view shown in diagram 2.
Look for bottlenecks and items to shift left. Graphical pipeline documentation should show hand-offs and bottlenecks. In the example in diagram 2, we saw that the build starts at 4:00pm but the SMP/E apply was scheduled for 8:00pm. Most builds completed in less than four hours so there was significant idle time. The result was we moved the build start time back to 5:00, which gave developers more opportunity to get fixes in that had “just missed” previous builds. Lots of small improvements like this can improve your development processes and efficiency.
Look for opportunities to automate. Pipeline documentation should also make clear what’s automated and what’s not. It should be easy to identify which processes are manual and should be candidates for automation. You need to assess the value of doing that automation so you can prioritize automation activities by the return on investment (ROI). Pick out the 1-2 highest value opportunities for automation and pursue those, and don’t try to do everything at once. It’s important to show positive results early to get buy-in from your development community.
Look for pain points to remove waste and inefficiencies. Pipeline documentation is a source for identifying pain points but not the only one. While developing the pipeline documentation, the developers should ask and document “what are the pain points”. But also a general question to the development, test, build, ID and support community on what their pain points are can be very beneficial. We were surprised with the low-hanging fruit that was identified which were process-related that yield better efficiencies for the development team. These small changes can go a long way toward getting buy-in with your development and test community. This is especially true for the ones that are rooted in “we’ve always done it this way”.
What about continuous delivery to customers? This goal can seem far-fetched for enterprise operating system level code but we looked at it differently. It’s true that our customers can be very conservative about what they will put into production. What if you were able to transform your current product delivery pipeline to continuously deliver code to SVT? Then you could think about other possibilities to give code earlier to our customers for early feedback. Our thinking needs to go beyond the traditional thinking of our current Beta and ESP programs. This plays into Design Thinking principles of sponsored users and early, regular feedback.
How can ENS afford to do DevOps with just our existing resources? It's important to acknowledge that DevOps is an investment and that creating pipeline workflows, automation and tooling work requires resources.
- We should collaborate with other z products where it is efficient and makes sense
- Treat DevOps as an investment that is weighted with other investments we make, e.g., new function development (including test)
- To make the necessary investments for DevOps in ENS, capacity for new function in releases under development is reduced. We go into this with our eyes open and make the right trade offs based on ROI
Conclusion. DevOps is a journey that we are on that involves the entire organization
- People – You need the right people to drive the culture change
- Process – Stop doing things that have no value (challenge the process)
- Technology – Standardize your tools and automation on current technology and limit the number of tools the team has to learn and use
- You start by understanding what you’re doing now and look for improvements, not by comparing yourself to the ideal case and getting discouraged over how far away you are from your goals
- Lots of small victories and improvements are the path to DevOps in mainframe operating systems
- Pipelines and improvements need to be developed from the bottom up (by the people doing the work) instead of the top down
- Using existing resource to make the investment for the future
If you are interested in our other blogs on our DevOps journey we have one on changing the culture and our process focus. We will continue to share our strategies and experiences with this blog series and welcome your feedback! You can also reach out to the authors Frank Varone (firstname.lastname@example.org) and Mike Fox (email@example.com).