DevOps fundamentally transforms the way that development and operations professionals work together by promoting a pragmatic set of principles and practices that help to deliver software and systems that are reliable, secure, and free from defects that might potentially threaten the success of the entire enterprise. The main focus of DevOps is to improve communication and collaboration, two essential qualities required for any team to work effectively together. Most organizations recognize that there are other stakeholders who can also benefit from the DevOps approach, including information security (InfoSec), quality assurance (QA) and testing. The scope of application lifecycle management (ALM) is broad and ranges from the service strategy and design envisioned by the product owner to the help desk calls when there is an incident that impacts the successful delivery of services to the customer. This article describes how to apply the DevOps principles and practices to the entire ALM.
Effects of changes on software reliability, security, and quality
Software and systems development is a complex endeavor that typically involves many stakeholders and can require considerable time and resources to complete. Application lifecycle management (ALM) provides the structure and processes to guide this effort from start to finish. The ALM helps to direct the behaviors of each of the resources that contribute to the system implementation, from the product owner who helps to define the requirements to the operations professionals who ensure that the services are continuously available.
Large enterprise systems development can be adversely effected by a defect introduced at any step in the software or systems development effort. Too often, organizational silos cause a lack of communication and collaboration and this problem can result in serious consequences. The principles of DevOps, which have been successfully applied to communication and collaboration between development and operations, can improve the way that each stakeholder within the ALM interacts with the other members of the team. The self-organizing, cross-functional team that has become widely associated with successful agile development projects is an essential component. These high performance teams take on a wider scope that makes the entire software and system development effort successful. The key to using DevOps best practices throughout the ALM is to understand the linkages between each function of the lifecycle.
ALM functions and linkages between functions
Many points within the ALM require communication and collaboration across multiple stakeholders. Some of these are phases in the lifecycle and others are simply points of intersection between functions that the team must understand and manage:
- Requirements definition
- Systems design
- Modeling dependencies
- Middleware management
- Database experts
- QA and testing
- Knowledge management
Any software or systems development effort must start with a clear specification of what is to be built.
Every stakeholder in the ALM needs to understand requirements. Requirements change over time, often because:
- You understand more about what you want to build.
- The market changes.
Changes to requirements as you learn more
With each iteration, business experts often refine and calibrate their requirements as they better understand the capabilities (or limitations) of the technology and systems. Obviously, changes to requirements need to be tracked and controlled. Many organizations use version control tools to keep a complete record of changes to requirements and to monitor the approvals for changes in requirements. It is also common to implement traceability between requirements and the test cases that are to be used to verify the functionality that is expected in the release. These are important aspects of tracking requirements, and they align with the DevOps principles of good communication and collaboration. Several IBM products support these functions of collaborative lifecycle management (CLM):
- IBM® Rational® Requirements Composer™ makes it easier to define, manage, and report on requirements in a lifecycle development project.
- IBM® Rational Team Concert™ helps you define and track work items.
- IBM® Rational Quality Manager, a web-based centralized test management environment, provides a collaborative solution for tracking and metrics reporting.
An even more dramatic example of a scenario that requires effective communication among the product owner, the development team, the test team, and the operations team, is the need to manage risk due to outside business pressures, including a competitor's rush to market.
Changes to requirements due to market forces
A release that is rushed into production often creates unacceptable risk. DevOps focuses on the need to manage deployment-related risk. It also creates tools and processes to provide the agility to make continuous delivery a competitive advantage. Sometimes, a missed deadline incurs a greater risk than a rushed deployment. As a deployment engineer, I know that sometimes, the business need to deliver the application to market faster is so great that a missed deadline puts the future of the company into jeopardy. To choose the best course of action for the company as a whole, technology experts and business experts need to collaborate and communicate effectively.
The value of delivering a beta version of a system to market ahead of the competition sometimes far outweighs the risk of a system glitch or outage. In these cases, you need to completely understand the risk and communicate it to all stakeholders. Generally, the risk can be mitigated with extra on-call support services, which the ITIL V3 framework calls early life support.
Take for example a use case in which a beta version of a new trading platform is delivered to a market of only two customers, who are delighted to try out the new beta platform. With extra support, the glitches are worked out. Soon after that, the sales team is flooded with new orders, and the company wins the market from the closest competitor.
In this example, the operations team understands the need to deploy a less-than- perfect system. The team delivers the fully-tested system to the rest of the customers a month later. In this case, the team agrees to deliver a beta system to production because the business professionals express a sense of urgency and recognize the competitive advantage of being first to market.
Systems design requires expertise from many different stakeholders. In the design phase, DevOps principles are essential for a project's success.
- The developers often understand the technology, but might not fully appreciate how the infrastructure can be scaled to meet peak performance requirements.
- The architecture team can help development and operations understand the capabilities and risks of a particular technology framework.
- The business users can help the technology experts understand the potential peak usage demands and the potential for long-term growth and performance requirements.
With the cross-functional view, the systems infrastructure team can create a pragmatic systems design that gets the product to market quickly, within budget, and with the ability to meet the long-term capacity demands. During the system design phase, the team needs to understand and document all of the dependencies, in terms of the component interfaces and with regard to runtime dependencies.
Software systems today have many excellent capabilities. They also involve significant complexities. The technology experts who develop, integrate, and customize these systems often possess a vast amount of expertise and in-depth knowledge of the technology and its dependencies. Too often these specialists are involved with the project only for a short time and then move on to the next project without leaving behind enough information on the dependencies and interfaces required to maintain the system.
Even though the software runs well when it is first deployed, if future changes are required, it's challenging to update the system without adequate knowledge of the dependencies. To prevent problems in the future, you need to communication and document a comprehensive model that shows the interfaces required and all of the dependencies. Because this information can quickly become obsolete and unreliable, systems need to be designed so that their dependencies can be automatically discovered and well-understood.
The architecture team can help the development team derive a methodology for documenting dependencies. For example, an XML file can be used to discover and communicate dependencies. To design complex systems that can be easily maintained, the development team can rely on the expertise of other key stakeholders including architecture, information security, QA, testing, and the specialists who maintain the middleware that is essential for the system to function successfully. Consider trying IBM® Rational® Rhapsody®, a model-driven development environment, for systems engineers and software developers who create real-time or embedded systems and software.
Middleware is often described as the plumbing that exists between the operating system and the application, or between the database and the application. Middleware is considered essential but invisible and can include services to support a message queue or a cache memory manager. Middleware is often managed by a dedicated team and requires specialized expertise. The middleware experts need to collaborate with the developers, operations team, and architecture groups, at a minimum. In many cases, it's helpful to also consult the database experts. IBM products that help manage the middleware are:
- IBM® WebSphere® MQ, an industry-leading messaging solution, integrates applications and Web services on almost any platform and helps you deploy a full-function enterprise service bus (ESB) as the connectivity layer for your service-oriented architecture (SOA).
- IBM® WebSphere® Application Server delivers the secure, scalable, resilient application infrastructure you need for SOA.
The database is a critical technology within any system and, too often, the database administrators are not included in the DevOps cross-functional team. Because the database can be a single point of failure for a system, changes to the database need to be understood and version-controlled, as is the case with any other configuration item. The DevOps approach to improved communication and collaboration recommends that database administrators be included, for the team to be successful. To plan application changes with an understanding of related database changes, the development team needs to collaborate with the database administrators. Quality and testing also need to be considered throughout the entire application lifecycle.
QA and testing
The QA and testing team needs to be an active member of the cross-functional team throughout the entire software and systems development lifecycle. Every stakeholder within the ALM can benefit from the decision to include QA and testing strategies throughout the early stages of requirements gathering, design and development, and functional and user acceptance testing. QA and testing also plays a key role in the maintenance of the system by helping to manage risk related to bug fixes and ongoing maintenance. Several IBM products support the QA and testing cycle:
- IBM® Rational® Quality Manager, a web-based centralized test management environment, provides a collaborative solution for tracking and metrics reporting.
- IBM® Rational Integration Tester is part of the IBM® Rational® Test Workbench, which combines the capabilities of functional, performance, regression, load, and integration testing automated testing of highly complex applications.
Many specialists with extensive knowledge and expertise are involved throughout the application lifecycle. The DevOps approach is to apply excellent communication and collaboration so that stakeholders can work as a cross-functional and highly effective team. It is important to capture the knowledge that each stakeholder possesses, so that the team can address and deal with incidents that often occur during the lifecycle of a system.
Continuous integration and delivery
Throughout the application lifecycle, technology experts need to constantly work to understand what the system is required to do and how it operates currently. Continuous integration and continuous delivery provide the fundamental techniques to integrate changes frequently, as they become available. It is much easier to integrate a small change than to integrate a group of changes. It is also much easier to back out a small change if it does not work than to determine why a group of changes breaks the system. Continuous integration works because it reduces complexity and makes it simpler to identify the root cause of integration issues. Problems are much easier to identify and fix if you have automated the integration process and if you perform this task frequently.
Continuous delivery operates with the same approach. To help a team that is struggling with release and deployment, get them to deliver smaller bundles of code, more often. Teams that release twice a week get better at the deployment task than teams who only release every other month. Even if you release the entire system each time, a smaller set of changes is inherently less risky. It is much easier to address issues if they do come up. Several products support iterative development:
- IBM Tivoli™ Provisioning Manager enables a dynamic infrastructure. It automates the management of physical servers, virtual servers, software, storage, and networks.
- IBM Tivoli Provisioning Manager for OS Deployment automates remote deployment of operating systems.
- IBM UrbanCode Deploy™ offers an automation deployment framework that reduces deployment errors and improves efficiency, correctness, and traceability.
The big payoff from continuous integration and continuous deployment from an ALM perspective is the ability to provide the latest code for review by all stakeholders. Milestone releases or even test releases from a scrum iteration can make it possible to verify requirements, to facilitate continuous testing, to update and improve systems design, and to empower each stakeholder to participate actively in the application lifecycle. These changes help to continuously improve the process.
Retrospectives in the ALM
Many organizations miss business opportunities because they fail to discuss their successes and failures in an open and honest way. The retrospective is a powerful tool that makes it possible for all of the participants in the ALM to discuss openly and directly what went well and what can be improved. The DevOps approach to foster open communication among stakeholders makes it imperative to hold retrospectives early and often. This practice develops a culture of honest and open communication.
DevOps is not just for development and operations anymore. All of the teams who participate in the application lifecycle possess important information that can help the cross-functional team be more effective and become a more highly performing team. Increased collaboration and communication between the development team and the operations team improves both productivity and reliability. Extend these DevOps principles and practices to the entire application lifecycle to transform your entire organization into a well-coordinated, high-performance team capable of even greater success.
- Improve your skills. Check the Rational training and certification catalog, which includes many types of courses on a wide range of topics. You can take some of them anywhere, anytime, and many of the Getting Started ones are free.
- Check out the other articles in this DevOps best practices series (developerWorks, 2013)
- What is ITIL? Learn more on the official ITIL site, the IT Service Management Forum (itSMF) site, and the itSMF (US-focused) site.
- Read how to build quality in at the beginning in Out of the Crisis (W. Edwards Deming, MIT, 1986).
- Explore DevOps for configuration management in the book Configuration Management Best Practices: Practical Methods that Work in the Real World (Bob Aiello and Leslie Sachs, Addison-Wesley 2010)
- Learn about IBM Rational Solution for Agile ALM for agile software development adoption and automation. Check out the following resources:
- Recorded demos: Short recorded demos (< 5 minutes each) that describe the key features of the solution.
- Technical solution: The latest Agile ALM with Scrum Rational Team Concert process template with in-context Scrum practice guidance, role-based dashboards, customized plans and common tasks help teams get started.
- Technical solution enablement: Free scenario, demos and tutorials can help all Scrum roles on a team to learn how to quickly implement and use the IBM Rational Solution for Agile ALM.
- Agile enablement: Social learning community, including a learning roadmap, forums to asks questions or share ideas and experiences and an experts zone to find peers, SMEs and leaders.
- Learn more about agile development in the book Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise (Addison-Wesley, 2011)
- Explore the Rational software area on developerWorks for technical resources, best practices, and information about Rational collaborative and integrated solutions for software and systems delivery.
- Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
- Attend a free developerWorks Live! briefing to get up-to-speed quickly on IBM products and tools, as well as IT industry trends.
- Watch developerWorks on-demand demos, ranging from product installation and setup demos for beginners to advanced functionality for experienced developers.
Get products and technologies
- Download a free trial version of Rational software.
- Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment.
- Check the Rational software forums to ask questions and participate in discussions.
- Get connected with your peers and keep up on the latest information in the Rational community.
- Ask and answer questions and increase your expertise when you get involved in the Rational forums, cafés, and wikis.
- Rate or review Rational software. It's quick and easy.
- Share your knowledge and help others who use Rational software by writing a developerWorks article. Find out what makes a good developerWorks article and how to proceed.
- Follow Rational software on Facebook, Twitter (@ibmrational), and YouTube, and add your comments and requests.