The need for DevOps
DevOps is both a movement and a methodology, based on the collaboration between developers and operations. Traditionally, IT has been organized around a specialization of knowledge. Roles have been imprisoned in silos that are oriented around specific components of the IT organization. To prosper in today's market, however, these silos must be bridged. As applications evolve to function in the cloud, IT organizations must reconsider how they develop, deploy, and manage them. Once upon a time, operations was at the end of the process, but now it must blend into the agile team so that the team can deploy and release more rapidly.
The agile methodology has revolutionized the software-creation process. Development organizations everywhere are being pushed relentlessly toward the agile methodology by the swirling winds of cloud computing, big data, social media, and the volatile nature of the market. This push toward agile is a reflection of the current business reality, aligning delivery with current business needs and goals. The process is accelerated by the turbo-charged feedback loop engendered by big data. Big data, in combination with social media, means that information can be gathered quickly and application changes made within days—conceivably, within hours.
The agile methodology and cloud computing represent a significant shift in the way operations manages applications, as well. In the past, IT operations teams focused on servers, and application were secondary. Cloud computing is an application-centric operations model and has changed the operations landscape.
DevOps, the methodology, is designed to integrate development, operations, and quality assurance (QA) into a collaborative delivery unit that improves product quality, enhances system performance, and improves time to market. It improves release management by standardizing environments, allowing code to move between environments without the impediments of the traditional siloed approach. DevOps allows organizations to bridge the silos and at the same time provides a means to incorporate full life-cycle governance at the project level. DevOps allows development, QA, and operations to come together through the entire process, working toward a common goal of rapid, high-quality delivery through collaboration.
Building a DevOps environment: the collaboration revolution
So, how do you begin this beneficial journey? Well, let's put a system together for an enterprise we will call Acme Knowledge Services. Acme Knowledge Services collects information, organizes it, and provides it to its subscribers. The company is moving away from the traditional IT team working structure and becoming a DevOps shop using agile Scrum, a process that requires an entirely new way of organizing projects and teams.
Resisting matrix thinking and waterfall habits
The first hurdle in the creation of a DevOps team is matrix thinking and waterfall habits. Traditional IT organizations are structured on a matrix model, with functional managers and project streams. An agile Scrum team is not a matrixed team, and a Scrum master is not a project manager. Agile teams must function as a unit at all times, assisting each other and partnering through the entire project. The Scrum master's job is to make sure that everyone has what he or she needs, continuously, and that the team is meeting its goals.
Waterfall habits are so ingrained in most developers and systems administrators that they no longer see waterfall as a methodology. Rather, they see it as the way. This mindset must be uprooted if the agile team is to succeed. Lack of participant buy-in is a huge point of failure.
The repository challenge
To work in the collaborative environment that DevOps requires, tools must be put in place to facilitate the new application creation, delivery, and deployment process, emphasizing automation. The technology you select at every step determines your level of ultimate success. Everything today is all about collaboration. A structure built on open source is one way to facilitate DevOps. Using open source allows you to more easily adapt to change. When it comes to the cloud, open source can help you keep your provider options open.
One of the most important technology decisions is the repository. Traditional version control systems (VCSs) don't provide the best support for the agile approach, because agile developers must constantly integrate their code into the baseline, which requires a complete project baseline build plus automated testing to validate the newly developed code. Infrastructure as code also relies on the repository, so any repository chosen must serve the needs of operations, as well. Code-driven automation means that the right repository is critical to the overall success of the project.
The concept of a repository began as a central location that every developer would use to check in and check out code, much like a library:
Figure 1. Centralized VCS
Projects today require repositories that give users the ability to split features into smaller parts and allow both development and testing in isolation, while providing the means to merge changes back into the main branch. Merging and branching are beyond the scope of the traditional repository:
Figure 2. Distributed VCS
Acme Knowledge Services decided to go with the Git repository.
An open source distributed VCS, Git manages projects of all sizes with accuracy, speed, and efficiency. The main difference between Git and other VCSs is the way Git organizes and manages data.
The traditional VCS stores data as deltas to a base file version, as in this example:
Figure 3. File check-in in a VCS
In contrast, Git captures a snapshot of the current structure each time a commit occurs, and then stores a reference to it. For files that haven't been touched since the last commit, Git stores only a link to the file most recently stored:
Figure 4. Git file check-in
Git is powerful, relying on simple commands. For example, to initialize your repository, use the following command:
To configure Git for the first time, run these commands:
git config --global user.name "John Doe" git config --global user.email "John.Doe@yourdomain.com"
The main drawback to Git adoption, especially for larger projects, is the perceived challenges related to administering a server for the Git repositories. No problem: Our intrepid team has found an application that sits on top of Git to harness its power and provide a user-friendly interface so you can simply follow the process flow provided for each task. It also has the advantage of easy integration with Atlassian JIRA, an issue-tracking product that Acme Knowledge Services already uses. JIRA is a popular bug-tracking, issue-tracking, and project management tool. Because the team was already using JIRA to track issues, Atlassian Stash was the logical choice.
Stash is the Git repository management solution designed for enterprise teams that allows everyone to easily collaborate based on the Git repositories without the learning curve. It simplifies repository management and provides a central location from which you can create, manage, and collaborate on your repository.
Using Stash, developers can readily find the most recent version of code as well as see all issues associated with a commit, while protecting the system by ensuring that a user's access level is appropriate for his or her role. The Stash interface makes working with the Git repository much simpler. For example, on the Files tab of Stash, you will find buttons, shown in Figure 5, that lead to an interactive process that guides you through these tasks:
- Click Clone to make a copy of a repository.
- Click Fork to create a new copy of the existing repository.
- Click Pull Request to submit your changes to the repository for review by your team before merging.
Figure 5. Stash buttons
In addition to Stash, Acme Knowledge Services added Atlassian Bamboo for building and testing, the logical piece to add to Stash and JIRA to enable continuous integration.
Bamboo and continuous integration
Continuous integration (CI) requires frequent code integration into a shared repository, perhaps several times a day, but at least once a day. Each time a developer checks in his or her code, an automated build is performed that verifies it. This CI and building allows teams to detect problems early. When issues do arise, the time it takes to track them down is minimized. CI won't stop bugs from occurring—that's just human nature—but you can detect them more rapidly and hunt them down more efficiently.
Another Atlassian product, Bamboo, is a CI server that automates release management for software applications. Acme Knowledge Services found that with JIRA, Stash, Bamboo, and the Atlassian wiki Confluence, it could follow all aspects of development, such as JIRA bugs, tasks and subtasks, developer commits, and approvals. An application's progress can now be observed from development to production.
The next step for Acme Knowledge Services was to design its infrastructure as code. Using Git, infrastructure code can be managed right along with the solution code.
Managing infrastructure as code
Infrastructure as code has become the foundation for an effective implementation of DevOps. Computing, storage, and networking are rapidly moving from physical to virtual environments. The management of these resources increasingly relies on software, making environment replication simpler and more easily repeatable. The ability to create infrastructure as code is one of the most important benefits of virtualization and cloud computing, allowing you to provision and manage entire application environments as well as infrastructure reliably on demand. You can build multiple versions of the various environments, making CI possible.
Infrastructure as code includes the entire product solution except for the developed software application itself, which means that you manage infrastructure setup just as you would application code development. You begin the process by selecting the right tools, and start developing the required solution, which you can then apply repeatedly. Infrastructure as code allows you to change the setup over time as the application develops. When it finally goes into production, it is a system that has been tested over time.
Infrastructure as code, properly implemented, ultimately means that an entire business can be reconstructed from a source code repository, a backup of your application data, and the required bare metal resources. Now that's a disaster recovery plan!
Acme Knowledge Services decided to use Opscode Chef to manage its move to infrastructure as code.
Configuring cloud services with Chef
An open source configuration management tool written in Ruby and Erlang, Chef works with Git and can be used to configure cloud services or simplify the tasks associated with configuring an enterprise's internal servers. The Chef systems and cloud infrastructure automation framework makes it easy to deploy servers and applications to any physical, virtual, or cloud location. You use cookbooks and recipes to tell the Chef client, which is installed on every node, how to configure each node in your organization.
Chef is state-based and requires that you write code to configure your infrastructure. Using Chef, you can create your entire infrastructure as code. The tool facilitates the creation of recipes for server configuration and application management. Chef recipes include descriptions of resources integral to the system's state—the running of services, installations, writing files, and related functionality—all of which ensure that the resources are correctly configured and the system's data points are automatically detected.
You can clone the Opscode cookbooks, which contain many valuable examples, by
using Stash or the
git command as follows:
git clone git://github.com/opscode/cookbooks
Choosing OpenStack and IBM SmartCloud
Things were working well at Acme Knowledge Services. Not only were daily operations agile and responsive, but customers were happy and business improved. So naturally, change being the only constant in life, management realized that they needed to move to the cloud.
Everyone in IT at Acme Knowledge Services was in agreement that whenever possible, open source was the way to go. So the team chose OpenStack as the cloud "operating system" of choice. OpenStack, an open source Infrastructure as a Service initiative, is used to create and manage virtual private servers in a cloud computing environment. Because it is not a proprietary cloud environment, using OpenStack reduces the risk of vendor lock-in.
After the decision was made to go with OpenStack, the team chose IBM SmartCloud® as their provider. OpenStack is the future, and IBM SmartCloud has embraced that reality. So, IBM SmartCloud became the system of choice, and IBM SmartCloud Orchestrator the capstone of the structure that Acme Knowledge Services built.
IBM SmartCloud Orchestrator
IBM SmartCloud Orchestrator adds runbook automation and a policy-based management engine to OpenStack. In addition, IBM SmartCloud Orchestrator can use the Chef recipes and cookbooks that Acme Knowledge Services had already created.
IBM's smart deployment methodologies and patterns of expertise would be an efficient fit for Acme. Patterns of expertise are XML-based definitions of infrastructure configurations essential to the provisioning and management of the system's computers, networking, storage, operating system, and middleware resources critical to specific application workloads. These infrastructure configuration definitions address the enterprise need for strict compliance and maintaining consistent production and service quality. Deploying application environments based on these definitions ensures compliance and consistency. Thus, the IBM SmartCloud Orchestrator role facilitates higher levels of system performance, essential system security, and all-round reliability. The tool is critical for system monitoring and cloud service management.
DevOps is a key component of the collaboration revolution. The foundation of collaboration is the efficacy with which cooperating forces and people are blended together to achieve goals. DevOps is both a movement and a methodology, designed to make working based on continuous integration and deployment easy. Building a stable culture of collaboration that is DevOps requires a rethinking of how you do development, testing, and operations. DevOps is a mindset: It flourishes in an atmosphere of interaction and interchange, because it is all about collaboration. A critical part of that mindset is the use of open source. If you think DevOps is all about operations, you've missed the boat. If you think agile is not about operations, you're on a sinking ship. The assets described here are an example of the disruptive, chaotic, but nonetheless connected and collaborative commercial environment.
- At the developerWorks DevOps zone discover expert how-to articles, insightful blogs, and free trial software. Learn how to apply key principles, best practices, IBM solutions, and open source tools to achieve continuous software delivery for sustained innovation and competitive advantage.
- The IBM Enterprise DevOps blog discusses current topics related to DevOps.
- Learn more about Scrum and other frameworks, methods, and practices in the IBM DevOps community.
- Learn more about IBM SmartCloud Orchestrator and Chef by reading the IBM SmartCloud Orchestrator 2.2 Content Pack for Opscode Chef.
Get products and technologies
- Learn more about IBM SmartCloud.
- Learn more about and watch a demo for IBM SmartCloud Orchestrator.
- Visit the Opscode Chef site to learn more about this configuration and deployment automation tool.
- Visit the Atlassian site to learn more about its suite of development collaboration tools, including JIRA, Stash, Confluence, and Bamboo.
- Learn more about Git functionality, key features, and the benefits of this repository structure.
- Learn more about OpenStack technologies.
- The DevOps community provides news, discussions, and training to help you and your organization build a foundation on agile development and DevOps practices.
Dig deeper into DevOps on developerWorks
Experiment with new directions in software development.
Updates on continuous delivery of software-driven innovation.
Software development in the cloud. Register today and get free private projects through 2014.
Evaluate IBM software and solutions, and transform challenges into opportunities.