Agile DevOps: The flattening of the software release process

How DevOps is transforming software delivery

What does it mean to "flatten" your software release process? How will this affect your organizational structure? In the first installment of the Agile DevOps series, DevOps expert Paul Duvall describes how developers and operations are working together on software delivery teams to streamline the process of developing and releasing software. He discusses such nascent topics as test-driven infrastructures, transient environments, and the Chaos Monkey — and how these techniques all work toward the goal of getting software to users more quickly and more often.


Paul Duvall, CTO, Stelligent

Paul DuvallPaul Duvall is the CTO of Stelligent. A featured speaker at many leading software conferences, he has worked in virtually every role on software projects: developer, project manager, architect, and tester. He is the principal author of Continuous Integration: Improving Software Quality and Reducing Risk (Addison-Wesley, 2007) and a 2008 Jolt Award Winner. He is also the author of Startup@Cloud and DevOps in the Cloud LiveLessons (Pearson Education, June 2012). He's contributed to several other books as well. Paul authored the 20-article Automation for the people series on developerWorks. He is passionate about getting high-quality software to users quicker and more often through continuous delivery and the cloud. Read his blog at

23 August 2012

When it comes to delivering software to users, Agile development has yet to live up to some of its promises in many organizations. The reasons for this are manyfold, but the central problem is a misalignment in organizational structure. More often than not, a company's development and operations teams are in separate corporate units with only one executive in common — the CEO, who usually trusts the subordinate executives to make appropriate decisions. The two teams' aims are in inherent conflict. Development is measured by the number of features it makes available to the business, while operations is measured by the system's up time — its stability. Because such companies lack a cohesive view of the software system, neither team is properly incentivized to achieve the goal of regularly delivering high-quality features that run in a stable environment.

About this series

Developers can learn a lot from operations, and operations can learn a lot from developers. This series of articles is dedicated to exploring the practical uses of applying an operations mindset to development, and vice versa — and of considering software products as holistic entities that can be delivered with more agility and frequency than ever before.

Thomas Friedman's 2005 book The World is Flat makes the case that factors such as globalization, software, the Internet, and the opening of certain developing countries' borders are converging to "flatten" traditional barriers to participation in the world's economy. The software industry is now seeing its own "flattening" trend within companies that seek competitive advantage: the flattening of software releases and organizational structures. This transformation — made possible by a convergence of automation, tools, collaboration, and industry best practices and patterns — is making software delivery at these companies faster, better, and cheaper.

In this series, I'll show how software systems can be viewed more holistically and how software organizations must be flattened to achieve the objective of regular delivery of stable features. This article introduces many of the topics that I'll cover throughout the series, including Agile, DevOps, emerging industry patterns, tools, and collaboration. All of them contribute to achieving this objective.

Get involved

developerWorks Agile transformation provides news, discussions, and training to help you and your organization build a foundation on agile development principles.

Agile + DevOps

Three of the 12 principles in the Agile Manifesto (see Resources) emphasize software delivery practices:

  • "Our highest priority is to satisfy the customer through early and continuous delivery of valuable software."
  • "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."
  • "Working software is the primary measure of progress."

The Manifesto was written more than 10 years ago, and many organizations are still seeking to adhere to these principles. The exciting thing, though, is that other organizations are exceeding these tenets. In some companies, the software is always ready to release, and there are no walls or silos around the teams: they have one delivery team — consisting of experts in development, QA, database, operations, and so on — continually delivering valuable software to their users. This is what agility is all about.

At its heart, the motivation leading to the DevOps movement is the frustration involved in releasing and maintaining software systems across teams. On the typical software project, and broadly speaking, two teams are responsible for developing and delivering software: development and operations. The two teams must collaborate, but they often have competing motivations. For development, it's delivering new features. For operations, it's ensuring the stability of the systems. Often, there's miscommunication between teams, which leads to failures late in the software delivery process.

One of the goals of the DevOps movement is for developers and operations people to work together in a more collaborative fashion. As a result, a new breed of DevOps engineers is emerging who take the best of both disciplines and combine them to deliver value to users. This is also manifesting in the rise of cross-functional teams that are experienced in development, configuration management, database, testing, and infrastructure.


A software pattern is a solution to a problem in a particular context (context being a key to distinguishing patterns from "best practices"). Some of the patterns related to Agile DevOps are scripted environments, test-driven infrastructures, Chaos Monkey, versioning everything, delivery pipeline, and DevOps dashboard. I introduce these patterns in this article, and you'll see more detail about them as the series progresses.

Scripted environments

By fully automating environment provisioning, you reduce the risks of deployment errors occurring in one environment that do not occur in other environments. By scripting environments, you verify the integrity of a particular version of the software (that is, that nothing has been modified) in target environments. By following the policy that nothing gets modified to an environment unless it's in a versioned script, automated, and part of a single path to production, you can better determine the root cause of deployment errors. The benefits of scripted environments include:

  • Environments are always in a known state.
  • They enable self-service environments and deployments.
  • They lessen the chance that deployments behave differently based on unique environment types.
  • Environments are a part of a single path (and version) to production.
  • They lessen the chance that knowledge is locked in team members' heads.
  • Deployments are more predictable and repeatable.

Infrastructure automation tools that you'll learn more about in this series support this pattern. Puppet is one such tool. The code snippet in Listing 1 exemplifies a basic Puppet program — known as a manifest:

Listing 1. Scripted environments: Puppet manifest httpd snippet
class httpd {
  package { 'httpd-devel':
    ensure => installed,
  service { 'httpd':
    ensure => running,
    enable => true,
    subscribe => Package['httpd-devel'],

Typically, numerous manifests script an infrastructure. These scripts are committed to a version-control repository just like application code.

Test-driven infrastructures

A simple premise of DevOps is the application of the best practices and patterns from development to operations and from operations to development. The test-driven approach of writing tests along with the code has its origins in software development. As infrastructure automation tools become more mainstream across organizations, engineers are applying test-driven practices to their infrastructure; the infrastructure is described in a script (as you saw in Listing 1), and these scripts have tests. Some of the many benefits of test-driven infrastructures are:

  • Problems manifest earlier because infrastructure changes are integrated with the rest of the software system through a Continuous Integration system.
  • The tests — and the scripted infrastructure — become documentation.
  • You can better isolate destructive changes, because everything is a script and described in tests.

Listing 2 illustrates a simple scenario of verifying that a web server is up and running. It uses a behavior-driven development (also known as acceptance-test driven development) tool called Cucumber, in which tests are described as scenarios.

Listing 2. Infrastructure test in Cucumber
Scenario: Is the proper version of Apache installed?
When I run "/usr/sbin/httpd -v"
Then I should see "2.2"

You'll see more test-driven infrastructure examples in future articles in this series.

Chaos Monkey

A couple of years ago, the Netflix tech team developed a continuous testing tool it calls the Chaos Monkey. The tool intentionally and randomly, but regularly, terminates instances in the Netflix production infrastructure to ensure that the systems continue to operate in the event of failure. This tool was recently released as open source (see Resources).

The principle Netflix follows is that "everything fails, all the time" (a mantra coined by Werner Vogels, CTO of As Netflix states, "The Chaos Monkey's job is to randomly kill instances and services within our architecture. If we aren't constantly testing our ability to succeed despite failure, then it isn't likely to work when it matters most — in the event of an unexpected outage" (see Resources). Later on in this series, I'll explore the Chaos Monkey and its role in a well-functioning DevOps environment.

Transient environments

The general rule of thumb with a transient environment is that self-service environments delivered to DevOps team members are as short-lived as possible, anywhere from a few hours to a few days — basically, just enough time to run through tests. One of the more significant problems on software-development projects is teams having fixed instances that no one else can touch because they took various team members days, weeks, or months to configure. This often is the result of environments that aren't scripted, making environments scarce resources. When something is scarce, you want to hold on to it and protect it.

When you have fully scripted environments, environments are no longer scarce. Everything is defined in a template and checked into a version-control system. You can terminate the environments based on the scripts as often as necessary. This pattern offers two advantages: it makes the resources available for someone else to use, and it reinforces the concept that everything must be automated — not nurtured and cultivated manually over weeks and months. You'll learn more about transient environments in this series.

Version everything

This pattern is simple and seemingly self-evident: version everything. Yes, everything: infrastructure, configuration, application code, and database scripts. Although that may be simple to understand, in my experience it's rare to find teams that version all of the artifacts required to create the software system.

The purpose of versioning everything is to establish a single source of truth (also known as canonical version or the system of record) for the software system; you treat software as a holistic unit. When versioning everything, team members aren't continually unclear or navigating myriad versions of the software system. In the series, I'll also cover an emerging practice of versioning parts of the software delivery system configuration itself.

There's one simple way to know if everything is being versioned: a new person on the team who gets a new machine and performs a single-command checkout from version control should be able to get a complete working software system from that checkout.

Delivery pipeline

With a Continuous Integration server such as Jenkins, you can also configure a delivery pipeline. Essentially, a delivery pipeline is a process in which different types of jobs are run based on the success of the previous job. In this pipeline, you can configure various stages including a commit stage, an acceptance stage, and so on. Each stage builds on the success of the previous stage; this approach ensures that you're reducing the risk of the release candidate in each successive stage until the software system is released to production. An example of a delivery pipeline — as represented in Jenkins — is shown in Figure 1:

Figure 1. Delivery pipeline as represented in the Jenkins continuous-integration server
Delivery pipeline as represented in the Jenkins continuous-integration server. There are three separate stages the software goes through. First, it runs any setup using the SetupVariables job after a build is run through a Continuous Integration process. Then it creates a target environment with the CreateTargetEnvironment job, and finally it runs the third stage to deploy the application into the target environment as represented in the DeployManateeApplication job. These three stages run as a unit of work in which if any of the jobs fail, it all fails.

DevOps dashboard

Once you have a cross-functional delivery team focused on delivering features and stability, it becomes immensely useful to get everyone on the same page — both figuratively and literally. A DevOps dashboard shows how each change affects the entire system in every stage in the progression to deploying to production. Later in the series, I'll detail an open source dashboard that provides real-time information on the software system under development.


Collaboration is one of the pillars of DevOps. Traditional development and operations teams tend to work in silos and to limit the amount of interteam communication until software release times. Ensuring collective ownership, establishing cross-functional teams, and broadening the skill sets of engineers are three ways to increase collaboration and break down the traditional barriers that prevent software from being delivered regularly.

Collective ownership

Collective ownership is a practice dating back to the establishment of extreme programming (XP) and, generally speaking, Agile methodologies. In the context of continuous delivery, however, the emphasis is on ensuring that all types of source files that make up a software system are available for any authorized team member to modify. These include the application code, configuration, the data, and even the infrastructure. Everything is scripted, and everything can be modified.

Cross-functional teams

Having a cross-functional team means that every team member is responsible for the delivery process. Any person on the team can modify any part of the software system. The corresponding antipattern is siloed teams: Development, testing, and operations have their own scripts and processes and are not part of the same team.

Cross-functional teams consist of representatives from all the necessary disciplines. Instead of treating each discipline as a separate centralized service organization, the delivery team becomes the key organizational construct. Teams work together in a dedicated fashion to deliver software consistently, without the time impediments inherent when teams communicate across the organization. Consider composing each team of (at least) business analysts, customer representatives, DBAs, developers, project managers, and QA and release engineers. With a cross-functional team, you reduce the "it's not my job" syndrome that stifles communication and the numerous "walls" preventing effective communication that are established between teams within and across physical locations.

Polyskilled engineers

A polyskilled engineer is a team member who is skilled in every area of the software delivery process. Developers should understand how to make database changes. Database administrators should understand how to write functional tests. Project managers should understand how to add scenarios to automated tests. In general, team members continue to perform the functions consistent with their roles. (For example, DBAs still generally focus on databases.) However, the shared knowledge greatly reduces the communication barriers introduced in large, geographically dispersed organizations. It also limits the need to rely on a few key individuals to get software released to users.


Tools are an important component of Agile DevOps. The tools you choose for your project depend on your particular requirements, but one essential specification is that the tool must run from the command line. This is because you want the pipeline to run in headless mode or with just one click. (If you have a legacy tool that does not provide a command-line option, you can try screen scrape to run in headless mode, but this is less than ideal and often requires regular maintenance.) In Table 1, you see a list of some of the types of tools in a typical DevOps toolset; focus more on the types of the tools rather than the tool names:

Table 1. Tools that support DevOps work
Tool TypeTools
Infrastructure automationBcfg2, CFEngine, Chef, CloudFormation, IBM Tivoli, Puppet
Deployment automationCapistrano, ControlTier, Func, Glu, RunDeck
Infrastructure as a ServiceAmazon Web Services, CloudStack, IBM SmartCloud, OpenStack, Rackspace
Build automationAnt, Maven, Rake, Gradle
Test automationJUnit, Selenium, Cucumber, easyb
Version controlSubversion, Git, IBM Rational ClearCase
Continuous IntegrationJenkins, CruiseControl, IBM Rational BuildForge

Keep in mind that the list in Table 1 is meant to be illustrative, not exhaustive (see Resources for a more detailed list of tools).

Smoother roads ahead for software delivery

The flattening of software releases and the organizations is enabled through Agile DevOps. In this article, I've commented on the state of Agile, what DevOps is, and how developers and operations can work more collaboratively to deliver features in stable environments more often. I introduced some of the emerging DevOps patterns that I will cover in more detail in future articles in this series. Finally, I provided a list of some of the tools that support Agile DevOps software delivery.

In the next article, you'll learn about the infrastructure automation tools that support the scripted environments pattern. I'll cover the leading open source infrastructure automation tools: Chef and Puppet.



Get products and technologies

  • Chaos Monkey: Chaos Monkey is the first software release in the Netflix "Simian Army."
  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.


  • Get involved in the developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.
  • The developerWorks Agile Transformation community provides news, discussions, and training to help you and your organization build a foundation on agile development principles.


developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into DevOps on developerWorks

Zone=DevOps, Java technology
ArticleTitle=Agile DevOps: The flattening of the software release process