![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
With software development projects, all too often development will consume all the time in the schedule when developing features and leave insufficient time to adequately address IT operations issues. Shortcuts are then taken in defining, creating, and testing everything the code relies upon, which includes databases, operating systems, network, virtualization, and so forth.
This scenario is certainly one cause of perpetual tension between development and IT operations. It also contributes to sub-optimal outcomes. The consequences are well-known: inadequately defined and specified environments, no repeatable procedures to deploy them, incompatibilities between deployed code and the environment, and so on.
The consequences for information security are especially grave. When a project runs out of time, steps at the end of the project get cut. Typically, that includes security testing and the time required to fix any vulnerabilities that are found.
In this article, I'll share the first of my six favorite DevOps patterns. In this pattern, you'll make environments available early in the development process and enforce a policy that the code and environment be tested together. The environment is defined as everything required to run the service (except for the application): servers, databases, networking, virtualization, and all the associated configurations.
By making environments available early in the development process, you'll create more deterministic and predictable releases with synchronized development, quality assurance (QA), and production environments. You can significantly reduce the cycle times of deployments, which enables higher deploy rates. Keep reading to learn how development and IT operations can speed up deployments. I'll also discuss how the information security department must change in this type of value stream.
Make environments available early in the development cycle
The goal of the pattern is to ensure that the code and the environment are tested and working together continually through the development process. By the time the code needs to be deployed, it's been exercised many times before and the most significant problems have already been found and fixed.
Thus, you can shrink the batch size (the size of changes you make), which reduces the risk of the change. If you're familiar with lean methods of manufacturing, you'll know that shrinking the batch size also decreases the cycle time and increases the flow of work. Figure 1 shows an example.
Figure 1. Frequent deployments
When development is using an Agile process, making a very simple and elegant policy change has an immediate and dramatic effect. According to traditional Agile methods, you're supposed to have working, shippable code at the end of each sprint interval (typically every two weeks). You must modify the Agile sprint policy so that instead of having just shippable, viable code at the end of each sprint, you also require a rebuildable environment that the code deploys into. Ideally, this should happen at the earliest sprints (sprint 0 and sprint 1).
Process for one-step automated environment creation
Instead of IT operations being responsible for creating the specifications of the production environment, they will build an automated environment creation process. This mechanism will create the production environment and the environments for development and QA.
By making environments, and the tools that create them, available early (perhaps even before the software project begins), developers and QA can run and test their code in consistent and stable environments with controlled variance from the production environment.
By keeping variance between the different stages—development, QA, integration test, production—as small as possible, you'll find and fix interoperability issues between the code and environment long before production deployment.
Ideally, the deployment mechanism you build is completely automated. Tools could include shell scripts, Puppet, Chef, Solaris Jumpstart, Redhat Kickstart, Debian Preseed, and so on.
Create the "single shared repository of truth"
With the exception of data, all parts of the software system should be rebuildable from scratch from version control. You must be able to use this repository to repeatedly and reliably reproduce a working software system for production and for all the pre-production environments. You can then restore service quickly and predictably, even when a catastrophic failure occurs.
To create the shared repository, development and IT operations will check the following into version control:
- Application code
- All supporting automated tests and any manual test scripts
- Any scripts that support build, deployment, database migration, and environment provisioning
- Any project artifacts (for example, requirements documentation, deployment procedures, release notes, and so forth)
- Configuration information required to provision servers (kickstart or preseed scripts, puppet or chef scripts, DNS zone files, firewall configuration rules)
Often, you'll also want to check in artifacts that your build and deployment processes rely upon. Examples include the deployment pipeline toolchain (compiler, continuous integration tool, testing tools) and the environments it depends upon (virtual machine images, puppet or chef configurations).
Information security and controls
By simply following the patterns described above, you can create more deterministic and predictable releases. The development, QA, and production environments will also be synchronized, which will invariably reduce the cycle time of deployments and subsequently increase the deploy rates.
Now, the information security team must properly embed controls into the upstream operations to prevent errors that can lead to a security breach. Or, they at least need to be able to detect and respond to a breach quickly.
Some actions that information security must take to create effective controls are discussed below.
Help the automated environment creation team
When the automated environment creation capability is being built (or already exists), information security needs to find this team and offer help to define the "golden build" standards. The goal is to ensure that the environments being created by the automated process are not only stable and reliable, but that they're secure and in a risk-reduced state.
Any sort of hardening guidance that you've defined in your security policies can now be integrated into the build process, which will significantly reduce the amount of variance and work required from information security.
Any security monitoring tools can also be added at this point in the process.
Find the continuous testing suites in the continuous deployment pipeline
The goal is to add as many security tests as possible into the continuous integration process to ensure that errors or defects are found as early as possible.
For example, if there are any recurring production problems (such as misconfiguration or errors), you should add a test in the pre-production tests to ensure that it doesn't happen again. Any sort of configuration enforcement should also be added here.
Define which changes and deployments can occur without triggering a full review or retest
Obviously, in order to get to tens, hundreds, or thousands of deploys per day, information security can't manually review code for each change. You must define which types of changes can be made without triggering a full security retest.
Conversely, you must also designate which areas of the code or environment have such security risks that any changes must be fully reviewed and retested—even if take six weeks to complete. For example, portions of code that handle user authorization might take a long time to review and retest.
In this article, I've shared my first favorite set of DevOps patterns. Creating environments early in the development process can yield predictable and secure releases.
Part 3 of this series explores another set of great DevOps patterns.
Learn
- The
Convergence of DevOps: Read how
DevOps is the culmination of three amazing and significant movements.
-
The History of DevOps:
Get a firsthand account of the history of DevOps.
-
The Phoenix Project: A Novel About IT, DevOps, and Helping Your
Business Win: Learn how to
recognize problems that happen in IT organizations, how
these problems jeopardize nearly every commitment the business makes, and
how DevOps techniques can fix the problems.
-
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr:
The seminal presentation by John Allspaw and Paul Hammond about
how Flickr achieved fast flow of features while maintaining
world-class stability, reliability, and security. They address the
benefits of this high rate of change, as well as the culture and
technology
needed to make it possible.
-
W. Edwards
Deming: Read about Deming on Wikipedia.
- Follow developerWorks on
Twitter.
- Get more information on security topics in
the
Security site on developerWorks.
Get products and technologies
-
Evaluate IBM
products in the way that suits you best: Download a product trial, try
a product online, use a product in a cloud environment, or spend a few hours
in the
SOA Sandbox learning how to implement Service Oriented Architecture efficiently.
Discuss
- Get involved in the My developerWorks
community. Connect with other developerWorks users while exploring the
developer-driven blogs, forums, groups, and wikis.
