DevOps distilled, Part 3: Integrate IT operations and information security into development

In this series of articles, learn about DevOps and how it can: create a collaborative relationship between development and IT operations; enable high deployment rates; and increase the reliability, resilience, and security of your production environment. In this article, learn how to amplify feedback loops by bringing Operations and Information Security into development. Encouraging collaboration and ensuring communication will help achieve your goals.


Gene Kim (, Author, IT Revolution Press

Photo of Gene KimGene Kim is the founder and former CTO of Tripwire. He has written two books: The Visible Ops Handbook and The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.

22 January 2013


DevOps for Dummies

Today's fast-moving world makes DevOps essential for any business aspiring to be agile and lean to respond rapidly to changing customer and marketplace demands. This book helps you understand DevOps and explains how your organization can gain real business benefits from it. You'll also discover how a holistic view of DevOps that encompasses the entire software delivery life cycle—from ideation and the conception of new business capabilities to implementation in production—can bring competitive advantage in a continuous delivery world.

Download "DevOps for Dummies."

Thus far in this series, Part 1 introduced the underlying principles of DevOps. Part 2 discussed creating environments early in the development process for more deterministic, predictable, and secure releases. Now, this article explains integrating IT operations and information security into development to amplify your feedback loop.

Amplifying feedback loops

One of my favorite quotes is from Patrick Lightbody, former CEO of BrowserMob, who said, "We found that when we woke developers up at 2 a.m. it was a phenomenal feedback loop, defects got fixed faster than ever."

The quote underscores a problem: Developers check in their code at 5 p.m. on Friday, high-five each other in the parking lot, and then go home, leaving IT operations to clean up any messes over the weekend. Compounding the problem, defects and known errors keep recurring in production, forcing IT operations to continually fight fires. The root cause is never fixed because development is focused on building new features.

The Second Way, discussed in Part 1, creates the right to left feedback loop. It shortens and amplifys feedback loops so corrections can be made continually.

An important element of The Second Way is to shorten and amplify feedback loops, thereby bringing development closer to the customer experience (which includes IT operations and the end users of the service being delivered).

Note the symmetry of ideas now. My favorite pattern #1, about making environments available early (see Part 2), is all about embedding IT operations into development. My favorite pattern #2 is about putting development into IT operations. The pattern puts development into the IT operations escalation chain, possibly in Level 3 support. Or, development could even be completely responsible for the success of the code deployments by either rolling back or fixing forward until service is restored to the customer.

The goal is not for Development to replace IT operations. You want to ensure that development sees the downstream effects of their work and changes. Developers walk in the shoes of IT operations enough to be motivated to fix issues quickly to help achieve global goals.

To see how information security integrates into a DevOps work stream, watch the presentation Put Your Robots To Work: Security Automation at Twitter by the Twitter product security team. (Or, see my full notes of their talk.) Watching this presentation should be a requirement for any information security personnel who want to see how they can integrate into the daily work of development and IT operations.

Of course, the Twitter presenters mention what it's like to be at an organization going through hyper-growth (the famous "Fail Whale" due to capacity issues). But much to the amusement of all of us, they describe the birth of the Twitter information security program, which was triggered by the hacking of the @barackobama account. That breach resulted in an FTC injunction, requiring Twitter to be secure for the next 15 years.

The newlyformed information security department made huge strides during the Twitter Hack Week, which occurs once every quarter when they're able to focus on proactive work. During Hack Weeks, everyone "works on whatever they want, which they then demo to the rest of the company," similar to what's done at Facebook. Hack Week projects often focus on work that reduces technical debt, automates manual work, improves processes, and so on.

The Twitter information security team wanted to focus on creating more automation but stay anchored in these framing principles:

  • Get the right information to the right people.
  • Find and fix bugs as quickly as possible.
  • Don't repeat your mistakes.
  • Analyze from many angles.
  • Let people prove you wrong.
  • Help people help themselves.
  • Automate dumb work.
  • Keep it tailored.

The great part about the Twitter presentation is how they describe integrating into the daily work of development and IT operations by automating most of their information security work.

Automating security

Justin Collins makes a fantastic point about the workflow around static code analysis. Even though the static code analysis step is "automated," information security still has to do a lot of waiting—waiting for the scan to complete, get back the big stack of reports, interpret the reports, and then find the person responsible for fixing it. And when the code changes, you have to do it all over again! Even though you're using automated tools, you're still doing a lot of manual work. So, they wanted to put their robots to work. By doing so, they can do less button-pushing tasks and do more stuff with creativity and judgement.

Back to that first Hack Week... The folks at Twitter starting building a series of tools that merged into the continuous integration process run by Jenkins. One tool they built was SADB, the Security Automation Dashboard. The logo is, of course, a sad bee.

In their talk, the team described all the amazing tools they built to put information security into the daily work of development and IT operations, including:

Takes input from brakeman, phantom gang, CSP, threatdeck, and roshambo. The outputs include emails that go to developers and the information security team.
Does static code analysis for Ruby on Rails, created primarily by Justin Collins. It has had 25 releases in the last year. Ideally, Brakeman runs not only at every code commit, but every time the developer saves code. See Resources for infomation about Brakeman.
Phantom Gang
Does dynamic application security testing (DAST). It complements Brakeman by looking for issues such as mixed content, sensitive forms posting over non-HTTPS, old versions of jquery that often pop up when new microsites are created, forms without authenticity token that are prone to forgery, and more. Phantom Gang is a bunch of node.js processes that emulate Webkit browser sessions, which are spun up as headless browser instances to see what users see.

The output of Phantom Gang goes to JIRA, as opposed to developers directly. Why? Often the issues found by Phantom Gang are more difficult than those traced to an individual developer.

"Twitter is a big fan of CSP. It's great for enforcing policy and protecting websites." They've configured CSP to disallow Javascript on the page itself; they have Javascript only served from themselves. When there's a violation of that policy, "it's almost assuredly a sign that there's a valid XSS attack underway." They also use their big data capability to look for site spikes, which is often an indicator of XSS attack. See Resources for more on how Twitter uses CSP.


This article highlighted the value of bringing IT operations and information security into development. The experience at Twitter illustrated how more automation and tools can contribute to a continuous integration process. It is highly recommended that information security teams view the Twitter presentation (see Resources).

Part 4 addresses standardizing the work of IT Operations to increase project predictability and accuracy.



Get products and technologies

  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.


  • Get involved in the My developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.


developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into Security on developerWorks

Zone=Security, Web development, DevOps
ArticleTitle=DevOps distilled, Part 3: Integrate IT operations and information security into development