![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
Thus far in this series, Part 1 introduced the underlying principles of DevOps. Part 2 discussed creating environments early in the development process for more deterministic, predictable, and secure releases. Now, this article explains integrating IT operations and information security into development to amplify your feedback loop.
One of my favorite quotes is from Patrick Lightbody, former CEO of BrowserMob, who said, "We found that when we woke developers up at 2 a.m. it was a phenomenal feedback loop, defects got fixed faster than ever."
The quote underscores a problem: Developers check in their code at 5 p.m. on Friday, high-five each other in the parking lot, and then go home, leaving IT operations to clean up any messes over the weekend. Compounding the problem, defects and known errors keep recurring in production, forcing IT operations to continually fight fires. The root cause is never fixed because development is focused on building new features.
An important element of The Second Way is to shorten and amplify feedback loops, thereby bringing development closer to the customer experience (which includes IT operations and the end users of the service being delivered).
Note the symmetry of ideas now. My favorite pattern #1, about making environments available early (see Part 2), is all about embedding IT operations into development. My favorite pattern #2 is about putting development into IT operations. The pattern puts development into the IT operations escalation chain, possibly in Level 3 support. Or, development could even be completely responsible for the success of the code deployments by either rolling back or fixing forward until service is restored to the customer.
The goal is not for Development to replace IT operations. You want to ensure that development sees the downstream effects of their work and changes. Developers walk in the shoes of IT operations enough to be motivated to fix issues quickly to help achieve global goals.
To see how information security integrates into a DevOps work stream, watch the presentation Put Your Robots To Work: Security Automation at Twitter by the Twitter product security team. (Or, see my full notes of their talk.) Watching this presentation should be a requirement for any information security personnel who want to see how they can integrate into the daily work of development and IT operations.
Of course, the Twitter presenters mention what it's like to be at an organization going through hyper-growth (the famous "Fail Whale" due to capacity issues). But much to the amusement of all of us, they describe the birth of the Twitter information security program, which was triggered by the hacking of the @barackobama account. That breach resulted in an FTC injunction, requiring Twitter to be secure for the next 15 years.
The newlyformed information security department made huge strides during the Twitter Hack Week, which occurs once every quarter when they're able to focus on proactive work. During Hack Weeks, everyone "works on whatever they want, which they then demo to the rest of the company," similar to what's done at Facebook. Hack Week projects often focus on work that reduces technical debt, automates manual work, improves processes, and so on.
The Twitter information security team wanted to focus on creating more automation but stay anchored in these framing principles:
- Get the right information to the right people.
- Find and fix bugs as quickly as possible.
- Don't repeat your mistakes.
- Analyze from many angles.
- Let people prove you wrong.
- Help people help themselves.
- Automate dumb work.
- Keep it tailored.
The great part about the Twitter presentation is how they describe integrating into the daily work of development and IT operations by automating most of their information security work.
Justin Collins makes a fantastic point about the workflow around static code analysis. Even though the static code analysis step is "automated," information security still has to do a lot of waiting—waiting for the scan to complete, get back the big stack of reports, interpret the reports, and then find the person responsible for fixing it. And when the code changes, you have to do it all over again! Even though you're using automated tools, you're still doing a lot of manual work. So, they wanted to put their robots to work. By doing so, they can do less button-pushing tasks and do more stuff with creativity and judgement.
Back to that first Hack Week... The folks at Twitter starting building a series of tools that merged into the continuous integration process run by Jenkins. One tool they built was SADB, the Security Automation Dashboard. The logo is, of course, a sad bee.
In their talk, the team described all the amazing tools they built to put information security into the daily work of development and IT operations, including:
- SADB
- Takes input from brakeman, phantom gang, CSP, threatdeck, and roshambo. The outputs include emails that go to developers and the information security team.
- Brakeman
- Does static code analysis for Ruby on Rails, created primarily by Justin Collins. It has had 25 releases in the last year. Ideally, Brakeman runs not only at every code commit, but every time the developer saves code. See Resources for infomation about Brakeman.
- Phantom Gang
- Does dynamic application security testing (DAST).
It complements Brakeman by looking for issues such as mixed content,
sensitive forms posting over non-HTTPS, old versions of jquery that often pop
up when new microsites are created, forms without authenticity token that
are prone to forgery, and more. Phantom Gang is a bunch of node.js processes
that emulate Webkit browser sessions, which are spun up as headless browser instances to see
what users see.
The output of Phantom Gang goes to JIRA, as opposed to developers directly. Why? Often the issues found by Phantom Gang are more difficult than those traced to an individual developer.
- CSP
- "Twitter is a big fan of CSP. It's great for enforcing policy and protecting websites." They've configured CSP to disallow Javascript on the page itself; they have Javascript only served from themselves. When there's a violation of that policy, "it's almost assuredly a sign that there's a valid XSS attack underway." They also use their big data capability to look for site spikes, which is often an indicator of XSS attack. See Resources for more on how Twitter uses CSP.
This article highlighted the value of bringing IT operations and information security into development. The experience at Twitter illustrated how more automation and tools can contribute to a continuous integration process. It is highly recommended that information security teams view the Twitter presentation (see Resources).
Part 4 addresses standardizing the work of IT Operations to increase project predictability and accuracy.
Learn
-
Check out the previous parts of this "DevOps distilled" series:
- Part 1: The three underlying principles
- Part 2: Create environments early for smooth, secure code releases
-
Put Your Robots to Work: Security Automation at
Twitter: A presentation by the product security team at Twitter that
explores how
information security integrates into a DevOps work stream.
-
Here's How The Amazing Twitter Infosec Team Helps DevOps:
Read Gene Kim's blog entry.
-
Brakeman 1.9.0 Released: Learn more about the static analysis
security scanner for Ruby on Rails.
-
Improving Browser Security with CSP: Read how
Twitter uses CSP.
- The
Convergence of DevOps: Read how
DevOps is the culmination of three amazing and significant movements.
-
The History of DevOps:
Get a firsthand account of the history of DevOps.
-
The Phoenix Project: A Novel About IT, DevOps, and Helping Your
Business Win: Learn how to
recognize problems that happen in IT organizations, how
these problems jeopardize nearly every commitment the business makes, and
how DevOps techniques can fix the problems.
-
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr:
The seminal presentation by John Allspaw and Paul Hammond about
how Flickr achieved fast flow of features while maintaining
world-class stability, reliability, and security. They address the
benefits of this high rate of change, as well as the culture and
technology
needed to make it possible.
-
W. Edwards
Deming: Read about Deming on Wikipedia.
- Follow developerWorks on
Twitter.
- Get more information on security topics in
the
Security site on developerWorks.
Get products and technologies
-
Evaluate IBM
products in the way that suits you best: Download a product trial, try
a product online, use a product in a cloud environment, or spend a few hours
in the
SOA Sandbox learning how to implement Service Oriented Architecture efficiently.
Discuss
- Get involved in the My developerWorks
community. Connect with other developerWorks users while exploring the
developer-driven blogs, forums, groups, and wikis.
Table of contents
Next steps from IBM
WebSphere Portal can help your small and midsize business achieve better website security, along with faster time to value with easily deployable and customizable example Web sites.
- Try: Download a free trial version of IBM WebSphere Portal Express V6.1, which offers collaboration, document management, and secure Web content management.
- Article: This article provides an overview of the WebSphere Portal security architecture, including internet security, and secure connections between clients.
- Tutorial: This tutorial take you through the steps to develop and test your first portlet WebSphere Portal Test Environment.
- Demo: These demos show Web 2.0 features available today within WebSphere Portal, as well as several features planned for the future.
- Buy: IBM WebSphere Portal Express
