![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
People at companies that create software products and systems frequently ask me, "How do we change our organization without actually changing it?" Of course, they don't use those actual words (at least not all the time), but that's the implication. They want to minimize the effort and risk involved in releasing software, but they realize it requires (cultural and technical) changes throughout the organization that they don't always have the power or influence to affect.
A cultural obstacle they often encounter is that traditional development and operations teams tend to work in silos, limiting the amount of inter-team communication until software release times. (And such communication is often confined to a series of tickets in an issue-tracking system.) Growing software enterprises must become more collaborative, or else they will cease to exist. The software industry is changing in this direction—much more quickly than some anticipated—as the result of cloud computing, which makes computing resources less scarce, and business demand. Companies that evolve will put those that don't out of business.
As emphasized in the introductory article of this series, collaboration across organizational boundaries is one of the anchors of agile DevOps. This article discusses how establishing cross-functional teams and broadening the skill sets of delivery-team members are ways to increase collaboration and break down the traditional barriers that prevent software from being delivered continuously.
The rise of cross-functional teams
A cross-functional team consists of experts across the entire software-delivery cycle, such as operations engineers, database administrators (DBAs), testers, developers, and analysts. Everyone on a cross-functional team contributes code to a version-control repository. For example, the operations engineer contributes infrastructure and configuration as code; the DBA contributes Data Definition Language (DDL), Data Modeling Language (DML), and datasets as code; the analyst contributes requirements as code; the developer contributes application code and configuration; and the testers contribute tests as code.
Every member of a cross-functional team is responsible for the delivery process. Any person on the team can modify any part of the software system. The corresponding antipattern is siloed teams, in which development, testing, and operations have their own scripts and processes and are not part of the same team.
Cross-functional teams, then, consist of individuals from all the disciplines responsible for developing and delivering software systems. Instead of treating each discipline as a separate centralized service organization, the delivery team becomes the key organizational construct. Teams work together in a dedicated fashion to consistently deliver software, without the time impediments inherent when teams communicate across the organization. Consider composing each team of (at least) business analysts, customer representatives, DBAs, developers, project managers, and quality-assurance (QA) and release engineers. With a cross-functional team, you reduce "it's not my job" syndrome and other "walls" that stifle communication between teams both within and across physical locations.
After some time, you'll find that skill sets often morph into more-complete engineers/analysts who begin doing more than just one part of the software-delivery process. For example, a programmer might create DDL and DML scripts. Or, business analysts define their requirements in acceptance-testing scripts (such as Cucumber) so that all changes run through automated testing based on the customer's requirements. Each team member knows that he or she needs to write tests, write scripts, version (see "Agile DevOps: Version everything"), and make everything they do part of a continuous system (see "Agile DevOps: Continuous delivery platform"). This is the case for all source artifacts: application code, configuration, infrastructure, and data. When team members become more polyskilled and silos are dismantled, communication is improved, and bottlenecks are removed from old-world software organizations.
Everything that makes up the complete software system is either a script (that is, a program) or an automated test. Everything that makes up the software system is versioned, and everything is incorporated into a continuous-delivery pipeline. This way, when anything any members contribute is checked into the version-control repository, it creates the entire software system—including environments, database(s), and the software application/service using the versioned configuration. Every component of the system is versioned, with no exceptions. The members of the cross-functional team are 100% dedicated to the project and are affiliated with no other projects. This approach enhances communication and reduces any process bottlenecks.
Making cultural change in an organization is the most essential element of employing DevOps. Nothing else significant will happen unless this happens. This section discusses a high-level approach to transitioning to—and functioning effectively with—cross-functional teams.
Attempts to change a large organization all at once are almost guaranteed to fail. The preferred approach is to start with a small pilot project that delivers something of business value to production within a relatively short time period—ideally, no more than 90 days. The pilot project should be something strategic—not an application that is rarely updated and delivering minimal business value. This new team will be a cross-functional team, and every member will be fully dedicated to the project. (They won't be working on other projects.) Every role required for developing the software system (operations, application developers, database, testers, and so forth) will be a part of this cross-functional team.
Once you've proven success with one project, you can scale out to a few more. These other projects will have team members from the first pilot project who will share their knowledge and each be a 100% member of the new team. After these projects are successful, you will double or triple subsequent pilot teams until all strategic projects are operating under the new model.
Because all service teams are small (no more than 10 people) and independent, you might wonder which organizational functions are centralized. Although every organization is different, with teams that employ DevOps and continuous-delivery principles, centralized functions are either those that develop the platform services used by the rest of the software-delivery team or those that perform system monitoring (application, network, security, and so on). There may be release coordinators, governance, and management, as well, but none of these centralized teams should be performing activities that adds to the wait time of the cross-functional teams (in other words, the services the centralized teams provide are completely self-service). Often, in these organizations, the service-delivery teams (of programmers, operations, database, testers, and so on) are on call (usually this rotates between team members) and responsible for fixing production issues.
One of the key features of DevOps is that a team member should never need another team member outside his cross-functional team to perform an activity as part of the delivery process. Everything must be self-service. Any team member should be capable of delivering software to production (see the "You build it, you run it! sidebar"). When team members are designing any feature in a software-delivery system, they need to consider how to develop it so that it can be used without delivery pipeline wait times involving submitting tickets, sending email, or using other communication mechanisms. (These tools are often still used with teams who embrace DevOps and Continuous Delivery. But in the context of the delivery pipeline, they are automated.)
With the overarching principle being to move all internal services toward self-service models, certain specific patterns apply that are covered in one way or another in this series so far. The most essential of these patterns are described in Table 1.
Table 1. Key patterns that support DevOps work
| Pattern | Description |
|---|---|
| Big visible dashboards | Teams across the organization get real-time information on the state of the software system, including build status, customer metrics, and availability. |
| Colocation | Teams are in physical proximity to one another to enhance communication. |
| Continuous integration | You build your software (environments, applications, and so on) with every change. |
| Cross-functional teams | Software-delivery teams are composed of experts across various disciplines, including programmers, testers, analysts, and operations. |
| Polyskilled experts | Reduce specialist silos by expanding skill sets on cross-functional teams. |
| Scripted deployments | Deploying software to environments is entirely scripted so that it can be run from a single command. |
| Scripted environments | Creation of environments is entirely scripted so that it can be run from a single command. |
| Self-service releases ("You build it, you run it.") | Any authorized person on the team can and does perform deployments to production. |
| Stop the line | Anyone can and should stop the continuous-integration system when necessary. |
| Test-driven everything | Write automated tests for everything: application, infrastructure, everything. This might include writing unit, acceptance, load, and performance tests. |
| Version everything | Version all artifacts: infrastructure, configuration, application code, and data. |
The death of old-world software development
You learned from this article that one of the keys to effective DevOps is breaking down silos and creating cross-functional teams that can deploy their software to production. Organizations that have long procurement and delivery cycles will either evolve or go out of business. In large organizations, evolving can take time and requires a change in organizational culture.
In the final article in this series, you'll learn how you can create a DevOps dashboard—a comprehensive view of a software system's state for development and operations teams to monitor in real time.
Learn
- "How Netflix gets out of the way of innovation" (Adrian
Cockcroft's Blog, December 2011): Cockcroft discusses how Netflix's
culture affects innovation.
- "There's No Such Thing as a 'Devops Team'" (Jez Humble,
continuousdelivery.com, October 2012): Humble explains why functional
silos and segregation of duties are problematic.
-
Andon: Learn about how an andon cord is used in lean manufacturing.
Software teams can apply the same concept in "stopping the
line."
- "Continuous Delivery: Patterns and Antipatterns in the Software
Lifecycle" (Paul Duvall, DZone, June 2011): Learn more than 40
patterns and antipatterns for continuous delivery.
- "Amazon's CTO: 'Amazon is a technology company. We just happen to do
retail'": (Brad McCarty, The Next Web, October 2011): In a
presentation video embedded in this article, Amazon CTO Werner Vogels
discusses his company's "you build it, you run it" approach to software
systems.
- Stay current with developerWorks technical events and webcasts focused on a
variety of IBM products and IT industry topics.
- Attend a free
developerWorks Live! briefing to get up-to-speed quickly on IBM
products and tools as well as IT industry trends.
- Follow developerWorks on
Twitter.
- Watch developerWorks on-demand demos ranging from product installation
and setup demos for beginners, to advanced functionality for experienced
developers.
Get products and technologies
-
Evaluate IBM
products in the way that suits you best: Download a product trial,
try a product online, use a product in a cloud environment, or spend a few
hours in the SOA Sandbox learning how to implement Service Oriented
Architecture efficiently.
Discuss
- Get involved in the developerWorks
community. Connect with other developerWorks users while exploring
the developer-driven blogs, forums, groups, and wikis.
- The developerWorks Agile transformation community provides news, discussions, and
training to help you and your organization build a foundation on agile
development principles.
Paul Duvall is the CTO of Stelligent. A featured speaker at many leading software conferences, he has worked in virtually every role on software projects: developer, project manager, architect, and tester. He is the principal author of Continuous Integration: Improving Software Quality and Reducing Risk (Addison-Wesley, 2007) and a 2008 Jolt Award Winner. He is also the author of Startup@Cloud and DevOps in the Cloud LiveLessons (Pearson Education, June 2012). He's contributed to several other books as well. Paul authored the 20-article Automation for the people series on developerWorks. He is passionate about getting high-quality software to users quicker and more often through continuous delivery and the cloud. Read his blog at Stelligent.com.
Table of contents
Next steps from IBM
Learn more, join the cloud community
