Implementing agile development in larger organizations can be a very complex and challenging endeavor. Agile practices that can scale and address the needs of enterprise endeavors, including mission-critical systems, involves a lot more than just a stand-up meeting each morning and fixed time box sprints. Successful organizations want to see each and every project that employs agile processes enjoy the same level of success in terms of productivity and quality. This article examines some of the critical success factors that are required to implement successful agile practices reliably in your organization. It also discusses how to implement robust and mature agile processes.
For decades, many process improvement gurus have challenged the importance of tools. The common wisdom has always been that the process is more important than the tools involved. This seems intuitive, in that experienced IT professionals have often seen successful high-performance teams that have done quite well despite the limitations of their existing (often legacy) toolsets. I can attest that many talented technology teams manage to do their jobs well despite having only limited legacy or (limited) open source toolsets available for their use. Experience shows that teams can succeed despite their existing toolsets. It is equally true that having excellent tools does not guarantee success. As the popular adage says, "A fool with a tool is still a fool." But some time ago, I had a life experience that significantly changed my view of the importance of tools.
This incident involved my mother suffering a heart attack, which made it impossible for her to live for the next year on her own. We initially set up our living room as her private recovery room and then set about building an extension to our house that gave Mom a private apartment with a handicapped-accessible bathroom. Pretty soon. we had a bulldozer among other specialized equipment right in our back yard. I was quickly reminded that skilled artisans require the right tools and processes to produce their best work. Great tools are not optional!
Most process improvement gurus focus on having a reliable and repeatable approach that can be applied to both large and small projects. There are many highly successful cross-functional teams that embrace agile practices and deliver fantastic results. Many of these projects are accomplished successfully with small teams. Still others require many more resources to meet their time-to-market and functional requirements. Large banking systems, medical devices, and mission-critical engineering systems often require very large teams to meet deadlines within an acceptable timeframe.
Agile practices work. Unfortunately, although many organizations report excellent success with using agile methods on smaller projects, these same companies often run into challenges when trying to use agile practices with larger projects. We need agile practices that work in both small projects and larger projects.
Scaling agile involves taking practices that were successful in the typical small team and using them with equal effectiveness on larger projects. Using agile methods in a larger context requires training members in a consistent way. It also requires tools to help guide the team in using agile processes. Although the agile approach values individuals and interactions over processes and tools, scaling it depends upon having right sized processes and the right tools to achieve success.
Processes, themselves, require the support of workflow automation to ensure that they are repeatable and traceable. This means that you need to have the right processes and the right tools implemented to get the job done. These tools must also help support individuals and interactions with collaboration, transparency, and traceability. Successful organizations also place the appropriate focus on IT governance.
Application lifecycle management (ALM) can help by supporting the entire software or systems delivery lifecycle, or SDLC (see Configuration Management Best Practices, cited in Resources). Mature agile practices are repeatable, predictable, and can scale to support the size of the team as required. Mature agile processes must also adhere closely to agile values and principles. This is very important, because it is all too common to hear of projects where there was a claim that the methodology employed was agile, but upon closer inspection, this attribution was revealed to just be an excuse to code without well-defined requirements. I like to refer to the degree to which a development methodology adheres to agile practices as being the agile process purity.
Agile processes should adhere to the values espoused by the Agile Manifesto (see Resources for a link). Obviously, this means that we value:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
As the Agile Manifesto specifies, "While we value the items on the right, we value the items on the left more." In terms of values and priorities, this is completely true. That said, some companies need more process and others need less, although picking the right tools is always essential for your success. Contracts, documentation and detailed plans are often an absolute requirement, too. Whether you are working with venture capital funding, an Internet start-up or seeking approval for your budget to implement a large-scale trading system, many organizations simply will not approve a project unless the so-called "items on the right" (listed above) are addressed to an acceptable degree.
Although I have done many deals on a handshake (without regret), there is certainly a place for contracts, as well. Mature agile processes help to place the right focus on the items on the left while still meeting the minimal requirements for the items on the right. This comprehensive and balanced approach is essential for scaling agile processes, which is a key requirement for many organizations, particularly those in highly regulated industries.
Many of my colleagues complain that their senior management is out of touch with the day-to-day work being completed by the average employee. Indeed, sometimes it can seem like senior management does not fully understand what it really takes to build complex technology systems. The truth is that technology workers can do a lot to provide clear and useful information to their supervisors to help them make better decisions and provide more effective leadership. Managing the information that reaches senior management helps ensure that they make informed decisions based upon accurate information.
This does not mean that senior management has to make all of the decisions. In a mature agile environment, self-managed teams also play a vital role in making many of the key decisions, although, ideally, at a point when all of the essential information is available. Many agile-lean enthusiasts call this the Last Responsible Moment to indicate putting off a decision until all of the facts are available so that the decision is based upon the most up-to-date information.
The mature agile organization has a culture of sharing essential information. If you have ever taken a long trip with small children, the question, "Are we there yet?" might be ingrained in your memory. Similarly, every project stakeholder wants to understand whether the project is progressing and, of course, at what speed.
Many agile practitioners use burnup and burndown charts, which show both the progress to date (features completed, for example) and the remaining time planned to complete the iteration or release. The product backlog maintains a list of requirements, often in the form of epics and user stories (where a story is a description of the functionality that is desired by the end user). Conversations help by describing the details of the story and should be recorded, along with tests that indicate the details that help determine when a story is complete (see User Stories Applied). Epics are usually a larger description that can typically be split into two or more stories.
The rate of progress in completing epics and stories is usually measured by what is called velocity (see The Software Project Manager's Bridge to Agility). Highly visible information radiators help to communicate essential information in real time. All of these forms of communication help to create and enable a culture of transparency that helps the entire organization improve productivity and quality.
Your tools need to be able to support this focus on real-time information sharing. Efficient communication is especially important when it comes to project planning.
One misconception about agile development is that it does not require planning. The Agile Manifesto indicates that responding to change has a higher priority than following a plan. That said, however, in a mature agile environment, you often need detailed, even comprehensive plans.
Therefore, planning in an agile environment can range from simple release and iteration planning to comprehensive, detailed plans that rival anything produced by a waterfall lifecycle plan. I have seen many organizations where planning was an absolute requirement for success.
Sometimes managers are forced to make decisions before all of the information is available. I have seen extreme cases where decisions where made based upon faulty assumptions just because the estimates were required before all of the necessary facts were known. The agile approach encourages honesty when planning by allowing for decisions to be made at the "last responsible moment." This waiting until as late in the game as advisable is very important, - especially when new technologies are being used and there are many unknowns or risks that cannot be easily controlled.
There is also a need for planning releases to ensure transparency as to which features can be completed within a specified time span. Planning in an agile context is particularly affected by the concept of fixed time box iterations, where each sprint must stay within a specific time period, which is usually two to four weeks.
Release planning involves determining exactly which functions are required for a specific release to be considered complete. The iteration plan specifies the features that are to be completed within the fixed time box sprint. The completion of each iteration might be useful from a development and testing perspective, but the milestone release itself should be shipped to the customer only when the functionality has been finished to a mature point that is viable for end-use testing. At no time should a "half-baked" release be delivered to an unsuspecting customer when the functionality is so incomplete that it cannot be reasonably tested.
Sometimes, the release created in a fixed time box sprint might have just enough functionality to show to the user as a tool to verify requirements and help clarify what needs to be built. But I have also seen cases where the customer received code that was by no means at a functioning milestone. This situation had a great impact on the relationship between the customer and the vendor, because the customer felt like (and indicated) that they received an incomplete and nonfunctioning system.
Instead, customers should receive baselined milestone releases that have a reasonable number of stories completed from the product backlog. This is precisely where configuration management best practices can enable the team to deliver reliably built baselined releases as often as they are productive.
Configuration management (CM) plays an exciting role in an agile context. In a mature agile context, CM helps to manage source code intelligently, including support for baselining milestones, automated independent builds, and multiple variants in the code (bug fixes, for example).
Build engineering adds value by creating repeatable procedures that reliably produce executable binary files (usually called configuration items) that can be readily identified to ensure that the correct version was built, packaged, and deployed.
Release engineering creates a reliable package, including a manifest that lists the contents of the build (often called a bill of materials or BOM). In many environments, the release package is created in a way that ensures that it is tamper-proof and facilitates the application deployment procedure.
More importantly, these procedures provide a mechanism to audit and verify that the correct configuration items were built, packaged, and deployed as required. These best practices are much easier to implement with a robust build framework. With the help of these techniques, CM plays an essential role in facilitating rapid iterative development. CM planning helps to specify a clear road map for all of the activities necessary to handle source code management, along with build, release, and deployment engineering.
Ideally, you use an automated process that can track configuration items as they are created and checked into the source code repository and then automatically built with unit tests (and even code analysis), with the current status reflected in real time. This helps keep developers focused on what they do best by eliminating distractions and provides transparency to all stakeholders, enabling management to make the best decisions based upon the most up-to-date information.
This level of automation also facilitates rapid iterative development.
Agile development requires many iterations. This creates a situation that demonstrates the value of automated application builds, packaging, and deployment. Many teams utilize continuous integration (CI) to build the application, often numerous times during the day. There are times when a nightly build is sufficient but iterative development quickly shows the need for a well-automated, repeatable application build, package, and deployment process that ultimately pushes the complete release to a test environment, using a mature deployment framework.
Rapid iterative development places a strong focus on the entire application development lifecycle. It also places the necessary focus on software quality, with support for automated unit tests and code analysis.
This broad view helps to support the entire software and systems development lifecycle.
Application lifecycle management (ALM) is a full lifecycle endeavor, and the tools that support the agile ALM need to help guide the entire lifecycle, from planning, gathering requirements, and design, all the way to application build, package, and deployment. In a mature agile context, everyone knows what they need to accomplish on a day-to-day basis.
This is another instance where the correct tool can help by creating an automated framework for the comprehensive handling of work items, including tasks and defects linked with related artifacts, including epics, stories, and test cases.
Tasks and defects can also have their own relationship structure.
It is very common for work items such as tasks and defects to have a hierarchical relationship. In some cases, this might mean that a task has one or more subtasks, which could include tasks or related defects that need to be addressed. Epics might have children that are stories, and those stories might have tasks and defects assigned to them. Arranging work items in a hierarchical fashion provides an excellent way to organize work and show which items have been completed and the work that still needs to be finished.
Another aspect of a mature agile process is the ability to coexist with non-agile processes.
It is very common for teams to employ agile methods in an environment where they need to coexist with waterfall methodologies. The most common form of this involves the team working on a day-to-day basis in an agile context while each team reports up to senior management through a traditional waterfall software or systems lifecycle.
Tools today need to be able to support hybrid development models with both agile and non-agile software development processes. Just as agile and non-agile methods need to coexist in harmony, agile methods also often need to exist and comply with the regulatory requirements indicated by industry standards and frameworks.
Many organizations must comply with U.S. federal government regulatory requirements, including Section 404 of the Sarbanes-Oxley Act of 2002 or HIPAA CFR 21. Industry standards include guidance from organizations such as the IEEE, ISO, or ANSI and also frameworks such as the well-respected ISACA Cobit or the itSMF ITIL v3. These best practices often require a separation of duties that is greatly facilitated by fully automated application build, package, and deployment procedures.
Traceability from work items to atomic change sets is also important to identify the precise changes completed and linked directly to the work item (change request) authorizing the change. Work items are usually managed in a workflow automation tool, and they progress through predefined states, such as creation, review, approval, closure, and verification.
Mature agile processes maintain all of the excellent efficiencies of agile development while still meeting all industry regulatory requirements as described in the specific standard or frameworks developed and accepted as the relevant authority and guidance. Process improvement is a journey, and mature agile processes also emphasize continuously improving the application development methodology.
One of the most important process improvement methods is to regularly review our past actions and reflect on how we can improve. Retrospectives provide the right mechanism to review our existing processes and discuss what worked well and what might be improved. Sometimes these retrospectives can be conducted in one specific location, but other times teams are distributed across one or more locations, thus they often require tools to support collaboration and communication.
Whether you have the luxury of getting everyone together in one room for pizza and perspectives or you have to handle getting everyone's input in a less traditional manner, retrospectives can help drive the entire process improvement effort. They provide a framework for honest reflection, constant communication, and continuous improvement.
Mature agile processes need to be repeatable and scalable to meet the needs of the organization. With all of the amazing success, there still needs to be work done to allow agile methods to be equally successful in every project within an organization, including larger projects with geographically distributed team members who often work in different time zones and might even speak different languages.
Mature agile software development also needs to coexist with non-agile processes and provide the transparency by providing clear and accurate information that is essential to supporting IT governance and compliance. Agile works! Mature agile development takes these practices to the next level, providing tremendous value and even greater competitive advantage.
- For more information related to this article:
- Configuration Management Best Practices: Practical Methods that Work in the Real World, by Bob Aiello and Leslie Sachs (Addison-Wesley Professional, 2010).
- Continuous integration in agile development How agile methods, continuous integration, and test-driven enhance design and development of complex systems, by Martin R. Bakal, Jennifer Althouse, and Paridhi Verma (developerWorks, 2012)
- The Agile Manifesto.
- The Software Project Manager's Bridge to Agility, by Michele Sliger and Stacia Broderick (Addison-Wesley Professional, 2008).
- User Stories Applied: For Agile Software Development, by Mike Cohn (Addison-Wesley Professional, 2004).
- Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
- Subscribe to the developerWorks weekly email newsletter, and choose the topics to follow.
- Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
- Attend a free developerWorks Live! briefing to get up-to-speed quickly on IBM products and tools, as well as IT industry trends.
- Watch developerWorks on-demand demos, ranging from product installation and setup demos for beginners to advanced functionality for experienced developers.
Get products and technologies
- Download a free trial version of Rational software.
- Evaluate other IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.
- Join the Rational software forums to ask questions and participate in discussions.
- Ask and answer questions and increase your expertise when you get involved in the Rational forums, cafés, and wikis.
- Join the Rational community to share your Rational software expertise and get connected with your peers.
- Rate or review Rational software. It's quick and easy.
Bob Aiello is a consultant, editor-in-chief of the CM Crossroads website for developers and co-author of Configuration Management Best Practices: Practical Methods that Work in the Real World (Addison-Wesley Professional, 2010 (cmbestpractices.com). He has more than 25 years of experience as a technical manager in several top NYC financial services firms, where he had company-wide responsibility for CM, often providing hands-on technical support for enterprise source code management tools, SOX/Cobit compliance, build engineering, continuous integration, and automated application deployment. Bob has served as the vice chair of the IEEE 828 Standards working group (CM Planning) and is a member of the IEEE Software and Systems Engineering Standards Committee (S2ESC) management board. He holds a maste's degree in industrial psychology from New York University and a B.S. in computer science and math from Hofstra University. You may contact him by email or on LinkedIn.