Agility@Scale: Strategies for Scaling Agile Software Development

1. Day. A typical day begins with a short coordination meeting, called a Scrum meeting in the Scrum method. After the daily coordination meeting the team collaborates throughout most of the day to perform their work. The day concludes with a working build, hopefully you had several working builds throughout the day, which depending on your situation may require a bit of stabilization work to achieve.
2. Iteration. DAD construction iterations begin with an iteration planning session (coordinate) where the team identifies a detailed task list of what needs to be done that iteration. Note that iteration modeling is often part of this effort. Throughout the iteration they collaborate to perform the implementation work. They conclude the iteration by producing a potentially consumable solution, a demo of that solution to key stakeholders, and a retrospective to identify potential improvements in the way that they work.
3. Release. The DAD lifecycle calls out three explicit phases - Inception, Construction, and Transition – which map directly to coordinate, collaborate, and conclude respectfully.

• Coordinate maps to plan
• Collaborate maps to do
• Conclude maps to check and act

There is a fair bit of rhetoric surrounding agile methods, some of which we subscribe to and some of which we don’t. We’d like to briefly examine the rhetoric which we’ve found to be the most misleading for people trying to be effective at adopting agile techniques. The following list is in the format X but Y, where X is the rhetoric and Y is the strategy promoted by the Disciplined Agile Delivery (DAD) process framework. This includes:

1. Requirements evolve throughout the lifecycle BUT the scope should still be agreed to at the beginning of the project. There has to be an initial vision for a project, a vision which your stakeholders should help define and then agree to, and to come to that vision you will need to perform some initial requirements envisioning. A list of high level features is part of this initial vision. Yes, the details are very likely to evolve over time but the fundamental goals of your project and scope of your effort needs to be defined early in your project. In a very small minority of situations you may not be able to get the right people together, either physically or virtually, to define the initial vision – this should be seen as a significant project risk.
2. Simple designs are best BUT the architecture should be thought out early in the lifecycle. Too many developers interpret the advice to focus on simple designs to mean that they should build everything from scratch. Yet more often than not the simplest design is to take advantage of what is already there, and the best way to do that is to work closely with people who understand your existing technical infrastructure. Investing in a little bit of architectural envisioning early in the lifecycle enables your team to identify existing enterprise assets that you can leverage, to identify your architectural options, and to select what appears to be the best option available to you. The details will still emerge over time, and some decisions will be deferred until a later date when it’s more appropriate to make them, but the bottom line is that disciplined agilists think before they act.
3. Teams should be self organizing BUT they are still constrained (and enhanced) by your organizational ecosystem. Intellectual workers, including IT professionals, are most effective when they have a say in what work they do and how they do it. IT professionals can improve their productivity by following common conventions, leveraging and building out a common “dev-ops” infrastructure, building towards a common vision, and by working to common business and technical visions.  In short, disciplined agile professionals are "enterprise aware".
4. Delivery teams don’t need prescriptive process definitions BUT they do need some high-level guidance to help organize their work. Individual IT professionals are typically highly-skilled and highly-educated people often with years of experience, and teams of such people clearly have a wide range of knowledge. As a result of this knowledge it is incredibly rare for such people to read detailed procedures for how to do their work. However, they often still require some high-level advice to help them to organize their work effectively. Teams can often benefit from techniques and patterns used by other teams and this knowledge sharing should be encouraged.
5. IT professionals know what to do BUT they’re still not process experts. A decade ago the strategy was to provide detailed process advice to teams, but recently the pendulum has swung the other way to provide little or no defined process at all. Over the last few years there’s been a trend within the agile community to advise teams to define their own process so that it’s tailored to their own unique situation. While this clearly strokes people’s egos, it’s relatively poor advice for several reasons. First, although every team is in a unique situation there is significant commonality so having at least a high-level process framework from which to start makes sense. Second, although these teams have a wide range of knowledge it might not be complete, nor consistent, nor is it clear what the trade-offs are of combining all the really good techniques that people know about. There is significant benefit in having a flexible process framework such as DAD which shows how everything fits together.
6. IT professionals should validate their own work to the best of their ability BUT they likely aren’t testing experts so therefore need help picking up the appropriate skills. The mantra in the agile community is to test often and test early, and better yet to test first. As a result agile teams have adopted a “whole team” approach where the development team does its own testing. This works when there are people on the team with sufficient testing skills and more importantly can transfer those skills to others. Minimally you will need to embed testers into your delivery teams, but you should also consider explicit training and mentoring of everyone on the team in testing and quality skills.  You may find my agile testing and quality strategies article to be an interesting read.
7. Disciplined agile teams work in an iterative manner BUT still follow a lifecycle which is serial over time. On any given day people on a DAD project team may be performing analysis, testing, design, programming, deployment, or a myriad of other activities and iterating back and forth between them. But, the DAD lifecycle includes three distinct phases which are performed in order. So, DAD is both iterative in the small but serial in the large.

This article has been replaced by an official "Disciplined Agile Manifesto".

The text of the original article remains below.

---------------------------------------------------------------------------------------

I've recently been working with Mark Lines of UPMentors and we've had some interesting discussions around evolving the Agile Manifesto which I thought I would share here to obtain feedback.  Note that this is not any sort of official position of IBM, nothing in my blog is by the way (unless explicitly stated so), nor is it some sort of devious plot to take over the agile world (although if we did have some sort of devious plot, we'd make the exact same claim).  What we hope to accomplish is to put some ideas out there in the hopes of getting an interesting conversation going.

Over the past decade we’ve applied the ideas captured in the Agile Manifesto and have learned from our experiences doing so.  What we’ve learned has motivated us to suggest changes to the manifesto to reflect the enterprise situations which we have applied agile and lean strategies in.  We believe that the changes we’re suggesting are straightforward:

1. Where the original manifesto focused on software development, a term which too many people have understood to mean only software development, we suggest that it should focus on solution delivery.
2. Where the original focused on customers, a word that for too many people appears to imply only the end users, we suggest that it focus on the full range of stakeholders instead.
3. Where the original manifesto focused on development teams, we suggest that the overall IT ecosystem and its improvement be taken into consideration.
4. Where the original manifesto focused on the understanding of, and observations about, software development at the time there has been some very interesting work done within the lean community since then (and to be fair there was very interesting work done within that community long before the Agile Manifesto was written).  We believe that the Agile Manifesto can benefit from lean principles.

Our suggested rewording of the Agile Manifesto follows, with our suggested changes in italics.

Updating the Values of the Agile Manifesto

We are uncovering better ways of developing software by doing it and helping others do it.  Through this work we have come to value:

1. Individuals and interactions over processes and tools
2. Working solutions over comprehensive documentation
3. Stakeholder collaboration over contract negotiation
4. Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Updating the Principles behind the Agile Manifesto

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable solutions.
2. Welcome changing requirements, even late in the solution delivery lifecycle. Agile processes harness change for the stakeholder’s competitive advantage.
3. Deliver working solutions frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
4. Stakeholders and developers must work together daily throughout the project.
5. Build projects around motivated individuals.  Give them the environment and support they need, and trust them to get the job done.
6. The most efficient and effective method of conveying information to and within a delivery team is face-to-face conversation.
7. Quantified business value is the primary measure of progress.
8. Agile processes promote sustainable development.  The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
9. Continuous attention to technical excellence and good design enhances agility.
10. Simplicity--the art of maximizing the amount of work not done--is essential.
11. The best architectures, requirements, and designs emerge from self-organizing teams.
12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
13. Leverage and evolve the assets within your organizational ecosystem, and collaborate with the people responsible for those assets to do so.
14. Visualize workflow to help achieve a smooth flow of delivery while keeping work in progress to a minimum.
15. The organizational ecosystem must evolve to reflect and enhance the efforts of agile teams, yet be sufficiently flexible to still support non-agile or hybrid teams. 

We’re agile – things evolve, including manifestos.  Looking forward to your feedback (add a comment).

Updates Since this Was First Published:

• February 2011: At the 10th Anniversary of the Agile Manifesto get together several people, including myself, brought up the idea of updating the Agile Manifesto.  Unfortunately the decision by the original writers of the manifesto was to adopt a very strict, non-agile change management approach where any changes to the manifesto would need the approval of all 17 of them.  So, pretty much zero chance of there ever being any changes (hence the lack of changes so far). 
• June 2012: This extension to the manifesto was retitled "A Disciplined Agile Manifesto" and published as in the book Disciplined Agile Delivery written by myself and Mark Lines.
• November 2012: Mark and I formed Scott Ambler+Associates.
• February 2013: The Disciplined Agile Consortium (DAC) is formed.  The mission of DAC is to evolve and support the Disciplined Agile Delivery (DAD) framework.
• April 2014: Official Disciplined Agile Manifesto posted on the Disciplined Agile Delivery (DAD) site.

Further reading:

• The Declaration of Interdependence (some other thoughts about, arguably, evolving the manifesto)
• Examining the Agile Manifesto
• Disciplined Agile Delivery website

1. Does the team regularly produce value for their stakeholders?
2. Does the team validate its own work to the best of its ability?
3. Are stakeholders actively involved?
4. Is the team self organizing?
5. Does the team strive to improve their process?

Some interesting results include:

• 94% of teams which are claiming to be agile are providing value to stakeholders on a regular basis.  
• 87% of teams which are claiming to be agile are validating their own work.  
• 95% of teams which are claiming to be agile are working closely with stakeholders.  
• 56% of teams which are claiming to be agile are self organizing. 
• 88% of teams which are claiming to be agile are improving the process that they follow throughout the lifecycle.  
• Teams which are claiming to be agile often aren't.  53% of "agile teams" meet the five criteria, although 72% meet all but the self-organization criteria.
• Teams which are moving towards agile but aren't there yet are reasonably close.  39% of those teams meet all five criteria and 63% meet all but self-organization.

• Whenever someone claims to be on an agile team you may want to explore that claim a bit deeper.
• The low level of self organization may be an indicator of cultural challenges with organizations in that their project managers aren't giving up sufficient control.  The Agility at Scale survey in November 2009 found that 59% of respondents who indicated that their organization hadn't adopted agile techniques yet that a rigid culture was hampering their efforts.  The IT Governance and Project Management survey in July 2009 discovered that "questionable behaviors", many of which were ethically questionable (I'm being polite), were far too common within IT project management.
• Although "agile teams" may not be as agile as they claim, they're still doing better than traditional V-model teams, as revealed (again) by the 2010 IT Project Success survey.
• If there was some sort of consensus within the agile community as to the criteria for determining whether a team is agile, I highly suspect that the agileness ratings would increase over time.  What gets measured often improves.
• However, how agile you are isn't anywhere near as important as getting better at what you're doing.  So perhaps I'm barking up the wrong tree on this issue.  ;-) 

This blog posting has been replaced by the more detailed article: Full Agile Delivery Lifecycles.  

Thank you for your patience.

1. Projects where the stakeholders are willing to actively work with you.
2. IT people who are flexible, willing to try new things, and willing to collaborate with one another.
3. IT people who are generalizing specialists, or at least willing to become so.
4. Finding a project which is of medium complexity (therefore it's "real" in the sense that it's significant to your organization) but not one where it can make or break your organization (therefore it's safe to experiment with).

1. Being agile may not have been the primary determinant of success.  You set up an environment where you have a good relationship with your stakeholders, where you have good people who want to work together, and the project is challenging but not impossible.  Oh, and by the way you adopted a few agile techniques as well.  Sounds to me that situation you could have adopted a few not-so-agile techniques instead and still succeed.  Although my various project success surveys, see my IT surveys page for details, have shown time and again that agile project teams are more successful than traditional project teams I haven't been able to tease out (yet) whether this success is attributable to agile or just attributable to improved project initiation efforts.
   
2. When adopting agile/lean widely across your organization, you can't cherry pick any more.  For the past few years I've been working with IT organizations that are in the process of adopting agile/lean strategies across their entire organization, not just across a few pilot projects.  What these organizations are finding is that they need to find ways to adopt agile where the business isn't as willing to work with IT, where some of the people aren't so flexible or collaborative, where some of the people are narrowly specialized and not as willing to expand their skills, or where the project exhibits scaling factors which motivates you to tailor your agile approach.  It's harder to succeed with agile in these situations because they're not as "cherry" as what you've experienced previously.  Luckily, if you've been successful previously then you now have some agile experienced people, you have successes to reference, and you've likely overcome some problems even in the cherry situations which you have learned from.  So, your cherry successes will hopefully improve your ability to succeed even in "non cherry" situations.
   
3. You need to work smarter, not harder. If the source of your success was actually from improved project initiation practices and not from agile, then recognize that and act accordingly.  Realistically part of your success was from that and part was from agile, and the organizations that adopt a measured improvement approach potentially have the data to determine which practices lead to success and which didn't.  Without the metrics you're effectively flying blind when it comes to deciding how to improve.  There is clearly a mandate for smarter work practices within IT, within your organization as a whole for that matter.
   

• Examining the Agile Manifesto 
• Reworking the Agile Manifesto for Enterprise Agile
• Scaling Agile: Start with a Disciplined Foundation
• Scaling Agile: The Software Development Context Framework
• Disciplined Agile Delivery (DAD) home page
• Generalizing Specialists: Improving Your IT Skills

1. People. People and the way they work together have a greater effect on the outcomes of a project than the processes they’re following or the products (tools and technologies) that they’re using. People issues include having visible executive sponsorship, building an environment of trust, empowering staff, focusing on leadership as well as management, recognizing that the primary gating factor when improving processes is people’s ability to absorb change, and promoting a cross-discipline strategy at both the team and individual levels.
2. Principles/philosophies. We’ve found both internally within IBM as well as with many of our customers that there is a need to define a common set of principles to provide a consistent foundation to enable effective teamwork and continuous process improvement. These principles help to guide people’s decisions when their processes and practices don’t directly address the situation which they find themselves in.
3. Practices/patterns. A practice is a self-contained, deployable component of a process.  You might find the IBM Practices interesting.
4. Products. This includes the technologies – such as databases, application servers, networks, and client platforms – and tools such as integrated development environments, testing tools, and project planning tools used to create solutions for stakeholders.
5. Processes. The previous 4Ps do not exist in a vacuum, we need some sort of glue to help piece all of this together. Minimally this glue is a lifecycle although more often than not it is a full process or method.

• Not providing sufficient training, education, and mentoring
• Not supporting pair programming
• Not reducing the creation of non-executable detailed speculations early in the project
• Not reducing the confirmatory testing being done by their independent QA/testing team
• Completely reducing initial requirements envisioning and architecture envisioning
• Completely reducing parallel independent testing

1. "Fixed price". At the beginning of the project develop, and then commit to, an initial estimate based on your up-front requirements and architecture modeling efforts. Hopefully this estimate is given as a range, studies have shown that up-front estimating techniques such as COCOMO II or function points are accurate within +/- 30% most of the time although my July 2009 State of the IT Union survey found that on average organizations are shooting for +/- 11% (their actuals come in at +/- 19% on average, but only after doing things such as dropping scope, changing the estimate, or changing the schedule). Fixed-price funding strategies are very risky in practice because they promote poor behavior on the part of development teams to overcome the risks foisted upon them as the result of this poor business decision. It is possible to do agile on a fixed budget but I really wouldn't recommend it (nor would I recommend it for traditional projects). If you're forced to take a fixed-price approach, and many teams are because the business hopes to reduce their financial risk via this approach not realizing that it actually increases their risk, then adopt strategies that reduce the risk.
2. Stage gate. Estimate and then fund the project for given periods of time. For example, fund the project for a 3-month period then evaluate it's viability, providing funding for another period of time only to the extent that it makes sense. Note that stages don't have to be based on specific time periods, they could instead be based on goals such as to intiate the project, prove the architecture with working code, or to build a portion of the system. Disciplined agile methods such as Open Unified Process have built in stage-gate decision points which enable this sort of strategy. When the estimation technique is pragmatic, the best approaches are to have either the team itself provide an estimate for the next stage or to have an expert provide a good gut feel estimate (or better yet have the expert work with the team to develop the estimate). Complex approaches such as COCOMO II or SLIM are often little more than a process facade covering up the fact that software estimating is more of an art or a science, and prove to be costly and time consuming in practice.
3. Time and materials (T&M). With this approach you pay as you go, requiring your management team to actually govern the project effectively. Many organizations believe a T&M strategy to be very risky, which it is when your IT governance strategy isn't very effective. An interesting variation, particularly in a situation where a service provider is doing the development, is an approach where a low rate is paid for their time which covers their basic costs, the cost of materials is paid out directly, and delivery bonuses are paid for working software. This spreads the risk between the customer/stakeholder and the service provider. The service provider has their costs covered but won't make a profit unless they consistently deliver quality software.

1. Something's Gotta Give: Argues for a flexibly approach to funding, schedule, and/or scope.
2. Agile on a Fixed Budget: Describes in detail how to take a fixed-price approach on agile projects.
3. The Dire Consequences of Fixed-Price IT Projects: Describes in detail the questionable behavior exhibited by IT teams when forced to take a fixed-price approach.
4. Is Fixed-Price Software Development Unethical?: Questions the entire concept of fixed-price IT projects, overviewing some of the overwhelming evidence against this really poor practice.
5. Reducing the Risk of Fixed-Price Projects: Describes viable strategies for addressing some of the problems resulting from the decision of fixed-price projects.
6. Strategies for Funding Software Development Projects: Describes several variations on the strategies described above.
7. Lies, Great Lies, and Software Development Project Plans: Summarizes some results from the July 2009 State of the IT Union survey which explored issues related to project funding (among many).

• Daily Scrum Meeting ==> Daily Stand-up Meeting
• Product Backlog ==> Work Item List
• Scrum Master ==> Team Lead or Team Coach
• Sprint ==> Iteration or Time Box

• Translating Scrum Terminology
• Implementing Scrum Site
• Scaling Scrum
• Roles on Agile Teams

• Not only working software, but also well-crafted software
• Not only responding to change, but also steadily adding value
• Not only individuals and interactions, but also a community of professionals
• Not only customer collaboration, but also productive partnerships