Agility@Scale: Strategies for Scaling Agile Software Development

When I talk to people about scaling agile techniques, or about agile software development in general, I often put describe strategies in terms of various risks.  I find that this is an effective way for people to understand the trade-offs that they're making when they choose one strategy over another.  The challenge with this approach is that you need to understand these risks that you're taking on, and the risks that you're mitigating, with the techniques that you adopt.  Therein lies the rub, because the purveyors of the various process religions ( oops I mean methodologies) rarely seem to coherently the discuss the risks which people take on (and there's always risk) when following their dogma (oops, I mean sage advice).

For example, consider the risks associated with the various strategies for initially specifying requirements or design.  At the one extreme we have the traditional strategy of writing initial detailed speculations, more on this term in a minute, and at the other extreme we have the strategy of just banging out code.  In between are Agile Modeling (AM) strategies such as requirements envisioning and architecture envisioning (to name a few AM strategies).  Traditionalists will often lean towards the former approach, particularly when several agile scaling factors apply, whereas disciplined agile developers will lean towards initial envisioning.  There are risks with both approaches.

Let's consider the risks involved with writing detailed speculations (there's that term again):

1. You're speculating, not specifying.  There is clearly some value with doing some up-front requirements or architecture modeling, although the data regarding the value of modeling is fairly slim (there is a lot of dogma about it though), but that value quickly drops off in practice.  However, the more you write the greater the chance that you're speculating what people want (when it comes to requirements) or how you're going to build it (when it comes to architecture/design).  Traditionalists will often underestimate the risks that they're taking on when they write big requirements up front (BRUF)  , or create big models up front (BMUF) in general, but in the case of BRUF the average is that a large percentage of the functionality produced is never used in practice -- this is because the detailed requirements "specifications" contained many speculations as to what people wanted, many of which proved to be poor guesses in practice.
2. You're effectively committing to decisions earlier than you should.  A side effect of writing detailed speculations is that by putting in the work to document, validate, and then update the detailed speculations the decisions contained in the speculations become firmer and firmer.  You're more likely to be willing to change the content of a two-page, high-level overview of your system requirements than you are to change the content of a 200-page requirements speculation that has been laboriously reviewed and accepted by your stakeholders.   In effect the decision of what should be built gets "carved in stone" early in the process.  One of the principles of lean software development is to defer decisions as late as possible, only making them when you need to, thereby maximizing your flexibility.  In this case by making requirements decisions early in the process through writing detailed speculations, you reduce your ability to deliver functionality which meets the actual needs of your stakeholders, thereby increasing project risk.
3. You're increasing communication risk.  We've known for decades that of all the means of communication that we have available to us, that sharing documentation with other people is the riskiest and least effective strategy available to us for communicating information (face-to-face communication around a shared sketching environment is the most effective).  At scale, particularly when the team is large or the team geographically distributed, you will need to invest a little more time producing specifications then when the team is co-located, to reduce the inherent risks associated with those scaling factors, but that doesn't give you license to write huge tomes.  Agile documentation strategies still apply at scale.  Also, if you use more sophisticated tooling you'll find it easier to promote collaboration on agile teams at scale.
4. You're traveling heavy.  Extreme Programming (XP) popularized the concept of traveling light. The basic idea is that any artifact that you create must be maintained throughout the rest of the project (why create a document if you have no intention of keeping it up to date).  The implication is that the more artifacts you create the slower you work due to the increased maintenance burden.

There are also risks involved with initial envisioning:

1. You still need to get the details.  Just because you're not documenting the details up front doesn't imply that you don't need to understand them at some point.  Agile Modeling includes several strategies for exploring details throughout the agile system development life cycle (SDLC), including iteration modeling performed at the beginning of each iteration as part of your overall iteration planning activities, just in time (JIT) model storming throughout the iteration, and test-driven development (TDD) for detailed JIT executable specification.
2. You need access to stakeholders. One of the fundamental assumptions of agile approaches is that you'll have active stakeholder participation throughout a project.  You need to be able to get information from your stakeholders in a timely manner for the previously listed AM  techniques to work effectively.  My experience is that this is fairly straightforward to achieve if you educate the business as to the importance of doing so and you stand up and fight for it when you need to.  Unfortunately many people don't insist on access to stakeholders and put their projects at risk as a result.
3. You may still need some documented speculations.  As noted previously you may in fact need to invest in some specifications, particularly at scale, although it's important to recognize the associated risks in doing so.  For example, in regulatory compliance situations you will find that you need to invest more in documented speculations simply to ensure that you fulfill your regulatory obligations (my advice, as always, is to read the regulations and then address them in a practical manner).

The ways that you approach exploring requirements, and formulating architecture/design, are important success criteria regardless of your process religion/methodology.  No strategy is risk free, and every strategy makes sense within given criteria.  As an IT professional you need to understand the risks involved with the various techniques so that you can make the trade-offs best suited for your situation.  One process size does not fit all.

My final advice is to take a look at the Disciplined Agile Delivery (DAD) framework as it provides a robust strategy for addressing the realities of agile software development in enterprise settings.

• The Future of the Unified Process Virtual Panel 1
• The Future of the Unified Process Virtual Panel 2

• Not providing sufficient training, education, and mentoring
• Not supporting pair programming
• Not reducing the creation of non-executable detailed speculations early in the project
• Not reducing the confirmatory testing being done by their independent QA/testing team
• Completely reducing initial requirements envisioning and architecture envisioning
• Completely reducing parallel independent testing

1. You adopt "standard" agile discipline.   Aspects of agile which require discipline include adopting practices such as test-driven development (TDD), active stakeholder participation, working collaboratively, shortening the feedback cycle, and many more.  These strategies are a great start to becoming disciplined IT professionals. 
2. You take a goal-driven approach.  When we first started working on the DAD framework I didn't want to create yet another prescriptive framework, particularly given Rational's track record with the Rational Unified Process (RUP) framework.  Rational has been pilloried for years for the prescriptive nature of RUP, which is unfortunate because there are a lot of great ideas in RUP that agile teams can benefit from, some of which we adopted in DAD and many of which are being actively reinvented with the agile community even as you read this. Furthermore, there are many prescriptive elements of the Scrum method that can get teams in trouble.  For example, Scrum prescribes that you hold a daily stand up meeting, often called a Scrum meeting, where everyone should answer three questions.  That's a great approach for teams new to agile, but it proves problematic in many situations due to it's prescriptive nature. Do you really need to do this once a day?  I've been on teams where we held coordination meetings twice a day and others only once a week.  Do you really need to stand up?  I've been on geographically distrubited agile teams where many of us were sitting down during coordination calls.  Do you really need to answer three questions, two of which are clearly focused on status regardless of claims otherwise?  I've been on lean teams where we met around our Kanban board and focused on potential blockers.  The answers to these questions depends on the context of the situation you find yourself in.  The challenge, at least from the point of view of a process framework, is how do you avoid falling into the trap of being overly prescriptive.  The strategy we adopted in DAD is to take a goal-driven approach.  The observation is that regardless of the situation you find yourself in there are common goals your team will need to fulfill.  For example, at the beginning of a project common goals include developing an initial plan, initially exploring the scope, initially identifying a technical strategy, and securing initial funding (amongst others).  Throughout construction you should coordinate your activities, improve the quality of your ecosystem, and produce a potentially consumable solution on a regular basis (more on this below).  So, instead of prescribing a daily stand up meeting the DAD framework instead indicates you should coordinate your activities, and gives several options for doing so (one of which is a Scrum meeting).  More importantly DAD describes the advantages and disadvantages of your options so that you can make the choice that's best suited for the situation your team finds itself in (see this blog posting for a detailed example of the types of tables included in the DAD book to help you through such process tailoring decisions).  In short, our experience is that it requires discipline to take a goal driven approach to agile delivery over the prescriptive strategies in other agile processes. 
3. You take a context-driven approach.  There are many tailoring factors, which I describe in the Software Development Context Framework (SDCF), which you need to consider when making process, tooling, and team structure decisions.  For example, a large team will adopt a different collection of practices and tools than a small team.  A geographically distributed team will adopt a different strategy than a team that is co-located.  You get the idea.  Other tailoring factors include compliance, team culture, organization culture, technical complexity, domain complexity, and project type.  It requires discipline to recognize the context of the situation you find yourself in and then act accordingly.
4. You deliver potentially consumable solutions.  One of the observations that we made early in the development of the DAD framework was that disciplined agile teams produce potentially consumable solutions, not just potentially shippable software.  Although delivery of high-quality, working software is important it is even more important that we deliver high-quality working solutions to our stakeholders. For example, not only are we writing software but we may also be updating the hardware on which it runs, writing supporting documentation, evolving the business processes around the usage of the system, and even evolving the organizational structure of the people working with the system.  In other words, disciplined agilists focus on solutions over software.  Furthermore, "potentially shippable" isn't sufficient: not only should it be shippable but it should also be usable and should be something people want to use.  In other words it should be consumable (a concept DAD adopted from IBM's Outside In Development).  Minimally IT professionals should have the skills and desire to produce good software, but what they really need are the skills and desire to provide good solutions. We need strong technical skills, but we also need strong "soft skills" such as user interface design and process design to name just two. The incremental delivery of potentially consumable solutions on an incremental basis requires discipline to do successfully.  DAD teams focus on repeatable results not repeatable processes.
5. You are enterprise aware.   Whether you like it or not, as you adopt agile you will constrained by the organizational ecosystem, and you will need to act accordingly. It takes discipline to work with enterprise professionals such as enterprise architects, data admistrators, portfolio managers, or IT governance people who may not be completely agile yet, and have the patience to help them. It takes discipline to work with your operations and support staff in a DevOps manner throughout the lifecycle, particularly when they may not be motivated to do so.  It requires discipline to accept and potentially enhance existing corporate development conventions (programming guidelines, data guidelines, UI guidelines, ...).  It requires discipline to accept that your organization has an existing technology roadmap that you should be leveraging, building out, and in some cases improving as you go.  In short, enterprise awareness requires a level of discipline not typically seen on many agile teams.
6. You adopt a full delivery lifecycle.  Empirically it is very easy to observe that at the beginning of an agile project there are some activities that you need to perform to initiate the project.  Similarly at the end of the project there are activities that you need to perform to release the solution into production or the marketplace.  The DAD process framework addresses the effort required for the full delivery effort, including project initiation, construction, and deployment.  Our experience is that it requires discipline on the part of IT professionals to include explicit phases for Inception/Initation, Construction, and Transition/Deployment and more importantly to focus the appropriate amount of effort on each.  One danger of explicit phases is that you run the risk of taking what's known as a Water-Scrum-Fall approach, a term coined by Dave West the person who wrote the forward for the DAD book, where you take an overly heavy/traditional approach to inception and transition in combination with a lighter agile approach to construction.  Water-Scrum-Fall occurs because many organizations haven't made a full transition to agile, often because they think it's only applicable to construction.  Our experience is that you can be very agile in your approach to inception and transition, experience we've built into the DAD framework.  Having said that it clearly requires discipline to keep inception activities short and similarly it requires discipline to reduce the "transition phase" to an activity. 
7. You adopt a wider range of roles.  An interesting side effect of adopting a full delivery lifecycle is that you also need to adopt a more robust set of roles.  For example, the Scrum method suggests three roles - Scrum Master, Product Owner, and Team Member - a reflection of the Scrum lifecycle's construction focus.  DAD suggests three primary roles - Team Lead, Product Owner, Team Member, Architecture Owner, and Stakeholder - as well as five secondary roles which may appear at scale.   
8. You embrace agile governance.  Governance establishes chains of responsibil­ity, authority and communication in support of the overall enterprise’s goals and strategy. It also establishes measurements, policies, standards and control mechanisms to enable people to carry out their roles and responsibilities effectively. You do this by balancing risk versus return on investment (ROI), setting in place effective processes and practices, defining the direction and goals for the department, and defining the roles that people play with and within the department.  It requires discipline to adopt an agile approach to governance, and that's something built right into the DAD framework.

1. Projects where the stakeholders are willing to actively work with you.
2. IT people who are flexible, willing to try new things, and willing to collaborate with one another.
3. IT people who are generalizing specialists, or at least willing to become so.
4. Finding a project which is of medium complexity (therefore it's "real" in the sense that it's significant to your organization) but not one where it can make or break your organization (therefore it's safe to experiment with).

1. Being agile may not have been the primary determinant of success.  You set up an environment where you have a good relationship with your stakeholders, where you have good people who want to work together, and the project is challenging but not impossible.  Oh, and by the way you adopted a few agile techniques as well.  Sounds to me that situation you could have adopted a few not-so-agile techniques instead and still succeed.  Although my various project success surveys, see my IT surveys page for details, have shown time and again that agile project teams are more successful than traditional project teams I haven't been able to tease out (yet) whether this success is attributable to agile or just attributable to improved project initiation efforts.
   
2. When adopting agile/lean widely across your organization, you can't cherry pick any more.  For the past few years I've been working with IT organizations that are in the process of adopting agile/lean strategies across their entire organization, not just across a few pilot projects.  What these organizations are finding is that they need to find ways to adopt agile where the business isn't as willing to work with IT, where some of the people aren't so flexible or collaborative, where some of the people are narrowly specialized and not as willing to expand their skills, or where the project exhibits scaling factors which motivates you to tailor your agile approach.  It's harder to succeed with agile in these situations because they're not as "cherry" as what you've experienced previously.  Luckily, if you've been successful previously then you now have some agile experienced people, you have successes to reference, and you've likely overcome some problems even in the cherry situations which you have learned from.  So, your cherry successes will hopefully improve your ability to succeed even in "non cherry" situations.
   
3. You need to work smarter, not harder. If the source of your success was actually from improved project initiation practices and not from agile, then recognize that and act accordingly.  Realistically part of your success was from that and part was from agile, and the organizations that adopt a measured improvement approach potentially have the data to determine which practices lead to success and which didn't.  Without the metrics you're effectively flying blind when it comes to deciding how to improve.  There is clearly a mandate for smarter work practices within IT, within your organization as a whole for that matter.