Agility@Scale: Strategies for Scaling Agile Software Development

When I talk to people about scaling agile techniques, or about agile software development in general, I often put describe strategies in terms of various risks.  I find that this is an effective way for people to understand the trade-offs that they're making when they choose one strategy over another.  The challenge with this approach is that you need to understand these risks that you're taking on, and the risks that you're mitigating, with the techniques that you adopt.  Therein lies the rub, because the purveyors of the various process religions ( oops I mean methodologies) rarely seem to coherently the discuss the risks which people take on (and there's always risk) when following their dogma (oops, I mean sage advice).

For example, consider the risks associated with the various strategies for initially specifying requirements or design.  At the one extreme we have the traditional strategy of writing initial detailed speculations, more on this term in a minute, and at the other extreme we have the strategy of just banging out code.  In between are Agile Modeling (AM) strategies such as requirements envisioning and architecture envisioning (to name a few AM strategies).  Traditionalists will often lean towards the former approach, particularly when several agile scaling factors apply, whereas disciplined agile developers will lean towards initial envisioning.  There are risks with both approaches.

Let's consider the risks involved with writing detailed speculations (there's that term again):

1. You're speculating, not specifying.  There is clearly some value with doing some up-front requirements or architecture modeling, although the data regarding the value of modeling is fairly slim (there is a lot of dogma about it though), but that value quickly drops off in practice.  However, the more you write the greater the chance that you're speculating what people want (when it comes to requirements) or how you're going to build it (when it comes to architecture/design).  Traditionalists will often underestimate the risks that they're taking on when they write big requirements up front (BRUF)  , or create big models up front (BMUF) in general, but in the case of BRUF the average is that a large percentage of the functionality produced is never used in practice -- this is because the detailed requirements "specifications" contained many speculations as to what people wanted, many of which proved to be poor guesses in practice.
2. You're effectively committing to decisions earlier than you should.  A side effect of writing detailed speculations is that by putting in the work to document, validate, and then update the detailed speculations the decisions contained in the speculations become firmer and firmer.  You're more likely to be willing to change the content of a two-page, high-level overview of your system requirements than you are to change the content of a 200-page requirements speculation that has been laboriously reviewed and accepted by your stakeholders.   In effect the decision of what should be built gets "carved in stone" early in the process.  One of the principles of lean software development is to defer decisions as late as possible, only making them when you need to, thereby maximizing your flexibility.  In this case by making requirements decisions early in the process through writing detailed speculations, you reduce your ability to deliver functionality which meets the actual needs of your stakeholders, thereby increasing project risk.
3. You're increasing communication risk.  We've known for decades that of all the means of communication that we have available to us, that sharing documentation with other people is the riskiest and least effective strategy available to us for communicating information (face-to-face communication around a shared sketching environment is the most effective).  At scale, particularly when the team is large or the team geographically distributed, you will need to invest a little more time producing specifications then when the team is co-located, to reduce the inherent risks associated with those scaling factors, but that doesn't give you license to write huge tomes.  Agile documentation strategies still apply at scale.  Also, if you use more sophisticated tooling you'll find it easier to promote collaboration on agile teams at scale.
4. You're traveling heavy.  Extreme Programming (XP) popularized the concept of traveling light. The basic idea is that any artifact that you create must be maintained throughout the rest of the project (why create a document if you have no intention of keeping it up to date).  The implication is that the more artifacts you create the slower you work due to the increased maintenance burden.

There are also risks involved with initial envisioning:

1. You still need to get the details.  Just because you're not documenting the details up front doesn't imply that you don't need to understand them at some point.  Agile Modeling includes several strategies for exploring details throughout the agile system development life cycle (SDLC), including iteration modeling performed at the beginning of each iteration as part of your overall iteration planning activities, just in time (JIT) model storming throughout the iteration, and test-driven development (TDD) for detailed JIT executable specification.
2. You need access to stakeholders. One of the fundamental assumptions of agile approaches is that you'll have active stakeholder participation throughout a project.  You need to be able to get information from your stakeholders in a timely manner for the previously listed AM  techniques to work effectively.  My experience is that this is fairly straightforward to achieve if you educate the business as to the importance of doing so and you stand up and fight for it when you need to.  Unfortunately many people don't insist on access to stakeholders and put their projects at risk as a result.
3. You may still need some documented speculations.  As noted previously you may in fact need to invest in some specifications, particularly at scale, although it's important to recognize the associated risks in doing so.  For example, in regulatory compliance situations you will find that you need to invest more in documented speculations simply to ensure that you fulfill your regulatory obligations (my advice, as always, is to read the regulations and then address them in a practical manner).

The ways that you approach exploring requirements, and formulating architecture/design, are important success criteria regardless of your process religion/methodology.  No strategy is risk free, and every strategy makes sense within given criteria.  As an IT professional you need to understand the risks involved with the various techniques so that you can make the trade-offs best suited for your situation.  One process size does not fit all.

My final advice is to take a look at the Disciplined Agile Delivery (DAD) framework as it provides a robust strategy for addressing the realities of agile software development in enterprise settings.