I have a young daughter and she's at the age where she wants to dress herself. The problem is that if we pick a single outfit and try to get her to wear it she refuses (I've lost count of the times I've heard "I don't want that"). At the other extreme if we let her pick her own outfit from the closet she'll be there for hours trying everything on. As experienced parents advise what we need to do is present her with two or three choices and ask her to pick what she wants.
So how does this relate to software development? Once again, let's look at extremes. First, consider Scrum's approach of prescribing a single way of doing things. For example, Scrum prescribes that you hold a daily meeting, called a Scrum, where everyone stands up and answers the same 3 questions. Scrum also prescribes a single change management strategy where you have a stack of requirements prioritized by business value. Scrum prescribes three roles - ScrumMaster, Product Owner, and Team Member - as well as other things. Don't get me wrong, these strategies are all great in certain circumstances but not for all. Prescribing one way of doing things is an extreme, so perhaps we shouldn't be surprised when people refuse to do it that way or struggle to make it work given the situation that they face.
At the other extreme consider RUP's approach where it presents repository of techniques from which to select the ones appropriate for you. The problem is that now we have an overwhelming way of doing things from which to choose, all of them good options in certain situations. So why are we surprised when teams struggle to identify a coherent tailoring of RUP?
Now let's consider the middle ground. The Disciplined Agile Delivery (DAD) process decision framework takes a goals-driven approach. So, instead of saying "hold a daily stand up meeting and answer these three questions" it says to regularly coordinate within the team and there are several ways of doing so (hold a Scrum meeting, hold a Kanban-style meeting, and so on). Yes, DAD does provide a large number of techniques to choose from (as does the agile community in general) but it also provides a straightforward way to choose between them. DAD does this by describing the advantages and disadvantages of each technique and suggests when, and when not, to use each approach. When people are presented with viable options, and the trade-offs associated with each, it's much more likely that they'll choose an approach that is better suited for their situation.
Scrum's single prescribed strategy works well only when that strategy is appropriate for the situation at hand. Similarly, telling my daughter exactly what to wear works well only when she's in the mood to wear that outfit. RUP's cafeteria approach to software process works well when you have the expertise, and time, to choose what's best for you. Similarly, asking my daughter to pick out her outfit from all the choices in her closet only works well when I've got a lot of time to wait for her. In both situations a better strategy is to present options, describe the trade offs, and then let people pick what's right for them given the context of the situation that they face. This is exactly what the DAD framework promotes.
I believe the goals-based approach of Disciplined Agile Delivery (DAD) represents an important step forward in the software process realm. It's time to recognize the extremes for what they are and move to a more viable middle ground.
A recurring discussion that I have with experienced agile developers is what it means to take a disciplined agile approach. The conversation usually starts off by some saying "but it already requires discipline to do agile", something that I fully agree with, followed by "therefore 'disciplined agile' is merely a marketing term", something which I don't agree with. The challenge with the "standard" agile discipline is that it is often focused on construction activities within a single project team, clearly important but also clearly not the full picture. There's more to an agile project than construction, and there's more to most IT departments than a single development project. In short, there are many opportunities for IT professionals to up their discipline, and thereby up their effectiveness, opportunities which we make explicit in the Disciplined Agile Delivery (DAD) framework.
Let's explore the many aspects to taking a disciplined agile approach:
You adopt "standard" agile discipline
. Aspects of agile which require discipline
include adopting practices such as test-driven development (TDD), active stakeholder participation, working collaboratively, shortening the feedback cycle
, and many more. These strategies are a great start to becoming disciplined IT professionals.
You take a goal-driven approach
. When we first started working on the DAD framework I didn't want to create yet another prescriptive framework, particularly given Rational's track record with the Rational Unified Process (RUP) framework. Rational has been pilloried for years for the prescriptive nature of RUP, which is unfortunate because there are a lot of great ideas in RUP that agile teams can benefit from, some of which we adopted in DAD and many of which are being actively reinvented with the agile community even as you read this. Furthermore, there are many prescriptive elements of the Scrum method that can get teams in trouble. For example, Scrum prescribes that you hold a daily stand up meeting, often called a Scrum meeting, where everyone should answer three questions. That's a great approach for teams new to agile, but it proves problematic in many situations due to it's prescriptive nature. Do you really need to do this once a day? I've been on teams where we held coordination meetings twice a day and others only once a week. Do you really need to stand up? I've been on geographically distrubited agile teams where many of us were sitting down during coordination calls. Do you really need to answer three questions, two of which are clearly focused on status regardless of claims otherwise? I've been on lean teams where we met around our Kanban board and focused on potential blockers. The answers to these questions depends on the context of the situation you find yourself in. The challenge, at least from the point of view of a process framework, is how do you avoid falling into the trap of being overly prescriptive. The strategy we adopted in DAD is to take a goal-driven approach. The observation is that regardless of the situation you find yourself in there are common goals your team will need to fulfill. For example, at the beginning of a project common goals include developing an initial plan, initially exploring the scope, initially identifying a technical strategy, and securing initial funding (amongst others). Throughout construction you should coordinate your activities, improve the quality of your ecosystem, and produce a potentially consumable solution on a regular basis (more on this below). So, instead of prescribing a daily stand up meeting the DAD framework instead indicates you should coordinate your activities, and gives several options for doing so (one of which is a Scrum meeting). More importantly DAD describes the advantages and disadvantages of your options so that you can make the choice that's best suited for the situation your team finds itself in (see this blog posting
for a detailed example of the types of tables included in the DAD book to help you through such process tailoring decisions). In short, our experience is that it requires discipline to take a goal driven approach
to agile delivery over the prescriptive strategies in other agile processes.
You take a context-driven approach
. There are many tailoring factors, which I describe in the Software Development Context Framework (SDCF)
, which you need to consider when making process, tooling, and team structure decisions. For example, a large team will adopt a different collection of practices and tools than a small team. A geographically distributed team will adopt a different strategy than a team that is co-located. You get the idea. Other tailoring factors include compliance, team culture, organization culture, technical complexity, domain complexity, and project type. It requires discipline to recognize the context of the situation you find yourself in and then act accordingly.
You deliver potentially consumable solutions
. One of the observations that we made early in the development of the DAD framework was that disciplined agile teams produce potentially consumable solutions, not just potentially shippable software. Although delivery of high-quality, working software is important it is even more important that we deliver high-quality working solutions to our stakeholders. For example, not only are we writing software but we may also be updating the hardware on which it runs, writing supporting documentation, evolving the business processes around the usage of the system, and even evolving the organizational structure of the people working with the system. In other words, disciplined agilists focus on solutions over software
. Furthermore, "potentially shippable" isn't sufficient: not only should it be shippable but it should also be usable and should be something people want to use. In other words it should be consumable (a concept DAD adopted from IBM's Outside In Development
). Minimally IT professionals should have the skills and desire to produce good software, but what they really need are the skills and desire to provide good solutions. We need strong technical skills, but we also need strong "
such as user interface design and process design to name just two.
The incremental delivery of potentially consumable solutions on an incremental basis requires discipline
to do successfully. DAD teams focus on repeatable results not repeatable processes
You are enterprise aware
. Whether you like it or not, as you adopt agile you will constrained by the organizational ecosystem, and you will need to act accordingly. It takes discipline to work with enterprise professionals such as enterprise architects, data admistrators, portfolio managers, or IT governance people who may not be completely agile yet, and have the patience to help them. It takes discipline to work with your operations and support staff in a DevOps
manner throughout the lifecycle, particularly when they may not be motivated to do so. It requires discipline to accept and potentially enhance existing corporate development conventions (programming guidelines, data guidelines, UI guidelines, ...). It requires discipline to accept that your organization has an existing technology roadmap that you should be leveraging, building out, and in some cases improving as you go. In short, enterprise awareness requires a level of discipline
not typically seen on many agile teams.
You adopt a full delivery lifecycle
. Empirically it is very easy to observe that at the beginning of an agile project there are some activities that you need to perform to initiate the project. Similarly at the end of the project there are activities that you need to perform to release the solution into production or the marketplace. The DAD process framework addresses the effort required for the full delivery effort, including project initiation, construction, and deployment. Our experience is that it requires discipline on the part of IT professionals to include explicit phases
for Inception/Initation, Construction, and Transition/Deployment and more importantly to focus the appropriate amount of effort on each. One danger of explicit phases is that you run the risk of taking what's known as a Water-Scrum-Fall
approach, a term coined by Dave West the person who wrote the forward for the DAD book, where you take an overly heavy/traditional approach to inception and transition in combination with a lighter agile approach to construction. Water-Scrum-Fall occurs because many organizations haven't made a full transition to agile, often because they think it's only applicable to construction. Our experience is that you can be very agile in your approach to inception and transition, experience we've built into the DAD framework. Having said that it clearly requires discipline to keep inception activities short
and similarly it requires discipline to reduce the "transition phase" to an activity
You adopt a wider range of roles
. An interesting side effect of adopting a full delivery lifecycle is that you also need to adopt a more robust set of roles. For example, the Scrum method suggests three roles - Scrum Master, Product Owner, and Team Member - a reflection of the Scrum lifecycle's construction focus. DAD suggests three primary roles - Team Lead, Product Owner, Team Member, Architecture Owner
, and Stakeholder - as well as five secondary roles which may appear at scale.
You embrace agile governance
. Governance establishes chains of responsibility, authority and communication in support of the overall enterprise’s goals and strategy. It also establishes measurements, policies, standards and control mechanisms to enable people to carry out their roles and responsibilities effectively. You do this by balancing risk versus return on investment (ROI), setting in place effective processes and practices, defining the direction and goals for the department, and defining the roles that people play with and within the department. It requires discipline to adopt an agile approach to governance
, and that's something built right into the DAD framework.
In previous blog postings I've defined Discipline Agile Delivery (DAD)
but I haven't shared the lifecycle (although I have done so in a couple of white papers, most recently Scaling Agile: An Executive Guide
) in this blog. Until now.
The DAD method combines strategies and practices from several software methods, including Scrum
, Extreme Programming (XP)
, Agile Modeling
, Agile Data
, and the Open Unified Process (OpenUP)
. One aspect of this "process idea reuse" is that the DAD extends the Scrum lifecycle to be a full-fledged delivery lifecycle. Figure 1 overviews the Scrum lifecycle, which addresses the construction aspects of agile delivery quite well. It depicts the product backlog which is a basic prioritized requirements stack
; the concept of delivering incrementally in consistent time boxes (what Scrum calls sprints and other methods such as DAD call iterations); holding a daily coordination meeting; and demoing your work at the end of the sprint/iteration/timebox to get feedback from key stakeholders. These are all great ideas, which is why the DAD method adopts and enhances them.Figure 1: The Scrum lifecycle. The focus is on the construction part of the lifecycle (click on the diagram for details).
Several years ago I wrote about the Agile lifecycle
and the need to extend it beyond construction, and this thinking is reflected in the DAD lifecycle of Figure 2. The DAD lifecycle extends the Scrum lifecycle in several important ways:
- It includes explicit phases. Sacrilege! Rhetoric aside, there are in fact serial aspects to agile software development. Project teams go through an initiation effort - called "warm up" in Eclipse Way, Inception in Unified Process, Sprint 0 in Scrum, and Iteration 0 in other methods - at the beginning of a project. Eventually the team will go through a release phase - called the "end game" in Eclipse Way
- It includes project initiation. During the Inception phase you perform basic project initiation activities such as requirements envisioning, architecture envisioning, initial release planning, getting funding for the project, and starting to build the team (among other tasks). In the summer of 2009 I ran the 2009 Agile Project Initiation survey which revealed that the average agile team spent 4 week on initiation activities such as this, that 89% did some sort of up-front requirements work, and that 85% did some sort of up-front architecture work.
- It includes release activities. The DAD lifecycle also includes a Transition phase where you release your solution into production or the marketplace. These activities typically include final testing and hardening of your solution, training end users, pilot/beta testing, finalizing documentation, running the solution in parallel with the system(s) being replaced, and so on.
- It explicitly indicates production activities. Many agile delivery teams are responsible for supporting the existing version(s) of their system which are currently running in production. Because there are existing versions, the team will be getting enhancement requests and defect reports from their operations and support people. Many of these requests can simply be treated as new requirements to be prioritized and put on the work item stack. Some need to be addressed right away, particularly "severity 1" defects, which requires the delivery team to "stop the line", address the problem, release a patch or hot fix, then reintegrate the changes into the version they're currently working on. My experience is that it's critical to include a production phase in your delivery lifecycle to make it explicit to the team that they need to take operations and support concerns into account.
- It explicitly enhances the product backlog. The product backlog has evolved into a work item list (or work item stack if you prefer). Disciplined agile teams put more than just requirements on the stack, as you read above it is common to treat defect reports as you do requirements.
- It explicitly includes critical milestone points. An important aspect of the DAD method is that not only does it support self organization such as Scrum but it does so within an appropriate governance framework. Disciplined agile teams recognize that they exist within a larger organization, that they should follow common development guidelines, that they should strive to leverage and build out the shared enterprise infrastructure, that they must report common metrics to senior management (hopefully automatically via the use of instrumented tooling such as we see on the Jazz platform).
One of the advantages of the DAD method over Scrum is that it doesn't require you to figure these common things out for yourself. Figure 2. The Disciplined Agile Delivery (DAD) lifecycle (basic). The focus is on the delivery portion of the system lifecycle, from starting a project to releasing the solution into production (click on the diagram for details).
Figure 3 depicts the advanced form of the DAD lifecycle. This form of the lifecycle occurs on experienced teams that have originally adopted the basic, Scrum-based lifecycle of Figure 2 and evolved it over time to be leaner. One of the implications of continuous improvement and continuous learning, key recommendations of the DAD process framework, is that your lifecycle will evolve. Primary changes that you see between Figure 2 and Figure 3 are the adoption of a work item pool
instead of a work item stack
and the abandonment of iterations (the critical observation is that practices such as detailed planning, demos, retrospectives and so on do not need to be on the same cadence, hence iterations disappear).
Figure 3. The Disciplined Agile Delivery (DAD) lifecycle (advanced). Over time you will improve your strategy towards a leaner approach (click on the diagram for details).
Figure 4. An older version of the basic DAD lifecycle using Scrum terminology. Use whatever terminology you're comfortable with, there is no "standard" (click on the diagram for details).
One of the differences that you may have noticed between the Scrum lifecycle of Figure 1 and the DAD lifecycle of Figure 2 is different terminology is used. For example, iteration is used instead of sprint. Work item list is used instead of product backlog. Although I believe that the terminology used in Figure 2 is more accurate, it really doesn't matter because it's easy to use Scrum terminology with DAD if you like. You can see this clearly in Figure 4. What is important is that you choose the terminology which you are most comfortable with, and be prepared to translate back and forth between terms as there are no standards and there never will be. As an aside, you might find Translating Scrum Terminology
to be of interest.
DAD is still a process framework that you'll need to tailor to meet your unique needs. More on this in future blog postings.
At IBM Rational we define disciplined agile delivery as:
Disciplined agile delivery is an evolutionary (iterative and incremental) approach which regularly produces high quality solutions in a cost effective and timely manner via a risk and value driven life cycle. It is performed in a highly collaborative, disciplined, and self-organizing manner within an appropriate governance framework, with active stakeholder participation to ensure that the team understands and addresses the changing needs of its stakeholders to maximize business value provided. Disciplined agile delivery teams provide repeatable results by adopting just the right amount of ceremony for the situation which they face.
Let’s explore the key points in this definition:
- Full delivery life cycle. Disciplined agile delivery processes have life cycles which are serial in the large and iterative in the small. Minimally they have a release rhythm which recognizes the need for start up/inception activities, construction activities, and deployment/transition activities. Better yet, they include explicit phases as well. It is very important to note that these are not the traditional waterfall phases – requirements, analysis, design, and so on – but instead different “seasons” of a project. The point is that we need to look beyond agile software development and consider the full complexities of solution delivery. Adopting a full delivery life cycle, not just a construction life cycle, is arguably the “zeroth” agile scaling factor.
- Evolutionary. Agile strategies are both iterative and incremental in nature. Iterative means that you are working in a non-serial manner, on any given day you may do some requirements analysis, some testing, some programming, some design, some more testing, and so on. Incremental means that you add new functionality and working code to the most recent build, until such time as the stakeholder determines there is enough value to release the product.
- Regularly produces high quality solutions. Agilists are said to be quality focused. They prefer to test often and early, and the more disciplined ones even take a test-first approach where they will write a single test and the just enough production code to fulfill that test (then they iterate). Many agile developers have adopted the practice of refactoring, which is a technique where you make simple changes to your code or schema which improves its quality without changing its semantics. Adoption of these sorts of quality techniques seems to work – it appears that agile teams are more likely to deliver high quality systems than traditional teams (according to the DDJ 2008 Project Success survey). Within IBM we take it one step further and focus on consumability, which encompasses quality and other features such as ease of deployment and system performance. Furthermore, although some agile methods promote the concept of producing “potentially shippable software” on a regular basis, disciplined agile delivery teams produce solutions: a portion of which may be software, a portion of which may be hardware, and a portion of which will be the manner in which the system is used.
- Cost effective and timely manner. Agile teams prefer to implement functionality in priority order [http://www.agilemodeling.com/essays/prioritizedRequirements.htm], with the priority being defined by their stakeholders (or a representative thereof). Working in priority order enables agile teams to maximize the return on investment (ROI) because they are working on the high-value functionality as defined by their stakeholders, thereby increasing cost effectiveness. Agile teams also prefer to produce potentially shippable solutions each iteration (an iteration is a time-box, typically 2-4 weeks in length), enabling their stakeholders to determine when they wish to have a release delivered to them and thereby improving timeliness. Short iterations reduce the feedback cycle, improving the chance that agile teams will discover problems early (they “fail fast”) and thereby enable them to address the problems when they’re still reasonably inexpensive to do so. The DDJ 2008 Project Success survey found that agile teams are in fact more likely to deliver good ROI than traditional teams and more likely to deliver in a timely manner.
- Value driven life cycle. One result of building a potentially shippable solution every iteration is that agile teams produce concrete value in a consistent and visible manner throughout the life cycle.
- Risk and value driven life cycle. Core agile processes are very clear about the need to produce visible value in the form of working software on a regular basis throughout the life cycle. Disciplined agile delivery processes take it one step further and actively mitigate risk early in the life cycle – during project start up you should come to stakeholder concurrence regarding the project’s scope, thereby reducing significant business risk, and prove the architecture by building a working skeleton of your system, thereby significantly reducing technical risk. They also help with transition to agile, allowing traditional funding models to use these milestones before moving to the finer grained iteration based funding that agile allows.
- Highly collaborative. People build systems, and the primary determinant of success on a development project is the individuals and the way that they work together. Agile teams strive to work closely together and effectively as possible. This is a characteristic that applies to both engineers on the team, as well as their leadership.
- Disciplined. Agile software development requires greater discipline on the part of practitioners that what is typically required by traditional approaches.
- Self organizing. This means that the people who do the work also plan and estimate the work.
- Self-organization within an appropriate governance framework. Self-organization leads to more realistic plans and estimates which are more acceptable to the people implementing them. At the same time these self-organizing teams must work within an appropriate governance framework which reflects the needs of their overall organizational environment. An “appropriate governance framework” explicitly enables disciplined agile delivery teams to effectively leverage a common infrastructure, to follow organizational conventions, and to work towards organizational goals. The point is that project teams, regardless of the delivery paradigm they are following, need to work within the governance framework of their organization. More importantly, effective governance programs should make it desirable to do so. Our experience is that traditional, command-and-control approaches to governance where senior management explicitly tells teams what to do and how to do it don’t work very well with agile delivery teams. We’ve also found that lean development governance, an approach which is based on collaboration and enablement, is far more effective in practice. Good governance increases the chance that agile delivery teams will build systems which fit into your overall organizational environment, instead of yet another stand-alone system which increases your overall maintenance burden and data quality problems.
- Active stakeholder participation. Agile teams work closely with their stakeholders, who include end users, managers of end users, the people paying for the project, enterprise architects, support staff, operations stuff, and many more. Within IBM we distinguish between four categories of stakeholder: principles/sponsors, partners (business partners and others), end users, and insiders These stakeholders, or their representatives (product owners in Scrum, or on-site customers in Extreme Programming, or a resident stakeholder in scaling situations), are expected to provide information and make decisions in a timely manner.
- Changing needs of stakeholders. As a project progresses your stakeholders will gain a better understanding of what they want, particularly if you’re showing them working software on a regular basis, and will change their “requirements” as a result. Changes in the business environment, or changes in organization priority, will also motivate changes to the requirements. There is a clear need for agile requirements change management [http://www.agilemodeling.com/essays/changeManagement.htm] on modern IT projects.
- Repeatable results. Stakeholders are rarely interested in how you delivered a solution but instead in what you delivered. In particular, they are often interested in having a solution which meets their actual needs, in spending their money wisely, in a high-quality solution, and in something which is delivered in a timely manner. In other words, they’re interested in repeatable results, not repeatable processes.
- Right amount of ceremony for the situation. Agile approaches minimize ceremony in favor of delivering concrete value in the form of working software, but that doesn’t mean they do away with ceremony completely. Agile teams will still hold reviews, when it makes sense to do so. DDJ’s 2008 Modeling and Documentation Survey found that agile teams will still produce deliverable documentation, such as operations manuals and user manuals, and furthermore are just as likely to do so as traditional teams. The DDJ September 2009 State of the IT Union survey found that the quality of the documentation delivered by agile teams was just as good as that delivered by traditional teams, although iterative teams (e.g. RUP teams) did better than both agile and traditional.