When adopting agile software development
techniques across a large number of teams within your organization it is important to provide a definition for what agile software development is, in addition to criteria
for what it means to be agile. Many people will point to the four values of the Agile Manifesto
and claim that's a good definition. Well... it might be a good definition for the visionaries and early adopters among us, but for people on the right-hand side of the technology adoption chasm (the early majority, late majority and the laggards) this isn't enough. Don't get me wrong, I'm a firm believer in the agile values but I like to cast them as philosophies instead of as a definition.
At IBM Software Group, the definition of disciplined agile software delivery which we have been sharing with our customers is:Disciplined agile software delivery is an evolutionary (iterative and incremental) approach to delivery which regularly produces high quality software in a cost effective and timely manner. It is performed in a highly collaborative and self-organizing manner, with active stakeholder participation to ensure that the team understand and addresses the changing needs of its stakeholders. Disciplined agile delivery teams provide repeatable results by adopting just the right amount of ceremony for the situation which they face.
I think that this is a pretty good definition, although I have no doubt that we'll evolve it over time.
I also suspect that the agile community will never settle on a common definition for what agile is and more than likely are smart enough not to even try. ;-)Further reading:
The popular Agile literature can often seam naive when it comes to how Agilists work with project stakeholders:- Extreme Programming (XP) has a practice called On-Site Customer where one or more people work closely with your team to provide information and to make decisions in a timely manner.- Scrum has the role of Product Owner who is the one single person that the development team goes to for decisions about requirements. - Agile Modeling (AM) has the practice of Active Stakeholder Participation which extends On-Site Customer to get the stakeholder(s) actively involved with the modeling effort through the use of inclusive tools and techniques.
These are great strategies for small, co-located teams doing straightforward development, but they quickly fall apart at scale. This occurs for several reasons:1. Stakeholders are a diverse group. Your stakeholders include end users, business management, project funders, enterprise architects, operations staff, support staff, other system development teams, and many others. Different people have different, and often contradictory, requirements and they certainly have different priorities. It's questionable whether a single person, or a handful of persons, can adequately represent this diverse group.2. One person becomes a bottleneck. Even with a small co-located team this is a problem, let alone one that is geographically distributed or one that is very large. There's no way that a single person can be available 24/7 in a responsive manner to support distributed teams.3. It's a difficult role. The Product Owner/Customer (POC) is responsible for representing the business to the development team. They're making important decisions on a regular basis, decisions which they'll be held accountable for.4. One person becomes a serious project risk. Not only is it questionable whether a single person can fairly represent all stakeholders, even if they could what happens if you lose that person? They effectively become a single point of failure for your team.
To scale this role, consider the following strategies:1. Recognize the true scope of the POC role. Not only are they stakeholder proxies they also are a development team representative to the stakeholder community as a whole. As stakeholder proxies they'll make decisions and prioritize the work, they'll run requirements elicitation sessions, they'll negotiate priorities, and they'll put the development team in contact with stakeholders who have expertise in specific aspects of the domain. As team representatives they'll often demo the current version of the system to other stakeholders, communicate the status of the project to people, and respond to various requests for information from the stakeholders.2. Have multiple people in it. A single POC works well for small, co-located teams developing simple software. At scale you'll soon discover that you need multiple people in this role so that they don't become a bottleneck. For distributed teams it's common to see each subteam have one or more POCs who are managed by a primary/chief POC. The primary POC typically works on the coordinating team with the chief architect (I'll talk about this role in a future blog posting) and the program manager (also a topic for a future blog posting).3. Train them in business analysis skills. The person(s) in the POC role need good business analysis skills. If fact, it's common for people who were formerly BAs for traditional teams to step into the POC role, particularly with BAs who originally come from the business side of your organization. This strategy has its advantages and disadvantages. As a BA they've likely got solid business knowledge but their instincts may motivate them to take a documentation-driven approach to providing information to the development team instead of a collaboration-based approach. Be careful.4. Consider the full system development lifecycle. There's far more to the POC role than supporting the development team during Construction iterations. During "Iteration 0", the Inception phase for an Agile RUP project or the warm-up phase for an Eclipse Way project, the POC(s) will often lead the initial requirements envisioning efforts. The product backlog, or better yet your work item list, needs to come from somewhere after all. During the release iteration(s), the Transition phase for RUP or the End-Game phase for Eclipse Way, the POC(s) will focus on communicating the upcoming release to the stakeholder community, will be actively involved with any final user acceptance testing (UAT), and may even be involved with training end users.
In my January 2008 column in Dr Dobb's Journal, posted at http://www.ddj.com/architect/204801134 , I provide detailed advice about how to scale the way that you work with stakeholders on Agile projects by applying the practices of Agile Model Driven Development (AMDD). There's no magic solution, you just need to choose to organize yourself effectively. The good news is that you can easily work with stakeholders at scale.[Read More
Although it might not be obvious, and important success factor in adopting agile techniques
is to be able to determine whether a team is agile or not. The challenge that many organizations face is that many teams will claim to be agile, yet management, who often has little or no experience with agile approaches, cannot tell which claims are true and which are over zealous (I'm being polite). The following are the criteria that I suggest you look for in a disciplined agile team:1. Produce working software on a regular basis
. This is one of the 12 principles behind the Agile Manifesto
, and in my experience is a critical differentiator between the teams that are agile and those that are merely claiming it. Ideally the team should produce potentially shippable software each iteration. That doesn't mean that they'll deploy the system into production, or the marketplace, each iteration but they could if required to do so. Typically the team will deploy into a pre-production testing environment or a demo enviroment at the end of each iteration (or more often for that matter).2. Do continuous regression testing, and better yet take a Test-Driven Development (TDD) approach
. Agile developers test their work to the best of their ability, minimally doing developer regression testing via a continuous integration (CI) strategy
and better yet do developer-level TDD
. This approach enables development teams to find defects early, thereby reducing the average cost of addressing the defects, it also helps them to deliver higher quality code and to move forward safely when adding or changing functionality.3. Work closely with their stakeholders, ideally on a daily basis
. A common practice of agile teams is to have an on-site customer or product owner who prioritizes requirements and provides information on a timely manner to the team. Disciplined agile teams take it one step further and follow the practice active stakeholder participation
where the stakeholders get actively involved with modeling and sometimes even development.4. Are self-organizing within a governance framework
. Agile teams are self-organizing, which means that the people doing the work determines how the work will be done, they're not told by a manager who may not even be directly involved with the work how it will be done. In other words the team does its own planning, including scheduling and estimation. Disciplined agile teams are self governing within an effective governance framework
.5. Regularly reflect on how they work together and then act to improve on their findings
. Most agile teams hold a short meeting at the end of each iteration to reflect upon how well things are working and how they could potentially improve the way that they are working together. Sometimes this is done in a more formalized manner in the form of a retrospective
, but often it's done informally. The team then acts on one or more of their suggested improvements the next iteration. Disciplined agile teams take this one step further and measure their software process improvement (SPI) progress over time: the act of taking these measures, perhaps via a product such as Rational Self Check
, helps to keep the team on track in their SPI efforts.
I have yet to discover an ad-hoc development team which met all five criteria, and most of them rarely meet two or three.Further reading:
The explicit phases of the Unified Process -- Inception, Elaboration, Construction, and Transition -- and their milestones are important strategies for scaling agile software development to meet the real-world needs of modern organizations. Yes, I realize that this is heresy for hard-core agilists who can expound upon the evils of serial development, yet these very same people also take a phased approach to development although are loathe to admit it. The issue is that the UP phases are like seasons of a project: although you'll do the same types of activities all throughout a project, the extent to which you do them and the way in which you do them change depending on your goals. For example, at the beginning of a development project if you want to be effective you need to do basic things like identify the scope of the project, identify a viable architecture strategy, start putting together your team, and obtain support for the project. Towards the end of a project your focus is on the activities surrounding the deployment of your system into production, including end-of-lifecycle testing efforts, training, cleaning up of documentation, piloting the system with a subset of users, and so on. In between you focus on building the system, including analysis, design, testing, and coding of it. Your project clearly progresses through different phases, or call them seasons if the term phase doesn't suit you, whether your team is agile or not.
The UP defines four phases, each of which address a different kind of risk:1. Inception. This phase focuses on addressing business risk by having you drive to scope concurrence amongst your stakeholders. Most projects have a wide range of stakeholdres, and if they don't agree to the scope of the project and recognize that others have conflicting or higher priority needs you project risks getting mired in political infighting. In the Eclipse Way this is called the "Warm Up" iteration and in other agile processes "Iteration 0".2. Elaboration. The goal of this phase is to address technical risk by proving the architecture through code. You do this by building and end-to-end skeleton of your system which implements the highest-risk requirements. Some people will say that this approach isn't agile, that your stakeholders should by the only ones to prioritize requirements. Yes, I agree with that, but I also recognize that there are a wide range of stakeholders, including operations people and enterprise architects who are interested in the technical viability of your approach. I've also noticed that the high-risk requirements are often the high-business-value ones anyway, so you usually need to do very little reorganization of your requirements stack.3. Construction. This phase focuses on implementation risk, addressing it through the creation of working software each iteration. This phase is where you put the flesh onto the skeleton.4. Transition. The goal of this phase is to address deployment risk. There is usually a lot more to deploying software than simply copying a few files onto a server, as I indicated above. Deployment is often a complex and difficult task, one which you often need good guidance to succeed at.
Each phase ends with a milestone review, which could be as simple as a short meeting, where you meet with prime stakeholders who will make a "go/no-go" decision regarding your project. They should consider whether the project still makes sense, perhaps the situation has changed, and that you're addressing the project risks appropriately. This is important for "agile in the small" but also for "agile in the large" because at scale your risks are often much greater. Your prime stakeholders should also verify that you have in fact met the criteria for exiting the phase. For example, if you don't have an end-to-end working skeleton of your system then you're not ready to enter the Construction phase. Holding these sorts of milestone reviews improves your IT governance efforts by giving senior management valuable visibility at the level that they actually need: when you have dozens or hundreds of projects underway, you can't attend all of the daily stand up meetings of each team, nor do you even want to read summary status reports.
These milestone reviews enable you to lower project risk. Last Autumn I ran a survey via Dr. Dobb's Journal (www.ddj.com) which explore how people actually define success for IT projects and how successful we really were. We found that when people define success in their own terms that Agile has a 71% success rate compared with 63% for traditional approaches. Although it's nice to that Agile appears to be lower risk than traditional approaches, a 71% success rate still implies a 29% failure rate. The point is that it behooves us to actively monitor development projects to determine if they're on track, and if not either help them to get back on track or cancel them as soon as we possibly can. Hence the importance of occasional milestone reviews where you make go/no-go decisions. If you're interested in the details behind the project, they can be found at http://www.ambysoft.com/surveys/success2007.html .
Done right, phases are critical to your project success, particularly at scale. Yes, the traditional community seems to have gone overboard with phase-based approaches, but that doesn't mean that we need to make the same mistakes. Let's keep the benefit without the cost of needless bureaucracy.[Read More
Again and again I've seen IT organizations suffering from what I call the "Bureaucracy is Discipline" antipattern. For example, filling out forms and reviewing documents are both bureaucratic activities, neither of which require significant skill nor discipline to accomplish. However, agile practices such as developing potentially shippable software every iteration is easy to say but requires great discipline to accomplish. Respecting the decisions of your stakeholders, particularly those pertaining to requirements prioritization, is easy to talk about but proves to require great discipline in practice (particularly when you don't agree with a decision). It's easy to talk about taking a test-driven approach to development
, but in practice it requires significant skill and discipline to actually do.
A "process smell" which indicates that your organization is suffering from this antipattern is a focus on following repeatable processes instead of focusing on repeatable results. An example of repeatable processes is following the same route to work every day regardless of driving conditions. An example of repeatable results is getting to work on time every day, but being willing to change your route as required, bicycling into work instead of driving, taking public transit, and so on. Nobody really cares how you get to work each day (the process), what they really care about is that you got to work on time (the result). Sadly, we've been told for decades now that repeatable processes are critical to our success in IT, yet when you step back and think about that's really a reflection of a bureaucratic approach. On the other hand, a focus on repeatable results is a reflection of a more disciplined approach. Interestingly, the DDJ 2008 Process Framework survey
found that given the choice that people would much rather have repeatable results over repeatable processes
when it comes to IT.
Mistaking bureaucracy for discipline, or rigour if you prefer that term, is a reflection of the cultural damage that has occurred over the years in IT organizations as the result of traditional philosophies and techniques. Unfortunately, this mistaken belief is a significant inhibitor to software process improvement (SPI) efforts, in particular agile adoption efforts
, which must be addressed if you're to be successful. Overcoming this challenge will require a significant cultural shift in some organizations, and many people (particularly the bureaucrats) will find this uncomfortable.Further reading:
I'd like to leave you with this parting thought: Bureaucracy is bureaucracy and discipline is discipline, please don't confuse the two
Modified by ScottAmbler
This article has been replaced by an official "Disciplined Agile Manifesto".
The text of the original article remains below.
I've recently been working with Mark Lines of UPMentors and we've had some interesting discussions around evolving the Agile Manifesto which I thought I would share here to obtain feedback. Note that this is not any sort of official position of IBM, nothing in my blog is by the way (unless explicitly stated so), nor is it some sort of devious plot to take over the agile world (although if we did have some sort of devious plot, we'd make the exact same claim). What we hope to accomplish is to put some ideas out there in the hopes of getting an interesting conversation going.
Over the past decade we’ve applied the ideas captured in the Agile Manifesto and have learned from our experiences doing so. What we’ve learned has motivated us to suggest changes to the manifesto to reflect the enterprise situations which we have applied agile and lean strategies in. We believe that the changes we’re suggesting are straightforward:
Where the original manifesto focused on software development, a term which too many people have understood to mean only software development, we suggest that it should focus on solution delivery.
Where the original focused on customers, a word that for too many people appears to imply only the end users, we suggest that it focus on the full range of stakeholders instead.
Where the original manifesto focused on development teams, we suggest that the overall IT ecosystem and its improvement be taken into consideration.
Where the original manifesto focused on the understanding of, and observations about, software development at the time there has been some very interesting work done within the lean community since then (and to be fair there was very interesting work done within that community long before the Agile Manifesto was written). We believe that the Agile Manifesto can benefit from lean principles.
Our suggested rewording of the Agile Manifesto follows, with our suggested changes in italics.
Updating the Values of the Agile Manifesto
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working solutions over comprehensive documentation
Stakeholder collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
Updating the Principles behind the Agile Manifesto
Our highest priority is to satisfy the customer through early and continuous delivery of valuable solutions.
Welcome changing requirements, even late in the solution delivery lifecycle. Agile processes harness change for the stakeholder’s competitive advantage.
Deliver working solutions frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Stakeholders and developers must work together daily throughout the project.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The most efficient and effective method of conveying information to and within a delivery team is face-to-face conversation.
Quantified business value is the primary measure of progress.
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Continuous attention to technical excellence and good design enhances agility.
Simplicity--the art of maximizing the amount of work not done--is essential.
The best architectures, requirements, and designs emerge from self-organizing teams.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Leverage and evolve the assets within your organizational ecosystem, and collaborate with the people responsible for those assets to do so.
Visualize workflow to help achieve a smooth flow of delivery while keeping work in progress to a minimum.
The organizational ecosystem must evolve to reflect and enhance the efforts of agile teams, yet be sufficiently flexible to still support non-agile or hybrid teams.
We’re agile – things evolve, including manifestos. Looking forward to your feedback (add a comment).
Updates Since this Was First Published:
In previous blog postings I've defined Discipline Agile Delivery (DAD)
but I haven't shared the lifecycle (although I have done so in a couple of white papers, most recently Scaling Agile: An Executive Guide
) in this blog. Until now.
The DAD method combines strategies and practices from several software methods, including Scrum
, Extreme Programming (XP)
, Agile Modeling
, Agile Data
, and the Open Unified Process (OpenUP)
. One aspect of this "process idea reuse" is that the DAD extends the Scrum lifecycle to be a full-fledged delivery lifecycle. Figure 1 overviews the Scrum lifecycle, which addresses the construction aspects of agile delivery quite well. It depicts the product backlog which is a basic prioritized requirements stack
; the concept of delivering incrementally in consistent time boxes (what Scrum calls sprints and other methods such as DAD call iterations); holding a daily coordination meeting; and demoing your work at the end of the sprint/iteration/timebox to get feedback from key stakeholders. These are all great ideas, which is why the DAD method adopts and enhances them.Figure 1: The Scrum lifecycle. The focus is on the construction part of the lifecycle (click on the diagram for details).
Several years ago I wrote about the Agile lifecycle
and the need to extend it beyond construction, and this thinking is reflected in the DAD lifecycle of Figure 2. The DAD lifecycle extends the Scrum lifecycle in several important ways:
- It includes explicit phases. Sacrilege! Rhetoric aside, there are in fact serial aspects to agile software development. Project teams go through an initiation effort - called "warm up" in Eclipse Way, Inception in Unified Process, Sprint 0 in Scrum, and Iteration 0 in other methods - at the beginning of a project. Eventually the team will go through a release phase - called the "end game" in Eclipse Way
- It includes project initiation. During the Inception phase you perform basic project initiation activities such as requirements envisioning, architecture envisioning, initial release planning, getting funding for the project, and starting to build the team (among other tasks). In the summer of 2009 I ran the 2009 Agile Project Initiation survey which revealed that the average agile team spent 4 week on initiation activities such as this, that 89% did some sort of up-front requirements work, and that 85% did some sort of up-front architecture work.
- It includes release activities. The DAD lifecycle also includes a Transition phase where you release your solution into production or the marketplace. These activities typically include final testing and hardening of your solution, training end users, pilot/beta testing, finalizing documentation, running the solution in parallel with the system(s) being replaced, and so on.
- It explicitly indicates production activities. Many agile delivery teams are responsible for supporting the existing version(s) of their system which are currently running in production. Because there are existing versions, the team will be getting enhancement requests and defect reports from their operations and support people. Many of these requests can simply be treated as new requirements to be prioritized and put on the work item stack. Some need to be addressed right away, particularly "severity 1" defects, which requires the delivery team to "stop the line", address the problem, release a patch or hot fix, then reintegrate the changes into the version they're currently working on. My experience is that it's critical to include a production phase in your delivery lifecycle to make it explicit to the team that they need to take operations and support concerns into account.
- It explicitly enhances the product backlog. The product backlog has evolved into a work item list (or work item stack if you prefer). Disciplined agile teams put more than just requirements on the stack, as you read above it is common to treat defect reports as you do requirements.
- It explicitly includes critical milestone points. An important aspect of the DAD method is that not only does it support self organization such as Scrum but it does so within an appropriate governance framework. Disciplined agile teams recognize that they exist within a larger organization, that they should follow common development guidelines, that they should strive to leverage and build out the shared enterprise infrastructure, that they must report common metrics to senior management (hopefully automatically via the use of instrumented tooling such as we see on the Jazz platform).
One of the advantages of the DAD method over Scrum is that it doesn't require you to figure these common things out for yourself. Figure 2. The Disciplined Agile Delivery (DAD) lifecycle (basic). The focus is on the delivery portion of the system lifecycle, from starting a project to releasing the solution into production (click on the diagram for details).
Figure 3 depicts the advanced form of the DAD lifecycle. This form of the lifecycle occurs on experienced teams that have originally adopted the basic, Scrum-based lifecycle of Figure 2 and evolved it over time to be leaner. One of the implications of continuous improvement and continuous learning, key recommendations of the DAD process framework, is that your lifecycle will evolve. Primary changes that you see between Figure 2 and Figure 3 are the adoption of a work item pool
instead of a work item stack
and the abandonment of iterations (the critical observation is that practices such as detailed planning, demos, retrospectives and so on do not need to be on the same cadence, hence iterations disappear).
Figure 3. The Disciplined Agile Delivery (DAD) lifecycle (advanced). Over time you will improve your strategy towards a leaner approach (click on the diagram for details).
Figure 4. An older version of the basic DAD lifecycle using Scrum terminology. Use whatever terminology you're comfortable with, there is no "standard" (click on the diagram for details).
One of the differences that you may have noticed between the Scrum lifecycle of Figure 1 and the DAD lifecycle of Figure 2 is different terminology is used. For example, iteration is used instead of sprint. Work item list is used instead of product backlog. Although I believe that the terminology used in Figure 2 is more accurate, it really doesn't matter because it's easy to use Scrum terminology with DAD if you like. You can see this clearly in Figure 4. What is important is that you choose the terminology which you are most comfortable with, and be prepared to translate back and forth between terms as there are no standards and there never will be. As an aside, you might find Translating Scrum Terminology
to be of interest.
DAD is still a process framework that you'll need to tailor to meet your unique needs. More on this in future blog postings.