In November 2011 Paul Gorans, the Accelerated Solution Delivery (ASD) practice lead in IBM GBS, and I ran an agile adoption survey
. The survey explored a range of issue, including the factors that appear to be associated with the success and failure of agile project teams. Paul wrote up his thoughts in his Agile State of the Art Survey
article on ibm.com and I did the same for Dr Dobb's Journal in Agile Success Factors
. This blog posting summarizes the results of the survey.
Factors which appear to accelerate agile adoption include:
- People are assigned to a single team
- Development teams have easy access to business expertise
- Development teams are organized for agile delivery (not traditional)
- Your organization has an agile support group/community of excellence
- Your organization is explicitly addressing barriers to agility
- There is executive sponsorship for agile
- Agile teams are measured on value creation
- Your organization's IT governance strategy includes an agile path
Factors which appear to decelerate agile adoption include:
- Agile teams are measured using traditional metrics
A few days later someone asked a series of questions that I thought would make an interesting blog posting, so here goes:How much of IBM's projects (in percentage) are agile at the moment?
I don’t have exact numbers, but I believe that 90%+ of our teams in SWG are applying agile techniques in practical ways that make sense for their projects. The primary goal is to be effective – in frequent releases, higher quality, and happy customers – not just agile. By the way, there is roughly 30,000 developers in SWG.Can all of IBM's projects work with an agile methodology?
It’s certainly possible, but it may not always make sense. Products that are in maintenance mode with few bugs or feature requirements may not benefit as much from agile practices -- those teams will likely continue to do whatever it is that they have been doing. Having said that, it's still highly desirable to apply agile techniques on maintenance projects.
Also, agile methods can be harder to use on some projects than others, for example, around hardware development. As a general rule, I believe that the majority of software projects can benefit from agile techniques. The primary determinant of whether a team can adopt agile techniques is culture and skill – not team size, the domain, or the degree of geographic distribution. That notion surprises many people who think that large agile teams
or geographically distributed agile teams
can’t succeed in adopting agile practices.Are agile projects sub-parts of large waterfall projects?
In some cases, that may happen. I’m sure it’s also true in reverse. We see many customers who are migrating from waterfall projects to a more agile way of doing things, and they often start this migration with smaller sub-projects. At IBM, we have tens of thousands of developers worldwide on hundreds of teams, so we have examples of pretty much any combination of agile, iterative, and traditional practices that you can imagine. There’s definitely not one size that fits all, which is a key aspect of the Disciplined Agile Delivery (DAD)
process framework.What do you think the impact of these numbers will be on the PM community?
The IBM PM community is embracing agile. And the reality is that a majority of development organizations around the world are moving to agile now as well (as much as 80% in some of the recent studies I’ve seen). I look forward to the increased adoption of agile methods by the PM community in general. The fact that PMI now offers an Agile Certified Practitioner training program certainly underscores the fact that agile practices are being adopted widely in the mainstream which is a great thing to see.
I'm happy to announce that IBM Rational's RP252 Advanced Disciplined Agile Delivery (DAD) workshop
is now available. This is a 3-day, hands-on workshop which teaches students the fundamentals of Disciplined Agile Delivery (DAD)
. This workshop is offered both publicly and privately.
There is a distinct rhythm, or cadence, at different levels of the agile process. We call this the agile 3C rhythm, for coordinate, collaborate, and conclude (which is sometimes called stabilize). The agile 3C rhythm occurs at three levels in Disciplined Agile Delivery (DAD):
- Day. A typical day begins with a short coordination meeting, called a Scrum meeting in the Scrum method. After the daily coordination meeting the team collaborates throughout most of the day to perform their work. The day concludes with a working build, hopefully you had several working builds throughout the day, which depending on your situation may require a bit of stabilization work to achieve.
- Iteration. DAD construction iterations begin with an iteration planning session (coordinate) where the team identifies a detailed task list of what needs to be done that iteration. Note that iteration modeling is often part of this effort. Throughout the iteration they collaborate to perform the implementation work. They conclude the iteration by producing a potentially consumable solution, a demo of that solution to key stakeholders, and a retrospective to identify potential improvements in the way that they work.
- Release. The DAD lifecycle calls out three explicit phases - Inception, Construction, and Transition – which map directly to coordinate, collaborate, and conclude respectfully.
The agile 3C rhythm is similar conceptually to Deming’s Plan, Do, Check, Act (PDCA) cycle:
- Coordinate maps to plan
- Collaborate maps to do
- Conclude maps to check and act
There is a fair bit of rhetoric surrounding agile methods, some of which we subscribe to and some of which we don’t. We’d like to briefly examine the rhetoric which we’ve found to be the most misleading for people trying to be effective at adopting agile techniques. The following list is in the format X but Y, where X is the rhetoric and Y is the strategy promoted by the Disciplined Agile Delivery (DAD) process framework. This includes:
- Requirements evolve throughout the lifecycle BUT the scope should still be agreed to at the beginning of the project. There has to be an initial vision for a project, a vision which your stakeholders should help define and then agree to, and to come to that vision you will need to perform some initial requirements envisioning. A list of high level features is part of this initial vision. Yes, the details are very likely to evolve over time but the fundamental goals of your project and scope of your effort needs to be defined early in your project. In a very small minority of situations you may not be able to get the right people together, either physically or virtually, to define the initial vision – this should be seen as a significant project risk.
- Simple designs are best BUT the architecture should be thought out early in the lifecycle. Too many developers interpret the advice to focus on simple designs to mean that they should build everything from scratch. Yet more often than not the simplest design is to take advantage of what is already there, and the best way to do that is to work closely with people who understand your existing technical infrastructure. Investing in a little bit of architectural envisioning early in the lifecycle enables your team to identify existing enterprise assets that you can leverage, to identify your architectural options, and to select what appears to be the best option available to you. The details will still emerge over time, and some decisions will be deferred until a later date when it’s more appropriate to make them, but the bottom line is that disciplined agilists think before they act.
- Teams should be self organizing BUT they are still constrained (and enhanced) by your organizational ecosystem. Intellectual workers, including IT professionals, are most effective when they have a say in what work they do and how they do it. IT professionals can improve their productivity by following common conventions, leveraging and building out a common “dev-ops” infrastructure, building towards a common vision, and by working to common business and technical visions. In short, disciplined agile professionals are "enterprise aware".
- Delivery teams don’t need prescriptive process definitions BUT they do need some high-level guidance to help organize their work. Individual IT professionals are typically highly-skilled and highly-educated people often with years of experience, and teams of such people clearly have a wide range of knowledge. As a result of this knowledge it is incredibly rare for such people to read detailed procedures for how to do their work. However, they often still require some high-level advice to help them to organize their work effectively. Teams can often benefit from techniques and patterns used by other teams and this knowledge sharing should be encouraged.
- IT professionals know what to do BUT they’re still not process experts. A decade ago the strategy was to provide detailed process advice to teams, but recently the pendulum has swung the other way to provide little or no defined process at all. Over the last few years there’s been a trend within the agile community to advise teams to define their own process so that it’s tailored to their own unique situation. While this clearly strokes people’s egos, it’s relatively poor advice for several reasons. First, although every team is in a unique situation there is significant commonality so having at least a high-level process framework from which to start makes sense. Second, although these teams have a wide range of knowledge it might not be complete, nor consistent, nor is it clear what the trade-offs are of combining all the really good techniques that people know about. There is significant benefit in having a flexible process framework such as DAD which shows how everything fits together.
- IT professionals should validate their own work to the best of their ability BUT they likely aren’t testing experts so therefore need help picking up the appropriate skills. The mantra in the agile community is to test often and test early, and better yet to test first. As a result agile teams have adopted a “whole team” approach where the development team does its own testing. This works when there are people on the team with sufficient testing skills and more importantly can transfer those skills to others. Minimally you will need to embed testers into your delivery teams, but you should also consider explicit training and mentoring of everyone on the team in testing and quality skills. You may find my agile testing and quality strategies article to be an interesting read.
- Disciplined agile teams work in an iterative manner BUT still follow a lifecycle which is serial over time. On any given day people on a DAD project team may be performing analysis, testing, design, programming, deployment, or a myriad of other activities and iterating back and forth between them. But, the DAD lifecycle includes three distinct phases which are performed in order. So, DAD is both iterative in the small but serial in the large.
Modified by ScottAmbler
This article has been replaced by an official "Disciplined Agile Manifesto".
The text of the original article remains below.
I've recently been working with Mark Lines of UPMentors and we've had some interesting discussions around evolving the Agile Manifesto which I thought I would share here to obtain feedback. Note that this is not any sort of official position of IBM, nothing in my blog is by the way (unless explicitly stated so), nor is it some sort of devious plot to take over the agile world (although if we did have some sort of devious plot, we'd make the exact same claim). What we hope to accomplish is to put some ideas out there in the hopes of getting an interesting conversation going.
Over the past decade we’ve applied the ideas captured in the Agile Manifesto and have learned from our experiences doing so. What we’ve learned has motivated us to suggest changes to the manifesto to reflect the enterprise situations which we have applied agile and lean strategies in. We believe that the changes we’re suggesting are straightforward:
Where the original manifesto focused on software development, a term which too many people have understood to mean only software development, we suggest that it should focus on solution delivery.
Where the original focused on customers, a word that for too many people appears to imply only the end users, we suggest that it focus on the full range of stakeholders instead.
Where the original manifesto focused on development teams, we suggest that the overall IT ecosystem and its improvement be taken into consideration.
Where the original manifesto focused on the understanding of, and observations about, software development at the time there has been some very interesting work done within the lean community since then (and to be fair there was very interesting work done within that community long before the Agile Manifesto was written). We believe that the Agile Manifesto can benefit from lean principles.
Our suggested rewording of the Agile Manifesto follows, with our suggested changes in italics.
Updating the Values of the Agile Manifesto
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working solutions over comprehensive documentation
Stakeholder collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
Updating the Principles behind the Agile Manifesto
Our highest priority is to satisfy the customer through early and continuous delivery of valuable solutions.
Welcome changing requirements, even late in the solution delivery lifecycle. Agile processes harness change for the stakeholder’s competitive advantage.
Deliver working solutions frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Stakeholders and developers must work together daily throughout the project.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The most efficient and effective method of conveying information to and within a delivery team is face-to-face conversation.
Quantified business value is the primary measure of progress.
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Continuous attention to technical excellence and good design enhances agility.
Simplicity--the art of maximizing the amount of work not done--is essential.
The best architectures, requirements, and designs emerge from self-organizing teams.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Leverage and evolve the assets within your organizational ecosystem, and collaborate with the people responsible for those assets to do so.
Visualize workflow to help achieve a smooth flow of delivery while keeping work in progress to a minimum.
The organizational ecosystem must evolve to reflect and enhance the efforts of agile teams, yet be sufficiently flexible to still support non-agile or hybrid teams.
We’re agile – things evolve, including manifestos. Looking forward to your feedback (add a comment).
Updates Since this Was First Published:
- Does the team regularly produce value for their stakeholders?
- Does the team validate its own work to the best of its ability?
- Are stakeholders actively involved?
- Is the team self organizing?
- Does the team strive to improve their process?
Some interesting results include:
94% of teams which are claiming to be agile are providing value to stakeholders on a regular basis.
87% of teams which are claiming to be agile are validating their own work.
95% of teams which are claiming to be agile are working closely with stakeholders.
56% of teams which are claiming to be agile are self organizing.
88% of teams which are claiming to be agile are improving the process that they follow throughout the lifecycle.
Teams which are claiming to be agile often aren't. 53% of "agile teams" meet the five criteria, although 72% meet all but the self-organization criteria.
Teams which are moving towards agile but aren't there yet are reasonably close. 39% of those teams meet all five criteria and 63% meet all but self-organization.
I believe that there are several important implications:
- Whenever someone claims to be on an agile team you may want to explore that claim a bit deeper.
The low level of self organization may be an indicator of cultural challenges with organizations in that their project managers aren't giving up sufficient control. The Agility at Scale survey
in November 2009 found that 59% of respondents who indicated that their organization hadn't adopted agile techniques yet that a rigid culture was hampering their efforts. The IT Governance and Project Management
survey in July 2009 discovered that "questionable behaviors", many of which were ethically questionable (I'm being polite), were far too common within IT project management.
Although "agile teams" may not be as agile as they claim, they're still doing better than traditional V-model teams, as revealed (again) by the 2010 IT Project Success
If there was some sort of consensus within the agile community as to the criteria for determining whether a team is agile, I highly suspect that the agileness ratings would increase over time. What gets measured often improves.
However, how agile you are isn't anywhere near as important as getting better at what you're doing. So perhaps I'm barking up the wrong tree on this issue. ;-)
Modified by ScottAmbler
This blog posting has been replaced by the more detailed article: Full Agile Delivery Lifecycles.
Thank you for your patience.
When you are first adopting agile techniques in your organization a common strategy is to run one or more pilot projects. When organizing these projects you typically do as much as you can to make them successful, such as finding:
- Projects where the stakeholders are willing to actively work with you.
- IT people who are flexible, willing to try new things, and willing to collaborate with one another.
- IT people who are generalizing specialists, or at least willing to become so.
- Finding a project which is of medium complexity (therefore it's "real" in the sense that it's significant to your organization) but not one where it can make or break your organization (therefore it's safe to experiment with).
In North America we refer to this as "cherry picking" because you're picking the cherry/best situation that you can find.
- Being agile may not have been the primary determinant of success. You set up an environment where you have a good relationship with your stakeholders, where you have good people who want to work together, and the project is challenging but not impossible. Oh, and by the way you adopted a few agile techniques as well. Sounds to me that situation you could have adopted a few not-so-agile techniques instead and still succeed. Although my various project success surveys, see my IT surveys page for details, have shown time and again that agile project teams are more successful than traditional project teams I haven't been able to tease out (yet) whether this success is attributable to agile or just attributable to improved project initiation efforts.
- When adopting agile/lean widely across your organization, you can't cherry pick any more. For the past few years I've been working with IT organizations that are in the process of adopting agile/lean strategies across their entire organization, not just across a few pilot projects. What these organizations are finding is that they need to find ways to adopt agile where the business isn't as willing to work with IT, where some of the people aren't so flexible or collaborative, where some of the people are narrowly specialized and not as willing to expand their skills, or where the project exhibits scaling factors which motivates you to tailor your agile approach. It's harder to succeed with agile in these situations because they're not as "cherry" as what you've experienced previously. Luckily, if you've been successful previously then you now have some agile experienced people, you have successes to reference, and you've likely overcome some problems even in the cherry situations which you have learned from. So, your cherry successes will hopefully improve your ability to succeed even in "non cherry" situations.
- You need to work smarter, not harder. If the source of your success was actually from improved project initiation practices and not from agile, then recognize that and act accordingly. Realistically part of your success was from that and part was from agile, and the organizations that adopt a measured improvement approach potentially have the data to determine which practices lead to success and which didn't. Without the metrics you're effectively flying blind when it comes to deciding how to improve. There is clearly a mandate for smarter work practices within IT, within your organization as a whole for that matter.
If you want to gain more insight into some of the issues that you'll face when adopting agile across your organization, I suspect that you'll find my recent paper Scaling Agile: An Executive Guide
to be interesting. I've got a more detailed paper in the works, so stay tuned to this blog.
Modified by ScottAmbler
An imporant step in scaling your agile strategy is to adopt a Disciplined Agile Delivery (DAD)
approach instead of one which is just focused on agile construction. One aspect of adopting a DAD approach it to mature your focus from just producing software to instead providing a solution which meets the needs of its stakeholders within the appropriate economic, cultural, and technical constraints. The fundamental observation is that as IT professionals we do far more than just develop software. Yes, this is clearly important, but in addressing the needs of our stakeholders we will often:
Provide new or upgraded hardware
Change the business/operational processes which stakeholders follow
Change the organizational structure in which our stakeholders work
Update supporting documentation
And yes, develop high-quality software
Although delivery of high-quality, working software is important it is even more important that we deliver high-quality working solutions to our stakeholders. Minimally IT professionals should have the skills and desire to produce good software, but what they really need are the skills and desire to provide good solutions. We need strong technical skills, but we also need strong "soft skills" such as user interface design and process design to name just two.
The shift to a solution-oriented focus from a software-oriented focus requires your agile teams to address some of the software-oriented prejudices which crept into the Agile Manifesto
. The people who wrote the manifesto (which I fully endorse) were for the most part software developers, consultants, and in many cases both. It is little wonder that this group would allow a bias towards software development creep into the language of their manifesto.
Just like there are 5Ps of marketing, there are also “5 Ps” of IT:
- People. People and the way they work together have a greater effect on the outcomes of a project than the processes they’re following or the products (tools and technologies) that they’re using. People issues include having visible executive sponsorship, building an environment of trust, empowering staff, focusing on leadership as well as management, recognizing that the primary gating factor when improving processes is people’s ability to absorb change, and promoting a cross-discipline strategy at both the team and individual levels.
- Principles/philosophies. We’ve found both internally within IBM as well as with many of our customers that there is a need to define a common set of principles to provide a consistent foundation to enable effective teamwork and continuous process improvement. These principles help to guide people’s decisions when their processes and practices don’t directly address the situation which they find themselves in.
- Practices/patterns. A practice is a self-contained, deployable component of a process. You might find the IBM Practices interesting.
- Products. This includes the technologies – such as databases, application servers, networks, and client platforms – and tools such as integrated development environments, testing tools, and project planning tools used to create solutions for stakeholders.
- Processes. The previous 4Ps do not exist in a vacuum, we need some sort of glue to help piece all of this together. Minimally this glue is a lifecycle although more often than not it is a full process or method.
My experience is that to be successful at software process improvement (SPI)
across your entire IT department that you must address these 5Ps. How you address each issue, and to what extent, will vary based on your situation.
My January 2010 DDJ Agile Update, Tragic Mistakes When Adopting Test Driven Development (TDD)
, is now online. In the article I summarize what I consider to be common, and tragic, mistakes that I'm seeing organizations make when they attempt to adopt TDD.
These mistakes include:
The article also goes into potential benefits of TDD as well as potential challenges that you're face when adopting it.
Modified by ScottAmbler
For several years now I've written various articles and newsletters on the topics of estimating and funding strategies for software development projects, and in particular for agile software development projects. Whenever I talk to people about agile software development, or coach them in how to succeed at it, some of the very first questions that I'll be asked, particularly by anyone in a management role, is how to fund agile software development projects. Apparently a lot of people think that you can only apply agile strategies on small, straightforward projects where it makes sense to do a time and materials (T&M) approach. In fact you can apply agile strategies in a much greater range of situations, as the various surveys
that myself and others are showing time and again. My goal with this blog posting is to summarize the various strategies for, and issues surrounding, the funding of agile software development projects.
There are three basic strategies for funding projects, although several variations
clearly exist. These strategies are:
- "Fixed price". At the beginning of the project develop, and then commit to, an initial estimate based on your up-front requirements and architecture modeling efforts. Hopefully this estimate is given as a range, studies have shown that up-front estimating techniques such as COCOMO II or function points are accurate within +/- 30% most of the time although my July 2009 State of the IT Union survey found that on average organizations are shooting for +/- 11% (their actuals come in at +/- 19% on average, but only after doing things such as dropping scope, changing the estimate, or changing the schedule). Fixed-price funding strategies are very risky in practice because they promote poor behavior on the part of development teams to overcome the risks foisted upon them as the result of this poor business decision. It is possible to do agile on a fixed budget but I really wouldn't recommend it (nor would I recommend it for traditional projects). If you're forced to take a fixed-price approach, and many teams are because the business hopes to reduce their financial risk via this approach not realizing that it actually increases their risk, then adopt strategies that reduce the risk.
- Stage gate. Estimate and then fund the project for given periods of time. For example, fund the project for a 3-month period then evaluate it's viability, providing funding for another period of time only to the extent that it makes sense. Note that stages don't have to be based on specific time periods, they could instead be based on goals such as to intiate the project, prove the architecture with working code, or to build a portion of the system. Disciplined agile methods such as Open Unified Process have built in stage-gate decision points which enable this sort of strategy. When the estimation technique is pragmatic, the best approaches are to have either the team itself provide an estimate for the next stage or to have an expert provide a good gut feel estimate (or better yet have the expert work with the team to develop the estimate). Complex approaches such as COCOMO II or SLIM are often little more than a process facade covering up the fact that software estimating is more of an art or a science, and prove to be costly and time consuming in practice.
- Time and materials (T&M). With this approach you pay as you go, requiring your management team to actually govern the project effectively. Many organizations believe a T&M strategy to be very risky, which it is when your IT governance strategy isn't very effective. An interesting variation, particularly in a situation where a service provider is doing the development, is an approach where a low rate is paid for their time which covers their basic costs, the cost of materials is paid out directly, and delivery bonuses are paid for working software. This spreads the risk between the customer/stakeholder and the service provider. The service provider has their costs covered but won't make a profit unless they consistently deliver quality software.
The point is that there are several strategies for funding agile software development projects, just like there are several strategies for funding traditional software development projects. My experience is that fixed-price funding strategies are incredibly poor practice which increases the risk of your software development projects dramatically. I recognize how hard it can be to change this desire on the part of our business stakeholders, but have also had success changing their minds. If you choose to perservere, which is a difficult decision to make, you can help your organization's decision makers to adopt more effective strategies. Like you they want to improve the effectiveness of your IT efforts.Further reading: (In recommended order)
- Something's Gotta Give: Argues for a flexibly approach to funding, schedule, and/or scope.
- Agile on a Fixed Budget: Describes in detail how to take a fixed-price approach on agile projects.
- The Dire Consequences of Fixed-Price IT Projects: Describes in detail the questionable behavior exhibited by IT teams when forced to take a fixed-price approach.
- Is Fixed-Price Software Development Unethical?: Questions the entire concept of fixed-price IT projects, overviewing some of the overwhelming evidence against this really poor practice.
- Reducing the Risk of Fixed-Price Projects: Describes viable strategies for addressing some of the problems resulting from the decision of fixed-price projects.
- Strategies for Funding Software Development Projects: Describes several variations on the strategies described above.
- Lies, Great Lies, and Software Development Project Plans: Summarizes some results from the July 2009 State of the IT Union survey which explored issues related to project funding (among many).
The Scrum community has adopted a different set of terms than the other agile methodologies. This is done on purpose to help people realize that Agile approaches are different than traditional approaches, which can help in their adoption, but it can also hinder people's understanding because some of the terminology is not only non-standard it really doesn't make much sense. Because of this I'm often asked by people that I'm coaching to convert back and forth between terms, and recently wrote a detailed article on the subject. The following summarizes the mapping:
- Daily Scrum Meeting ==> Daily Stand-up Meeting
- Product Backlog ==> Work Item List
- Scrum Master ==> Team Lead or Team Coach
- Sprint ==> Iteration or Time Box
For more details read my article Translating Scrum Terminology
which includes explanations of a wider range of Scrum terms and discussions of why some of them really are questionable. Further reading:
I just wanted to share with you the Manifesto for Software Craftsmanship
which extends the Agile Manifesto
. The Manifesto for Software Craftsmanship states:As aspiring Software Craftsmen we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:
- Not only working software, but also well-crafted software
- Not only responding to change, but also steadily adding value
- Not only individuals and interactions, but also a community of professionals
- Not only customer collaboration, but also productive partnerships
That is, in pursuit of the items on the left we have found the items on the right to be indispensable.
I view this manifesto as an important step in the maturation of software development. More on this in a future blog posting.[Read More