I have a young daughter and she's at the age where she wants to dress herself. The problem is that if we pick a single outfit and try to get her to wear it she refuses (I've lost count of the times I've heard "I don't want that"). At the other extreme if we let her pick her own outfit from the closet she'll be there for hours trying everything on. As experienced parents advise what we need to do is present her with two or three choices and ask her to pick what she wants.
So how does this relate to software development? Once again, let's look at extremes. First, consider Scrum's approach of prescribing a single way of doing things. For example, Scrum prescribes that you hold a daily meeting, called a Scrum, where everyone stands up and answers the same 3 questions. Scrum also prescribes a single change management strategy where you have a stack of requirements prioritized by business value. Scrum prescribes three roles - ScrumMaster, Product Owner, and Team Member - as well as other things. Don't get me wrong, these strategies are all great in certain circumstances but not for all. Prescribing one way of doing things is an extreme, so perhaps we shouldn't be surprised when people refuse to do it that way or struggle to make it work given the situation that they face.
At the other extreme consider RUP's approach where it presents repository of techniques from which to select the ones appropriate for you. The problem is that now we have an overwhelming way of doing things from which to choose, all of them good options in certain situations. So why are we surprised when teams struggle to identify a coherent tailoring of RUP?
Now let's consider the middle ground. The Disciplined Agile Delivery (DAD) process decision framework takes a goals-driven approach. So, instead of saying "hold a daily stand up meeting and answer these three questions" it says to regularly coordinate within the team and there are several ways of doing so (hold a Scrum meeting, hold a Kanban-style meeting, and so on). Yes, DAD does provide a large number of techniques to choose from (as does the agile community in general) but it also provides a straightforward way to choose between them. DAD does this by describing the advantages and disadvantages of each technique and suggests when, and when not, to use each approach. When people are presented with viable options, and the trade-offs associated with each, it's much more likely that they'll choose an approach that is better suited for their situation.
Scrum's single prescribed strategy works well only when that strategy is appropriate for the situation at hand. Similarly, telling my daughter exactly what to wear works well only when she's in the mood to wear that outfit. RUP's cafeteria approach to software process works well when you have the expertise, and time, to choose what's best for you. Similarly, asking my daughter to pick out her outfit from all the choices in her closet only works well when I've got a lot of time to wait for her. In both situations a better strategy is to present options, describe the trade offs, and then let people pick what's right for them given the context of the situation that they face. This is exactly what the DAD framework promotes.
I believe the goals-based approach of Disciplined Agile Delivery (DAD) represents an important step forward in the software process realm. It's time to recognize the extremes for what they are and move to a more viable middle ground.
A recurring discussion that I have with experienced agile developers is what it means to take a disciplined agile approach. The conversation usually starts off by some saying "but it already requires discipline to do agile", something that I fully agree with, followed by "therefore 'disciplined agile' is merely a marketing term", something which I don't agree with. The challenge with the "standard" agile discipline is that it is often focused on construction activities within a single project team, clearly important but also clearly not the full picture. There's more to an agile project than construction, and there's more to most IT departments than a single development project. In short, there are many opportunities for IT professionals to up their discipline, and thereby up their effectiveness, opportunities which we make explicit in the Disciplined Agile Delivery (DAD) framework.
Let's explore the many aspects to taking a disciplined agile approach:
You adopt "standard" agile discipline
. Aspects of agile which require discipline
include adopting practices such as test-driven development (TDD), active stakeholder participation, working collaboratively, shortening the feedback cycle
, and many more. These strategies are a great start to becoming disciplined IT professionals.
You take a goal-driven approach
. When we first started working on the DAD framework I didn't want to create yet another prescriptive framework, particularly given Rational's track record with the Rational Unified Process (RUP) framework. Rational has been pilloried for years for the prescriptive nature of RUP, which is unfortunate because there are a lot of great ideas in RUP that agile teams can benefit from, some of which we adopted in DAD and many of which are being actively reinvented with the agile community even as you read this. Furthermore, there are many prescriptive elements of the Scrum method that can get teams in trouble. For example, Scrum prescribes that you hold a daily stand up meeting, often called a Scrum meeting, where everyone should answer three questions. That's a great approach for teams new to agile, but it proves problematic in many situations due to it's prescriptive nature. Do you really need to do this once a day? I've been on teams where we held coordination meetings twice a day and others only once a week. Do you really need to stand up? I've been on geographically distrubited agile teams where many of us were sitting down during coordination calls. Do you really need to answer three questions, two of which are clearly focused on status regardless of claims otherwise? I've been on lean teams where we met around our Kanban board and focused on potential blockers. The answers to these questions depends on the context of the situation you find yourself in. The challenge, at least from the point of view of a process framework, is how do you avoid falling into the trap of being overly prescriptive. The strategy we adopted in DAD is to take a goal-driven approach. The observation is that regardless of the situation you find yourself in there are common goals your team will need to fulfill. For example, at the beginning of a project common goals include developing an initial plan, initially exploring the scope, initially identifying a technical strategy, and securing initial funding (amongst others). Throughout construction you should coordinate your activities, improve the quality of your ecosystem, and produce a potentially consumable solution on a regular basis (more on this below). So, instead of prescribing a daily stand up meeting the DAD framework instead indicates you should coordinate your activities, and gives several options for doing so (one of which is a Scrum meeting). More importantly DAD describes the advantages and disadvantages of your options so that you can make the choice that's best suited for the situation your team finds itself in (see this blog posting
for a detailed example of the types of tables included in the DAD book to help you through such process tailoring decisions). In short, our experience is that it requires discipline to take a goal driven approach
to agile delivery over the prescriptive strategies in other agile processes.
You take a context-driven approach
. There are many tailoring factors, which I describe in the Software Development Context Framework (SDCF)
, which you need to consider when making process, tooling, and team structure decisions. For example, a large team will adopt a different collection of practices and tools than a small team. A geographically distributed team will adopt a different strategy than a team that is co-located. You get the idea. Other tailoring factors include compliance, team culture, organization culture, technical complexity, domain complexity, and project type. It requires discipline to recognize the context of the situation you find yourself in and then act accordingly.
You deliver potentially consumable solutions
. One of the observations that we made early in the development of the DAD framework was that disciplined agile teams produce potentially consumable solutions, not just potentially shippable software. Although delivery of high-quality, working software is important it is even more important that we deliver high-quality working solutions to our stakeholders. For example, not only are we writing software but we may also be updating the hardware on which it runs, writing supporting documentation, evolving the business processes around the usage of the system, and even evolving the organizational structure of the people working with the system. In other words, disciplined agilists focus on solutions over software
. Furthermore, "potentially shippable" isn't sufficient: not only should it be shippable but it should also be usable and should be something people want to use. In other words it should be consumable (a concept DAD adopted from IBM's Outside In Development
). Minimally IT professionals should have the skills and desire to produce good software, but what they really need are the skills and desire to provide good solutions. We need strong technical skills, but we also need strong "
such as user interface design and process design to name just two.
The incremental delivery of potentially consumable solutions on an incremental basis requires discipline
to do successfully. DAD teams focus on repeatable results not repeatable processes
You are enterprise aware
. Whether you like it or not, as you adopt agile you will constrained by the organizational ecosystem, and you will need to act accordingly. It takes discipline to work with enterprise professionals such as enterprise architects, data admistrators, portfolio managers, or IT governance people who may not be completely agile yet, and have the patience to help them. It takes discipline to work with your operations and support staff in a DevOps
manner throughout the lifecycle, particularly when they may not be motivated to do so. It requires discipline to accept and potentially enhance existing corporate development conventions (programming guidelines, data guidelines, UI guidelines, ...). It requires discipline to accept that your organization has an existing technology roadmap that you should be leveraging, building out, and in some cases improving as you go. In short, enterprise awareness requires a level of discipline
not typically seen on many agile teams.
You adopt a full delivery lifecycle
. Empirically it is very easy to observe that at the beginning of an agile project there are some activities that you need to perform to initiate the project. Similarly at the end of the project there are activities that you need to perform to release the solution into production or the marketplace. The DAD process framework addresses the effort required for the full delivery effort, including project initiation, construction, and deployment. Our experience is that it requires discipline on the part of IT professionals to include explicit phases
for Inception/Initation, Construction, and Transition/Deployment and more importantly to focus the appropriate amount of effort on each. One danger of explicit phases is that you run the risk of taking what's known as a Water-Scrum-Fall
approach, a term coined by Dave West the person who wrote the forward for the DAD book, where you take an overly heavy/traditional approach to inception and transition in combination with a lighter agile approach to construction. Water-Scrum-Fall occurs because many organizations haven't made a full transition to agile, often because they think it's only applicable to construction. Our experience is that you can be very agile in your approach to inception and transition, experience we've built into the DAD framework. Having said that it clearly requires discipline to keep inception activities short
and similarly it requires discipline to reduce the "transition phase" to an activity
You adopt a wider range of roles
. An interesting side effect of adopting a full delivery lifecycle is that you also need to adopt a more robust set of roles. For example, the Scrum method suggests three roles - Scrum Master, Product Owner, and Team Member - a reflection of the Scrum lifecycle's construction focus. DAD suggests three primary roles - Team Lead, Product Owner, Team Member, Architecture Owner
, and Stakeholder - as well as five secondary roles which may appear at scale.
You embrace agile governance
. Governance establishes chains of responsibility, authority and communication in support of the overall enterprise’s goals and strategy. It also establishes measurements, policies, standards and control mechanisms to enable people to carry out their roles and responsibilities effectively. You do this by balancing risk versus return on investment (ROI), setting in place effective processes and practices, defining the direction and goals for the department, and defining the roles that people play with and within the department. It requires discipline to adopt an agile approach to governance
, and that's something built right into the DAD framework.
It's been awhile since I've posted anything here, but that will soon change. I'm excited about some ideas that I have for upcoming blog postings around disciplined agile architecture and what it means to take a disciplined approach to agile solution delivery. Organizations around the world are finding that Disciplined Agile Delivery
(DAD) provides a solid foundation from which to scale agile, and I'm going to continue sharing my observations and experiences about scaling DAD here in this blog.
As many of you may know I left IBM in June 2012, after a six-year stint as Chief Methodologist for IT within IBM Rational, to form Scott Ambler + Associates
with Mark Lines. Mark is my co-author on the IBM Press book Disciplined Agile Delivery
, a topic I've written about extensively in this blog over the years. Since I left IBM I've remained in contact with several key people within Rational and have started working with them on several fronts. For example Walker Royce, Alan Brown, and myself are co-authoring a paper about scaling agile which we hope will be accepted at a prestigious academic conference in 2013. I'm also working with IBM Rational at the upcoming Agile Development Conference East
in Orlando, November 4-9. Both Mark and I will be presenting on DAD and Rational will have a booth and be throwing a party on Wednesday night which I'm looking forward to. Rational has thrown some pretty good conference parties in the past and I'm guessing that they will do so this time too. Hope to see you there.
In short, stay tuned as there's some exciting stuff coming your way soon.
The basic idea behind DevOps
is that your development strategy and operations strategy should reflect one another, that you should strive to optimize the whole IT process. This implies that development teams should work closely with your operations staff to deliver new releases smoothly into production and that your operations staff should work closely with development teams to streamline critical production issues.
DevOps has its source in agile software development, and it is an explicit aspect of the Disciplined Agile Delivery (DAD)
process framework. As a result there is a collection of agile development strategies which enable effective DevOps throughout the agile delivery lifecycle. These strategies include:
- Initial requirements envisioning. Disciplined agile teams invest time at the beginning of the project to identify the high-level scope in a light-weight, collaborative manner. This includes common operations requirements such as the need to backup and restore data sources, to instrument the solution so that it can be monitored in real time by operations staff, or to architect the solution in a modular manner to enable easier deployment.
- Initial architecture envisioning. Disciplined agile teams will also identify a viable architectural strategy which reflects the requirements of their stakeholders and your organization’s overall architectural strategy (hence the need to work closely with your enterprise architects and operations staff). One goal is to ensure that the team is building (or buying) a solution which will work well with the existing operational infrastructure and to begin negotiating any infrastructural changes (such as deploying new technologies) early in the project. Another goal is to ensure that operations-oriented requirements are addressed by the architecture from the very start.
- Initial release planning. As part of release planning the disciplined agile team works closely with their operations group to identify potential release windows to aim for, any release blackout periods to avoid, and the need for operations-oriented milestone reviews later in the lifecycle (if appropriate).
- Active stakeholder participation. Disciplined agile teams work closely with their stakeholders, including both operations and support staff, all the way through the lifecycle to ensure that their evolving needs are understood.
- Continuous integration (CI). This is a common technical agile practice where the solution is built/compiled, regression tested, and maybe even run through code analysis tools. CI promotes greater quality which in turn enables easier releases into production.
- Parallel independent testing. For enterprise-class development or at scale, particularly when the domain or technology is very complex or in regulatory environments, disciplined agile team will find they need to support their whole team testing efforts with an independent test team running in parallel to the development team. These testing issues often include validation of non-functional requirements – such as security, performance, and availability concerns – and around production system integration. All of these issues are of clear importance for operations departments.
- Continuous deployment. With this practice you automate the promotion of your working solution between environments. By automating as much of the deployment effort as possible, and by running it often, the development team increases the chance of a successful deployment and thereby reduces the risk to the operations environment. Note that deployment into production is generally not automatic, as this is an important decision to be made by your operations/release manager(s).
- Continuous documentation. With this practice supporting documentation, including operations and support documentation, is evolved throughout the lifecycle in concert with the development of new functionality.
- Production release planning. This is the subset of your release planning efforts which focuses on the activities required to deploy into production.
- Production readiness reviews. There should be at least one review, performed by the person(s) responsible for your operations environment, before the solution is deployed into production. The more critical the system, the more product readiness reviews may be required.
- End-of-lifecycle testing. Minimally you will need to run your full automated regression test suite against your baselined code once construction ends. There may also be manual acceptance reviews or testing to be performed, and any appropriate fixing and retesting required to ensure that the solution is truly ready for production.
There’s more to it though than simply adopting some good practices. Your process must also embrace several supporting philosophies. The Disciplined Agile Delivery (DAD)
process framework not only adopts the practices listed above, and more, but it also promotes several philosophies which enable DevOps:
- Delivery teams should be enterprise aware, that they should work with people such as operations staff and enterprise architects to understand and work towards a common operational infrastructure for your organization.
- Operations and support people should be recognized as key stakeholders of the solution being worked on.
- The delivery team should focus on solutions over software. Software is clearly important, but we will often provide new or upgraded hardware, supporting documentation (including operations and support procedures), change the business/operational processes that stakeholders follow, and even help change the organizational structure in which our stakeholders work.
- Your process should include an explicit governance strategy. Effective governance strategies motivate and enable development teams to leverage and enhance the existing infrastructure, follow existing organizational conventions, and work closely with enterprise teams – all of which help to streamline operations and support of the delivered solutions.
For more detail about this topic, I think that you will find the article I wrote for the December 2011 issue of Cutter IT Journal entitled “Disciplined Agile Delivery and Collaborative DevOps
” to be of value.
In November 2011 Paul Gorans, the Accelerated Solution Delivery (ASD) practice lead in IBM GBS, and I ran an agile adoption survey
. The survey explored a range of issue, including the factors that appear to be associated with the success and failure of agile project teams. Paul wrote up his thoughts in his Agile State of the Art Survey
article on ibm.com and I did the same for Dr Dobb's Journal in Agile Success Factors
. This blog posting summarizes the results of the survey.
Factors which appear to accelerate agile adoption include:
- People are assigned to a single team
- Development teams have easy access to business expertise
- Development teams are organized for agile delivery (not traditional)
- Your organization has an agile support group/community of excellence
- Your organization is explicitly addressing barriers to agility
- There is executive sponsorship for agile
- Agile teams are measured on value creation
- Your organization's IT governance strategy includes an agile path
Factors which appear to decelerate agile adoption include:
- Agile teams are measured using traditional metrics
A few days later someone asked a series of questions that I thought would make an interesting blog posting, so here goes:How much of IBM's projects (in percentage) are agile at the moment?
I don’t have exact numbers, but I believe that 90%+ of our teams in SWG are applying agile techniques in practical ways that make sense for their projects. The primary goal is to be effective – in frequent releases, higher quality, and happy customers – not just agile. By the way, there is roughly 30,000 developers in SWG.Can all of IBM's projects work with an agile methodology?
It’s certainly possible, but it may not always make sense. Products that are in maintenance mode with few bugs or feature requirements may not benefit as much from agile practices -- those teams will likely continue to do whatever it is that they have been doing. Having said that, it's still highly desirable to apply agile techniques on maintenance projects.
Also, agile methods can be harder to use on some projects than others, for example, around hardware development. As a general rule, I believe that the majority of software projects can benefit from agile techniques. The primary determinant of whether a team can adopt agile techniques is culture and skill – not team size, the domain, or the degree of geographic distribution. That notion surprises many people who think that large agile teams
or geographically distributed agile teams
can’t succeed in adopting agile practices.Are agile projects sub-parts of large waterfall projects?
In some cases, that may happen. I’m sure it’s also true in reverse. We see many customers who are migrating from waterfall projects to a more agile way of doing things, and they often start this migration with smaller sub-projects. At IBM, we have tens of thousands of developers worldwide on hundreds of teams, so we have examples of pretty much any combination of agile, iterative, and traditional practices that you can imagine. There’s definitely not one size that fits all, which is a key aspect of the Disciplined Agile Delivery (DAD)
process framework.What do you think the impact of these numbers will be on the PM community?
The IBM PM community is embracing agile. And the reality is that a majority of development organizations around the world are moving to agile now as well (as much as 80% in some of the recent studies I’ve seen). I look forward to the increased adoption of agile methods by the PM community in general. The fact that PMI now offers an Agile Certified Practitioner training program certainly underscores the fact that agile practices are being adopted widely in the mainstream which is a great thing to see.
On Nov 16 2011, Kim Werner, Agile Coach from ATSC and Liz Parnell, Solution Design Manager from Blue Cross Blue Shield of North Carolina (BCBSNC), gave a webcast
sharing their experiences regarding how BCBSNC adopted a few Agile techniques, with the help of some good coaching, and adopted some IBM Rational Jazz tools
(Rational Team Concert and Rational Requirements Composer) to reduce time to market and lower development costs. BCBCNC works in the highly-regulated healthcare industry, so faced a few interesting constraints when adopting agile.
During the second week of August the Agile 2011 conference was held in Salt Lake City (SLC). As you likely know the Agile Manifesto was formulated 10 years ago in SLC so it was apropos to hold it there. There was some excitement around the 10 year anniversary of the manifesto, with a panel session with the 17 authors of it. Sadly there seemed to be little excitement around the efforts of the 10th anniversay agile workshop
in February which proposed a potential path forward for the agile community. I found the conference to be an evolutionary improvement over the conferences of the past few years, which is a very good thing because the focus since 2008 has moved beyond the "cool" new programming techniques to include the issues that enterprises face.
Starting at the Agile 2008 conference I've seen an uptick in interest in what I would consider some of the more mature topics in agile development, although I'm unfortunately still seeing significant confusion out there too, in part due to over-exuberence of people new to agile. For example, there's people still asking about basic issues about agile architecture
and agile database
techniques, although I was really happy to see more coherent discussions around scaling agile
. My own presentation about the Agile Scaling Model
was well attended and I suspect I opened a few people's eyes regarding the realities that we face (yes, there's a lot more to it than holding a "scrum of scrums", yeesh). We have a long way to go until people really start to understand scaling issues, but we're clearly on the path to getting there.
The conference show floor was interesting, with a wide range of vendors offering services and products focused on agile and lean. One thing that I noticed was many vendors had large monitors showing off their ability to support lean task boards, which for the most part they all looked the same. At the IBM booth we were showing off some of the Jazz tools
, in particular Rational Team Concert (RTC)
. For a long time now we've been giving away fully functional, with no time limit, licenses of RTC for teams of up to 10 people. Something worth checking out.
The Agile 201x conferences hosted by the Agile Alliance are always a good investment of your time and money, and Agile 2011 was no exception. See you at Agile 2012 in the great state of Texas!
I recently recorded an audio podcast
about Collaborative Development and Operations (DevOps) and how it relates to Disciplined Agile Delivery (DAD)
. The podcast is 17 minutes long and covers a range of topics including:
- What are the challenges typically faced by Disciplined Agile Development teams as they begin to transition their working solutions into production?
- How best could this gap between Development and Operations be closed so that they worked collaboratively rather than in silos?
- If this DevOps gap was decreased or even closed, what would the measurable value be to Agile projects and their business stakeholders?
On a related note, IBM's Collaborative Development and Operations
landing page has some great resources if you're interested in how to implement.
My new white paper, Disciplined Agile Delivery: An Introduction
, is now available free of charge from IBM.com. The paper overviews the Disciplined Agile Delivery (DAD) process framework, a hybrid comprised of strategies from Scrum, XP, Agile Modeling, and other agile methods which is people first, learning oriented, and enterprise aware. DAD is the basis from which you can scale agile.
- Context counts - The Agile Scaling Model
- People first - People, and the way they interact with each other, are the primary determinant of success for a solution delivery project.
- Learning-oriented - The DAD process framework promotes the ideas that team members should collaborate closely and learn from each other, that the team should invest effort to learn from their experiences and evolve their approach, and that individuals should do so as well.
- Hybrid - DAD adopts and tailors strategies from Scrum, XP, Agile Modeling, UP, Kanban, and many others. It addresses many of the issues Mark Kennaley discusses in SDLC 3.0.
- IT solution focused - DAD teams produce potentially consumable solutions every construction iteration. This extends Scrum's "potentially shippable" strategy to explicitly address usability/consumability plus the fact that we're really delivering full solutions not just software.
- Goal-driven delivery life cycle - The DAD lifecycle is focused on delivery, not just construction. Furthermore it is goals-driven, the DAD process framework suggests various strategies to fulfill those goals but does not prescribe specific practices.
- Risk and value driven - The DAD lifecycle is risk and value driven. It extends Scrum's value-driven lifecycle which produces potentially shippable software each sprint/iteration so that it explicitly includes light-weight milesstones such as ensuring stakeholder consensus as to the scope of the project early in the lifecycle, proving the architecture with working code early in the lifecycle, ensuring sufficient functionality exists before transition, and ensuring production readiness before actual release of the solution.
- Enterprise aware - The DAD process framework promotes the ideas that DAD teams should work closely with their enterprise architecture groups to ensure they leverage and evolve the existing infrastructure, adopt and follow corporate guidelines, and work to the overall organizational vision. DAD teams are self organizing with appropriate governance.