Agility@Scale: Strategies for Scaling Agile Software Development

I'm happy to announce that IBM Rational's RP252 Advanced Disciplined Agile Delivery (DAD) workshop is now available.  This is a 3-day, hands-on workshop which teaches students the fundamentals of Disciplined Agile Delivery (DAD).  This workshop is offered both publicly and privately.

There is a fair bit of rhetoric surrounding agile methods, some of which we subscribe to and some of which we don’t. We’d like to briefly examine the rhetoric which we’ve found to be the most misleading for people trying to be effective at adopting agile techniques. The following list is in the format X but Y, where X is the rhetoric and Y is the strategy promoted by the Disciplined Agile Delivery (DAD) process framework. This includes:

1. Requirements evolve throughout the lifecycle BUT the scope should still be agreed to at the beginning of the project. There has to be an initial vision for a project, a vision which your stakeholders should help define and then agree to, and to come to that vision you will need to perform some initial requirements envisioning. A list of high level features is part of this initial vision. Yes, the details are very likely to evolve over time but the fundamental goals of your project and scope of your effort needs to be defined early in your project. In a very small minority of situations you may not be able to get the right people together, either physically or virtually, to define the initial vision – this should be seen as a significant project risk.
2. Simple designs are best BUT the architecture should be thought out early in the lifecycle. Too many developers interpret the advice to focus on simple designs to mean that they should build everything from scratch. Yet more often than not the simplest design is to take advantage of what is already there, and the best way to do that is to work closely with people who understand your existing technical infrastructure. Investing in a little bit of architectural envisioning early in the lifecycle enables your team to identify existing enterprise assets that you can leverage, to identify your architectural options, and to select what appears to be the best option available to you. The details will still emerge over time, and some decisions will be deferred until a later date when it’s more appropriate to make them, but the bottom line is that disciplined agilists think before they act.
3. Teams should be self organizing BUT they are still constrained (and enhanced) by your organizational ecosystem. Intellectual workers, including IT professionals, are most effective when they have a say in what work they do and how they do it. IT professionals can improve their productivity by following common conventions, leveraging and building out a common “dev-ops” infrastructure, building towards a common vision, and by working to common business and technical visions.  In short, disciplined agile professionals are "enterprise aware".
4. Delivery teams don’t need prescriptive process definitions BUT they do need some high-level guidance to help organize their work. Individual IT professionals are typically highly-skilled and highly-educated people often with years of experience, and teams of such people clearly have a wide range of knowledge. As a result of this knowledge it is incredibly rare for such people to read detailed procedures for how to do their work. However, they often still require some high-level advice to help them to organize their work effectively. Teams can often benefit from techniques and patterns used by other teams and this knowledge sharing should be encouraged.
5. IT professionals know what to do BUT they’re still not process experts. A decade ago the strategy was to provide detailed process advice to teams, but recently the pendulum has swung the other way to provide little or no defined process at all. Over the last few years there’s been a trend within the agile community to advise teams to define their own process so that it’s tailored to their own unique situation. While this clearly strokes people’s egos, it’s relatively poor advice for several reasons. First, although every team is in a unique situation there is significant commonality so having at least a high-level process framework from which to start makes sense. Second, although these teams have a wide range of knowledge it might not be complete, nor consistent, nor is it clear what the trade-offs are of combining all the really good techniques that people know about. There is significant benefit in having a flexible process framework such as DAD which shows how everything fits together.
6. IT professionals should validate their own work to the best of their ability BUT they likely aren’t testing experts so therefore need help picking up the appropriate skills. The mantra in the agile community is to test often and test early, and better yet to test first. As a result agile teams have adopted a “whole team” approach where the development team does its own testing. This works when there are people on the team with sufficient testing skills and more importantly can transfer those skills to others. Minimally you will need to embed testers into your delivery teams, but you should also consider explicit training and mentoring of everyone on the team in testing and quality skills.  You may find my agile testing and quality strategies article to be an interesting read.
7. Disciplined agile teams work in an iterative manner BUT still follow a lifecycle which is serial over time. On any given day people on a DAD project team may be performing analysis, testing, design, programming, deployment, or a myriad of other activities and iterating back and forth between them. But, the DAD lifecycle includes three distinct phases which are performed in order. So, DAD is both iterative in the small but serial in the large.

The popular Agile literature can often seam naive when it comes to how Agilists work with project stakeholders:- Extreme Programming (XP) has a practice called On-Site Customer where one or more people work closely with your team to provide information and to make decisions in a timely manner.- Scrum has the role of Product Owner who is the one single person that the development team goes to for decisions about requirements. - Agile Modeling (AM) has the practice of Active Stakeholder Participation which extends On-Site Customer to get the stakeholder(s) actively involved with the modeling effort through the use of inclusive tools and techniques.

These are great strategies for small, co-located teams doing straightforward development, but they quickly fall apart at scale. This occurs for several reasons:1. Stakeholders are a diverse group. Your stakeholders include end users, business management, project funders, enterprise architects, operations staff, support staff, other system development teams, and many others. Different people have different, and often contradictory, requirements and they certainly have different priorities. It's questionable whether a single person, or a handful of persons, can adequately represent this diverse group.2. One person becomes a bottleneck. Even with a small co-located team this is a problem, let alone one that is geographically distributed or one that is very large. There's no way that a single person can be available 24/7 in a responsive manner to support distributed teams.3. It's a difficult role. The Product Owner/Customer (POC) is responsible for representing the business to the development team. They're making important decisions on a regular basis, decisions which they'll be held accountable for.4. One person becomes a serious project risk. Not only is it questionable whether a single person can fairly represent all stakeholders, even if they could what happens if you lose that person? They effectively become a single point of failure for your team.

To scale this role, consider the following strategies:1. Recognize the true scope of the POC role. Not only are they stakeholder proxies they also are a development team representative to the stakeholder community as a whole. As stakeholder proxies they'll make decisions and prioritize the work, they'll run requirements elicitation sessions, they'll negotiate priorities, and they'll put the development team in contact with stakeholders who have expertise in specific aspects of the domain. As team representatives they'll often demo the current version of the system to other stakeholders, communicate the status of the project to people, and respond to various requests for information from the stakeholders.2. Have multiple people in it. A single POC works well for small, co-located teams developing simple software. At scale you'll soon discover that you need multiple people in this role so that they don't become a bottleneck. For distributed teams it's common to see each subteam have one or more POCs who are managed by a primary/chief POC. The primary POC typically works on the coordinating team with the chief architect (I'll talk about this role in a future blog posting) and the program manager (also a topic for a future blog posting).3. Train them in business analysis skills. The person(s) in the POC role need good business analysis skills. If fact, it's common for people who were formerly BAs for traditional teams to step into the POC role, particularly with BAs who originally come from the business side of your organization. This strategy has its advantages and disadvantages. As a BA they've likely got solid business knowledge but their instincts may motivate them to take a documentation-driven approach to providing information to the development team instead of a collaboration-based approach. Be careful.4. Consider the full system development lifecycle. There's far more to the POC role than supporting the development team during Construction iterations. During "Iteration 0", the Inception phase for an Agile RUP project or the warm-up phase for an Eclipse Way project, the POC(s) will often lead the initial requirements envisioning efforts. The product backlog, or better yet your work item list, needs to come from somewhere after all. During the release iteration(s), the Transition phase for RUP or the End-Game phase for Eclipse Way, the POC(s) will focus on communicating the upcoming release to the stakeholder community, will be actively involved with any final user acceptance testing (UAT), and may even be involved with training end users.

In my January 2008 column in Dr Dobb's Journal, posted at http://www.ddj.com/architect/204801134 , I provide detailed advice about how to scale the way that you work with stakeholders on Agile projects by applying the practices of Agile Model Driven Development (AMDD). There's no magic solution, you just need to choose to organize yourself effectively. The good news is that you can easily work with stakeholders at scale.[Read More]

On April 25, 2013 I gave a webcast for the Global Rational User Community entitled Disciplined Agile Delivery: Going beyond Scrum .  During the webcast a large number of questions were asked but unfortunately I couldn’t get to all of them.  So I’ve taken the opportunity to write up the answers in this blog posting.

1. Fequently asked questions. Many of the questions are addressed in the DAD FAQ.
2. DAD elevator pitch - I will be starting work in a couple of weeks for a company that has just started its Agile journey this year by implementing Scrum. What would Scott put in an elevator chat as to why they should be moving towards DAD.  The Disciplined Agile Delivery (DAD) decision process framework is a people-first, learning-oriented hybrid agile approach to IT solution delivery. It has a risk-value delivery lifecycle, is goal-driven, is enterprise aware, and is scalable.  There are clearly some interesting aspects to the DAD framework. DAD is a hybrid approach which extends Scrum with proven strategies from Agile Modeling (AM), Extreme Programming (XP), Unified Process (UP), Kanban, Lean Software Development, Outside In Development (OID) and several other methods. DAD is a non-proprietary, freely available framework. DAD extends the construction-focused lifecycle of Scrum to address the full, end-to-end delivery lifecycle from project initiation all the way to delivering the solution to its end users. It also supports lean and continuous delivery versions of the lifecycle: unlike other agile methods, DAD doesn’t prescribe a single lifecycle because it recognizes that one process size does not fit all. DAD includes advice about the technical practices such as those from Extreme Programming (XP) as well as the modeling, documentation, and governance strategies missing from both Scrum and XP. But, instead of the prescriptive approach seen in other agile methods, including Scrum, the DAD framework takes a goals-driven approach. In doing so DAD provides contextual advice regarding viable alternatives and their trade-offs, enabling you to tailor DAD to effectively address the situation in which you find yourself. By describing what works, what doesn’t work, and more importantly why, DAD helps you to increase your chance of adopting strategies that will work for you.  The article Introduction to Disciplined Agile Delivery provides a more detailed description.
3. The book: Could you please repeat the name of the book that Scott is talking about?  The book is Disciplined Agile Delivery: A Practitioner’s Guide to Agile Software Delivery in the Enterprise published by IBM Press, June 2012.  The Disciplined Agile Delivery (DAD) website  and the Disciplined Agile Consortium website are also good DAD resources.
4. The Surveys: Where is the data published about geographic distribution and effectiveness?  All of my survey data, the original questions as asked, and my analysis can be downloaded free of charge from my IT Surveys page.  If you ever see a request from me to fill out a survey, please take a few minutes and do so.  I think you'll agree that my surveys page is in fact a very useful resource, so please contribute when you can.
5. Project success criteria: Where would a goal of defining success criteria fall?  In the DAD book we describe how the success criteria for the project should be initially identified during Inception. Success criteria, like other things, could evolve throughout the project.  You might find the IT Process Success Surveys to be of interest as several of them explored what success criteria projects actually have.  Interestingly, its rarely “on time, on budget, to specification”.
6. Transitioning to DAD: How would you typically phase a DAD implementation, let's say in a project pilot where they haven't been exposed yet to Agile?  Same question, any variance if the development team has already started Scrum?  This is a fairly complex question to answer.  The short story is that you need to invest some time to understand what your strengths and weaknesses are so that you can identify what you need to focus on.  You will then likely need to pilot strategies/techniques which are new to your organization before rolling them out widely.  You may also need to invest in training and coaching/mentoring depending on your needs.  My company, Scott Ambler + Associates, offers these sorts of services and more for organizations interesting in adopting disciplined agile strategies. 
7. Skills: I am aware that ideally developer should also be able to test but in reality theses are usually separate roles. Is it in line with your beliefs\suggestions?  Many organizations that are new to agile still have roles that reflect their existing strategy.  Non-agile approaches often have people in specific roles such as programmer, tester, designer, and so on. In DAD we promote a different set of agile roles that reflect agile thinking.  The implication is that you’re going to have to help individuals transition over to the new way of thinking, something we cover in Chapter 4 of the DAD book.  You might also find Mark Lines' blog, No role in DAD for an Analyst?  to provide some insights into issues surrounding the transition from traditional to agile roles.
8. Teaming: What happens with the Product Owner and the Architecture Owner don’t Agree?  See my blog What Happens When People Don’t Agree 
9. Skillsets: Different people of the team have different skills, experience, and time horizons.  We can't all be generalists, can we?  There are several agile roles in DAD, each of which have different rights, responsibilities, and skillsets.  So we’re not promoting the idea that everyone have the same skillset.  However, we do promote the philosophy that people should strive to be T-skilled generalizing specialists so as to improve their productivity.
10. Teaming: What are your thoughts on team cohesion?  Teams will gel over time.  Being co-located helps.  Having people who are dedicated 100% to the team helps.  Building a team of people who want to be there helps.  Self organization helps. 
11. Tool support: Is there an RMC plug-in for DAD + Is there any software behind DAD... or some software that supports it... such as Jazz (RTC) for Agile/Scrum?  Yes, IBM Rational does in fact have an RMC plug in for DAD.  There is also a template for Microsoft TFS from RDA Corp, Software Development Expert’s Practices Advisor supports DAD comprehensively, and I’m currently working with MethodPark to do so too.    Stay tuned to the Disciplined Agile Delivery website for information about tool support.
12. Governance: Regarding enterprise governance and enterprise IT, how do you start taking a culture that imposes common-process, common-tools, central (often outsourced) IT services, and heavyweight stage/gate across all organizations, and get that evolving toward a (still enterprise aware) lean/agile approach?  This is a hard one.  I’m often called into organizations to help with this very issue.  The challenge is that you need to have a deep understanding of IT governance techniques as well as how to govern agile teams.  Governance is something we discussed in detail in the DAD book as well as on the DAD site, see Adopting Agile Governance Requires Discipline. 
13. Executable specifications: TDD was mentioned, can you comment on Behavior Driven Development (BDD) and any impact on this lifecycle?  BDD is a slight nuance to acceptance test-driven development (ATDD).  BDD/ATDD and TDD are both potential practices that you might choose to follow on a DAD team.  I’ve written a fair bit about agile testing and quality strategies and about TDD in detail.
14. Architecture: When does the first version of architecture gets established and what would require to establish it? You typically start thinking about architecture early in a DAD project during Inception following a practice called architecture envisioning.  I’ve also written a fair bit about agile architecture techniques and the potential misconceptions about agile architecture that you may find interesting.
15. Travel plans: Will you be coming to South Africa at some point?  The good news is that Mark Lines, my co-author, will be in South Africa the week of May 20 in Joburg and the following week in Capetown.  Contact us for details.  I hope to be visiting SA later this year but exact dates haven’t been set yet.  So, please stay tuned on Twitter at @scottwambler for further announcements. 
16. Other travel plans: If you'd like me to speak at a local event, including corporate conferences or training events, please contact me. 

 

 

 

 

The Disciplined Agile Consortium recently launched a certification programme for practitioners of Disciplined Agile Delivery (DAD).   There are three practitioner certifications:

1. Disciplined Agile Yellow Belt. This beginner certification indicates to colleagues and employers that you are eager to learn Disciplined Agile Delivery (DAD) strategies that enable you to increase your skills and abilities as a software professional.
2. Disciplined Agile Green Belt. This intermediate certification indicates that you are experienced at DAD and are on your way to becoming a generalizing specialist. You have the potential to be a “junior coach” under the guidance of a senior coach (someone who is likely a Disciplined Agile Black Belt).
3. Disciplined Agile Black Belt. This expert certification indicates that you are a trusted expert with significant proficiency at DAD. You can coach other people in disciplined agile strategies and advise organizations in the adoption and tailoring of the DAD framework.

Differentiate yourself in the marketplace. Certification in Disciplined Agile Delivery (DAD) means something to clients and employers because it needs to be earned. Certification in DAD tells the marketplace you understand how to deliver an agile solution from end-to-end with experience in enterprise-class development.

Again and again I've seen IT organizations suffering from what I call the "Bureaucracy is Discipline" antipattern. For example, filling out forms and reviewing documents are both bureaucratic activities, neither of which require significant skill nor discipline to accomplish. However, agile practices such as developing potentially shippable software every iteration is easy to say but requires great discipline to accomplish. Respecting the decisions of your stakeholders, particularly those pertaining to requirements prioritization, is easy to talk about but proves to require great discipline in practice (particularly when you don't agree with a decision). It's easy to talk about taking a test-driven approach to development, but in practice it requires significant skill and discipline to actually do.

A "process smell" which indicates that your organization is suffering from this antipattern is a focus on following repeatable processes instead of focusing on repeatable results. An example of repeatable processes is following the same route to work every day regardless of driving conditions. An example of repeatable results is getting to work on time every day, but being willing to change your route as required, bicycling into work instead of driving, taking public transit, and so on. Nobody really cares how you get to work each day (the process), what they really care about is that you got to work on time (the result). Sadly, we've been told for decades now that repeatable processes are critical to our success in IT, yet when you step back and think about that's really a reflection of a bureaucratic approach. On the other hand, a focus on repeatable results is a reflection of a more disciplined approach. Interestingly, the DDJ 2008 Process Framework survey found that given the choice that people would much rather have repeatable results over repeatable processes when it comes to IT.

Mistaking bureaucracy for discipline, or rigour if you prefer that term, is a reflection of the cultural damage that has occurred over the years in IT organizations as the result of traditional philosophies and techniques. Unfortunately, this mistaken belief is a significant inhibitor to software process improvement (SPI) efforts, in particular agile adoption efforts, which must be addressed if you're to be successful. Overcoming this challenge will require a significant cultural shift in some organizations, and many people (particularly the bureaucrats) will find this uncomfortable.

Further reading:

• The Discipline of Agile
• Measured Capability Improvement Framework (MCIF)
• Repeatable Results Over Repeatable Processes

I'd like to leave you with this parting thought: Bureaucracy is bureaucracy and discipline is discipline, please don't confuse the two.[Read More]

I recently recorded an audio podcast about Collaborative Development and Operations (DevOps) and how it relates to Disciplined Agile Delivery (DAD).  The podcast is 17 minutes long and covers a range of topics including:

• What are the challenges typically faced by Disciplined Agile Development teams as they begin to transition their working solutions into production?
• How best could this gap between Development and Operations be closed so that they worked collaboratively rather than in silos?
  
• If this DevOps gap was decreased or even closed, what would the measurable value be to Agile projects and their business stakeholders?

On a related note, IBM's Collaborative Development and Operations landing page has some great resources if you're interested in how to implement.

Although it might not be obvious, and important success factor in adopting agile techniques is to be able to determine whether a team is agile or not. The challenge that many organizations face is that many teams will claim to be agile, yet management, who often has little or no experience with agile approaches, cannot tell which claims are true and which are over zealous (I'm being polite). The following are the criteria that I suggest you look for in a disciplined agile team:1. Produce working software on a regular basis. This is one of the 12 principles behind the Agile Manifesto, and in my experience is a critical differentiator between the teams that are agile and those that are merely claiming it. Ideally the team should produce potentially shippable software each iteration. That doesn't mean that they'll deploy the system into production, or the marketplace, each iteration but they could if required to do so. Typically the team will deploy into a pre-production testing environment or a demo enviroment at the end of each iteration (or more often for that matter).2. Do continuous regression testing, and better yet take a Test-Driven Development (TDD) approach. Agile developers test their work to the best of their ability, minimally doing developer regression testing via a continuous integration (CI) strategy and better yet do developer-level TDD. This approach enables development teams to find defects early, thereby reducing the average cost of addressing the defects, it also helps them to deliver higher quality code and to move forward safely when adding or changing functionality.3. Work closely with their stakeholders, ideally on a daily basis. A common practice of agile teams is to have an on-site customer or product owner who prioritizes requirements and provides information on a timely manner to the team. Disciplined agile teams take it one step further and follow the practice active stakeholder participation where the stakeholders get actively involved with modeling and sometimes even development.4. Are self-organizing within a governance framework. Agile teams are self-organizing, which means that the people doing the work determines how the work will be done, they're not told by a manager who may not even be directly involved with the work how it will be done. In other words the team does its own planning, including scheduling and estimation. Disciplined agile teams are self governing within an effective governance framework.5. Regularly reflect on how they work together and then act to improve on their findings. Most agile teams hold a short meeting at the end of each iteration to reflect upon how well things are working and how they could potentially improve the way that they are working together. Sometimes this is done in a more formalized manner in the form of a retrospective, but often it's done informally. The team then acts on one or more of their suggested improvements the next iteration. Disciplined agile teams take this one step further and measure their software process improvement (SPI) progress over time: the act of taking these measures, perhaps via a product such as Rational Self Check, helps to keep the team on track in their SPI efforts.

I have yet to discover an ad-hoc development team which met all five criteria, and most of them rarely meet two or three.

Further reading:

• Lean Development Governance
• Examining the Agile Manifesto
• Norm Kerth's Retrospectives Home Page
• Measured Capability Improvement Framework (MCIF)

[Read More]

When adopting agile software development techniques across a large number of teams within your organization it is important to provide a definition for what agile software development is, in addition to criteria for what it means to be agile. Many people will point to the four values of the Agile Manifesto and claim that's a good definition. Well... it might be a good definition for the visionaries and early adopters among us, but for people on the right-hand side of the technology adoption chasm (the early majority, late majority and the laggards) this isn't enough. Don't get me wrong, I'm a firm believer in the agile values but I like to cast them as philosophies instead of as a definition.

At IBM Software Group, the definition of disciplined agile software delivery which we have been sharing with our customers is:Disciplined agile software delivery is an evolutionary (iterative and incremental) approach to delivery which regularly produces high quality software in a cost effective and timely manner. It is performed in a highly collaborative and self-organizing manner, with active stakeholder participation to ensure that the team understand and addresses the changing needs of its stakeholders. Disciplined agile delivery teams provide repeatable results by adopting just the right amount of ceremony for the situation which they face.

I think that this is a pretty good definition, although I have no doubt that we'll evolve it over time.

I also suspect that the agile community will never settle on a common definition for what agile is and more than likely are smart enough not to even try. ;-)
Further reading:

• Agile Criteria
• Agile Manifesto
• Rational's Agile Software Development Home Page (for webcasts, whitepapers, ...)

[Read More]

At IBM Rational we define disciplined agile delivery as:

Disciplined agile delivery is an evolutionary (iterative and incremental) approach which regularly produces high quality solutions in a cost effective and timely manner via a risk and value driven life cycle. It is performed in a highly collaborative, disciplined, and self-organizing manner within an appropriate governance framework, with active stakeholder participation to ensure that the team understands and addresses the changing needs of its stakeholders to maximize business value provided. Disciplined agile delivery teams provide repeatable results by adopting just the right amount of ceremony for the situation which they face.

Let’s explore the key points in this definition:

1. Full delivery life cycle.  Disciplined agile delivery processes have life cycles which are serial in the large and iterative in the small.  Minimally they have a release rhythm which recognizes the need for start up/inception activities, construction activities, and deployment/transition activities.  Better yet, they include explicit phases as well. It is very important to note that these are not the traditional waterfall phases – requirements, analysis, design, and so on – but instead different “seasons” of a project. The point is that we need to look beyond agile software development and consider the full complexities of solution delivery.  Adopting a full delivery life cycle, not just a construction life cycle, is arguably the “zeroth” agile scaling factor. 
2. Evolutionary.  Agile strategies are both iterative and incremental in nature.  Iterative means that you are working in a non-serial manner, on any given day you may do some requirements analysis, some testing, some programming, some design, some more testing, and so on.  Incremental means that you add new functionality and working code to the most recent build, until such time as the stakeholder determines there is enough value to release the product.
3. Regularly produces high quality solutions. Agilists are said to be quality focused.  They prefer to test often and early, and the more disciplined ones even take a test-first approach where they will write a single test and the just enough production code to fulfill that test (then they iterate). Many agile developers have adopted the practice of refactoring, which is a technique where you make simple changes to your code or schema which improves its quality without changing its semantics.  Adoption of these sorts of quality techniques seems to work – it appears that agile teams are more likely to deliver high quality systems than traditional teams (according to the DDJ 2008 Project Success survey). Within IBM we take it one step further and focus on consumability, which encompasses quality and other features such as ease of deployment and system performance.  Furthermore, although some agile methods promote the concept of producing “potentially shippable software” on a regular basis, disciplined agile delivery teams produce solutions: a portion of which may be software, a portion of which may be hardware, and a portion of which will be the manner in which the system is used.
4. Cost effective and timely manner.  Agile teams prefer to implement functionality in priority order [http://www.agilemodeling.com/essays/prioritizedRequirements.htm], with the priority being defined by their stakeholders (or a representative thereof).  Working in priority order enables agile teams to maximize the return on investment (ROI) because they are working on the high-value functionality as defined by their stakeholders, thereby increasing cost effectiveness. Agile teams also prefer to produce potentially shippable solutions each iteration (an iteration is a time-box, typically 2-4 weeks in length), enabling their stakeholders to determine when they wish to have a release delivered to them and thereby improving timeliness.  Short iterations reduce the feedback cycle, improving the chance that agile teams will discover problems early (they “fail fast”) and thereby enable them to address the problems when they’re still reasonably inexpensive to do so.  The DDJ 2008 Project Success survey found that agile teams are in fact more likely to deliver good ROI than traditional teams and more likely to deliver in a timely manner.
5. Value driven life cycle. One result of building a potentially shippable solution every iteration is that agile teams produce concrete value in a consistent and visible manner throughout the life cycle. 
6. Risk and value driven life cycle.  Core agile processes are very clear about the need to produce visible value in the form of working software on a regular basis throughout the life cycle.  Disciplined agile delivery processes take it one step further and actively mitigate risk early in the life cycle – during project start up you should come to stakeholder concurrence regarding the project’s scope, thereby reducing significant business risk, and prove the architecture by building a working skeleton of your system, thereby significantly reducing technical risk.  They also help with transition to agile, allowing traditional funding models to use these milestones before moving to the finer grained iteration based funding that agile allows.
7. Highly collaborative.  People build systems, and the primary determinant of success on a development project is the individuals and the way that they work together.  Agile teams strive to work closely together and effectively as possible.  This is a characteristic that applies to both engineers on the team, as well as their leadership.
8. Disciplined.  Agile software development requires greater discipline on the part of practitioners that what is typically required by traditional approaches. 
9. Self organizing. This means that the people who do the work also plan and estimate the work.
10. Self-organization within an appropriate governance framework. Self-organization leads to more realistic plans and estimates which are more acceptable to the people implementing them.  At the same time these self-organizing teams must work within an appropriate governance framework which reflects the needs of their overall organizational environment.  An “appropriate governance framework” explicitly enables disciplined agile delivery teams to effectively leverage a common infrastructure, to follow organizational conventions, and to work towards organizational goals.  The point is that project teams, regardless of the delivery paradigm they are following, need to work within the governance framework of their organization.  More importantly, effective governance programs should make it desirable to do so.  Our experience is that traditional, command-and-control approaches to governance where senior management explicitly tells teams what to do and how to do it don’t work very well with agile delivery teams.  We’ve also found that lean development governance, an approach which is based on collaboration and enablement, is far more effective in practice.  Good governance increases the chance that agile delivery teams will build systems which fit into your overall organizational environment, instead of yet another stand-alone system which increases your overall maintenance burden and data quality problems.
11. Active stakeholder participation.  Agile teams work closely with their stakeholders, who include end users, managers of end users, the people paying for the project, enterprise architects, support staff, operations stuff, and many more.  Within IBM we distinguish between four categories of stakeholder: principles/sponsors, partners (business partners and others), end users, and insiders These stakeholders, or their representatives (product owners in Scrum, or on-site customers in Extreme Programming, or a resident stakeholder in scaling situations), are expected to provide information and make decisions in a timely manner.
12. Changing needs of stakeholders.  As a project progresses your stakeholders will gain a better understanding of what they want, particularly if you’re showing them working software on a regular basis, and will change their “requirements” as a result.  Changes in the business environment, or changes in organization priority, will also motivate changes to the requirements.  There is a clear need for agile requirements change management [http://www.agilemodeling.com/essays/changeManagement.htm] on modern IT projects.
13. Repeatable results.  Stakeholders are rarely interested in how you delivered a solution but instead in what you delivered.  In particular, they are often interested in having a solution which meets their actual needs, in spending their money wisely, in a high-quality solution, and in something which is delivered in a timely manner.  In other words, they’re interested in repeatable results, not repeatable processes.
14. Right amount of ceremony for the situation.  Agile approaches minimize ceremony in favor of delivering concrete value in the form of working software, but that doesn’t mean they do away with ceremony completely. Agile teams will still hold reviews, when it makes sense to do so.  DDJ’s 2008 Modeling and Documentation Survey found that agile teams will still produce deliverable documentation, such as operations manuals and user manuals, and furthermore are just as likely to do so as traditional teams.  The DDJ September 2009 State of the IT Union survey found that the quality of the documentation delivered by agile teams was just as good as that delivered by traditional teams, although iterative teams (e.g. RUP teams) did better than both agile and traditional.

 Suggested reading:

• The Agile System Delivery Life Cycle (SDLC)
• The Agile Scaling Model (ASM): Adapting Agile Methods for Complex Environments (an IBM Rational white paper)
• The Disciplined Agile Delivery (DAD) home page
  

This blog posting has been replaced by the more detailed article: Full Agile Delivery Lifecycles.  

Thank you for your patience.

 

 

The basic idea behind DevOps is that your development strategy and operations strategy should reflect one another, that you should strive to optimize the whole IT process.  This implies that development teams should work closely with your operations staff to deliver new releases smoothly into production and that your operations staff should work closely with development teams to streamline critical production issues.

DevOps has its source in agile software development, and it is an explicit aspect of the Disciplined Agile Delivery (DAD) process framework.  As a result there is a collection of agile development strategies which enable effective DevOps throughout the agile delivery lifecycle.  These strategies include:

1. Initial requirements envisioning.  Disciplined agile teams invest time at the beginning of the project to identify the high-level scope in a light-weight, collaborative manner.  This includes common operations requirements such as the need to backup and restore data sources, to instrument the solution so that it can be monitored in real time by operations staff, or to architect the solution in a modular manner to enable easier deployment. 
2. Initial architecture envisioning. Disciplined agile teams will also identify a viable architectural strategy which reflects the requirements of their stakeholders and your organization’s overall architectural strategy (hence the need to work closely with your enterprise architects and operations staff).  One goal is to ensure that the team is building (or buying) a solution which will work well with the existing operational infrastructure and to begin negotiating any infrastructural changes (such as deploying new technologies) early in the project.   Another goal is to ensure that operations-oriented requirements are addressed by the architecture from the very start. 
3. Initial release planning.  As part of release planning the disciplined agile team works closely with their operations group to identify potential release windows to aim for, any release blackout periods to avoid, and the need for operations-oriented milestone reviews later in the lifecycle (if appropriate).
4. Active stakeholder participation.  Disciplined agile teams work closely with their stakeholders, including both operations and support staff, all the way through the lifecycle to ensure that their evolving needs are understood.   
5. Continuous integration (CI).  This is a common technical agile practice where the solution is built/compiled, regression tested, and maybe even run through code analysis tools.  CI promotes greater quality which in turn enables easier releases into production.  
6. Parallel independent testing. For enterprise-class development or at scale, particularly when the domain or technology is very complex or in regulatory environments, disciplined agile team will find they need to support their whole team testing efforts with an independent test team running in parallel to the development team.  These testing issues often include validation of non-functional requirements – such as security, performance, and availability concerns – and around production system integration.  All of these issues are of clear importance for operations departments.   
7. Continuous deployment.  With this practice you automate the promotion of your working solution between environments.  By automating as much of the deployment effort as possible, and by running it often, the development team increases the chance of a successful deployment and thereby reduces the risk to the operations environment. Note that deployment into production is generally not automatic, as this is an important decision to be made by your operations/release manager(s). 
8. Continuous documentation.  With this practice supporting documentation, including operations and support documentation, is evolved throughout the lifecycle in concert with the development of new functionality. 
9. Production release planning.  This is the subset of your release planning efforts which focuses on the activities required to deploy into production. 
10. Production readiness reviews.  There should be at least one review, performed by the person(s) responsible for your operations environment, before the solution is deployed into production.  The more critical the system, the more product readiness reviews may be required. 
11. End-of-lifecycle testing.  Minimally you will need to run your full automated regression test suite against your baselined code once construction ends.  There may also be manual acceptance reviews or testing to be performed, and any appropriate fixing and retesting required to ensure that the solution is truly ready for production. 

For more detail about this topic, I think that you will find the article I wrote for the December 2011 issue of Cutter IT Journal entitled “Disciplined Agile Delivery and Collaborative DevOps” to be of value.

My new white paper, Disciplined Agile Delivery: An Introduction, is now available free of charge from IBM.com. The paper overviews the Disciplined Agile Delivery (DAD) process framework, a hybrid comprised of strategies from Scrum, XP, Agile Modeling, and other agile methods which is people first, learning oriented, and enterprise aware. DAD is the basis from which you can scale agile.

Topics covered:

• Context counts - The Agile Scaling Model
• People first - People, and the way they interact with each other, are the primary determinant of success for a solution delivery project.
• Learning-oriented - The DAD process framework promotes the ideas that team members should collaborate closely and learn from each other, that the team should invest effort to learn from their experiences and evolve their approach, and that individuals should do so as well.
• Hybrid - DAD adopts and tailors strategies from Scrum, XP, Agile Modeling, UP, Kanban, and many others. It addresses many of the issues Mark Kennaley discusses in SDLC 3.0.
  
• IT solution focused - DAD teams produce potentially consumable solutions every construction iteration.  This extends Scrum's "potentially shippable" strategy to explicitly address usability/consumability plus the fact that we're really delivering full solutions not just software.
• Goal-driven delivery life cycle - The DAD lifecycle is focused on delivery, not just construction.  Furthermore it is goals-driven, the DAD process framework suggests various strategies to fulfill those goals but does not prescribe specific practices.
  
• Risk and value driven - The DAD lifecycle is risk and value driven.  It extends Scrum's value-driven lifecycle which produces potentially shippable software each sprint/iteration so that it explicitly includes light-weight milesstones such as ensuring stakeholder consensus as to the scope of the project early in the lifecycle, proving the architecture with working code early in the lifecycle, ensuring sufficient functionality exists before transition, and ensuring production readiness before actual release of the solution.
  
• Enterprise aware - The DAD process framework promotes the ideas that DAD teams should work closely with their enterprise architecture groups to ensure they leverage and evolve the existing infrastructure, adopt and follow corporate guidelines, and work to the overall organizational vision.  DAD teams are self organizing with appropriate governance.

IBM Rational recently published an update to my Agility@Scale e-book, which can be downloaded free of charge.  The e-book is a 21 page, 2.3 meg PDF (sorry about the size, guess the graphics did it) .   It overviews the Agile Scaling Model (ASM) (which has since been replaced by the Software Development Context Framework (SDCF) ), Disciplined Agile Delivery (DAD), the scaling factors of agility at scale, and ends with some advice for becoming as agile as you can be.  In short it's a light-weight coverage of some of the things I've been writing about in this blog the past couple of years.  Could be a good thing to share with the decision makers in your organization if they're considering adoption agile strategies.

This article has been replaced by an official "Disciplined Agile Manifesto".

 

The text of the original article remains below.

---------------------------------------------------------------------------------------

I've recently been working with Mark Lines of UPMentors and we've had some interesting discussions around evolving the Agile Manifesto which I thought I would share here to obtain feedback.  Note that this is not any sort of official position of IBM, nothing in my blog is by the way (unless explicitly stated so), nor is it some sort of devious plot to take over the agile world (although if we did have some sort of devious plot, we'd make the exact same claim).  What we hope to accomplish is to put some ideas out there in the hopes of getting an interesting conversation going.

Over the past decade we’ve applied the ideas captured in the Agile Manifesto and have learned from our experiences doing so.  What we’ve learned has motivated us to suggest changes to the manifesto to reflect the enterprise situations which we have applied agile and lean strategies in.  We believe that the changes we’re suggesting are straightforward:

1. Where the original manifesto focused on software development, a term which too many people have understood to mean only software development, we suggest that it should focus on solution delivery.
2. Where the original focused on customers, a word that for too many people appears to imply only the end users, we suggest that it focus on the full range of stakeholders instead.
3. Where the original manifesto focused on development teams, we suggest that the overall IT ecosystem and its improvement be taken into consideration.
4. Where the original manifesto focused on the understanding of, and observations about, software development at the time there has been some very interesting work done within the lean community since then (and to be fair there was very interesting work done within that community long before the Agile Manifesto was written).  We believe that the Agile Manifesto can benefit from lean principles.

Our suggested rewording of the Agile Manifesto follows, with our suggested changes in italics.

Updating the Values of the Agile Manifesto

We are uncovering better ways of developing software by doing it and helping others do it.  Through this work we have come to value:

1. Individuals and interactions over processes and tools
2. Working solutions over comprehensive documentation
3. Stakeholder collaboration over contract negotiation
4. Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Updating the Principles behind the Agile Manifesto

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable solutions.
2. Welcome changing requirements, even late in the solution delivery lifecycle. Agile processes harness change for the stakeholder’s competitive advantage.
3. Deliver working solutions frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
4. Stakeholders and developers must work together daily throughout the project.
5. Build projects around motivated individuals.  Give them the environment and support they need, and trust them to get the job done.
6. The most efficient and effective method of conveying information to and within a delivery team is face-to-face conversation.
7. Quantified business value is the primary measure of progress.
8. Agile processes promote sustainable development.  The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
9. Continuous attention to technical excellence and good design enhances agility.
10. Simplicity--the art of maximizing the amount of work not done--is essential.
11. The best architectures, requirements, and designs emerge from self-organizing teams.
12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
13. Leverage and evolve the assets within your organizational ecosystem, and collaborate with the people responsible for those assets to do so.
14. Visualize workflow to help achieve a smooth flow of delivery while keeping work in progress to a minimum.
15. The organizational ecosystem must evolve to reflect and enhance the efforts of agile teams, yet be sufficiently flexible to still support non-agile or hybrid teams. 

We’re agile – things evolve, including manifestos.  Looking forward to your feedback (add a comment).

 

Updates Since this Was First Published:

• February 2011: At the 10th Anniversary of the Agile Manifesto get together several people, including myself, brought up the idea of updating the Agile Manifesto.  Unfortunately the decision by the original writers of the manifesto was to adopt a very strict, non-agile change management approach where any changes to the manifesto would need the approval of all 17 of them.  So, pretty much zero chance of there ever being any changes (hence the lack of changes so far). 
• June 2012: This extension to the manifesto was retitled "A Disciplined Agile Manifesto" and published as in the book Disciplined Agile Delivery written by myself and Mark Lines.
• November 2012: Mark and I formed Scott Ambler+Associates.
• February 2013: The Disciplined Agile Consortium (DAC) is formed.  The mission of DAC is to evolve and support the Disciplined Agile Delivery (DAD) framework.
• April 2014: Official Disciplined Agile Manifesto posted on the Disciplined Agile Delivery (DAD) site.

Further reading:

• The Declaration of Interdependence (some other thoughts about, arguably, evolving the manifesto)
• Examining the Agile Manifesto
• Disciplined Agile Delivery website

 

 

 

The explicit phases of the Unified Process -- Inception, Elaboration, Construction, and Transition -- and their milestones are important strategies for scaling agile software development to meet the real-world needs of modern organizations. Yes, I realize that this is heresy for hard-core agilists who can expound upon the evils of serial development, yet these very same people also take a phased approach to development although are loathe to admit it. The issue is that the UP phases are like seasons of a project: although you'll do the same types of activities all throughout a project, the extent to which you do them and the way in which you do them change depending on your goals. For example, at the beginning of a development project if you want to be effective you need to do basic things like identify the scope of the project, identify a viable architecture strategy, start putting together your team, and obtain support for the project. Towards the end of a project your focus is on the activities surrounding the deployment of your system into production, including end-of-lifecycle testing efforts, training, cleaning up of documentation, piloting the system with a subset of users, and so on. In between you focus on building the system, including analysis, design, testing, and coding of it. Your project clearly progresses through different phases, or call them seasons if the term phase doesn't suit you, whether your team is agile or not.

The UP defines four phases, each of which address a different kind of risk:1. Inception. This phase focuses on addressing business risk by having you drive to scope concurrence amongst your stakeholders. Most projects have a wide range of stakeholdres, and if they don't agree to the scope of the project and recognize that others have conflicting or higher priority needs you project risks getting mired in political infighting. In the Eclipse Way this is called the "Warm Up" iteration and in other agile processes "Iteration 0".2. Elaboration. The goal of this phase is to address technical risk by proving the architecture through code. You do this by building and end-to-end skeleton of your system which implements the highest-risk requirements. Some people will say that this approach isn't agile, that your stakeholders should by the only ones to prioritize requirements. Yes, I agree with that, but I also recognize that there are a wide range of stakeholders, including operations people and enterprise architects who are interested in the technical viability of your approach. I've also noticed that the high-risk requirements are often the high-business-value ones anyway, so you usually need to do very little reorganization of your requirements stack.3. Construction. This phase focuses on implementation risk, addressing it through the creation of working software each iteration. This phase is where you put the flesh onto the skeleton.4. Transition. The goal of this phase is to address deployment risk. There is usually a lot more to deploying software than simply copying a few files onto a server, as I indicated above. Deployment is often a complex and difficult task, one which you often need good guidance to succeed at.

Each phase ends with a milestone review, which could be as simple as a short meeting, where you meet with prime stakeholders who will make a "go/no-go" decision regarding your project. They should consider whether the project still makes sense, perhaps the situation has changed, and that you're addressing the project risks appropriately. This is important for "agile in the small" but also for "agile in the large" because at scale your risks are often much greater. Your prime stakeholders should also verify that you have in fact met the criteria for exiting the phase. For example, if you don't have an end-to-end working skeleton of your system then you're not ready to enter the Construction phase. Holding these sorts of milestone reviews improves your IT governance efforts by giving senior management valuable visibility at the level that they actually need: when you have dozens or hundreds of projects underway, you can't attend all of the daily stand up meetings of each team, nor do you even want to read summary status reports.

These milestone reviews enable you to lower project risk. Last Autumn I ran a survey via Dr. Dobb's Journal (www.ddj.com) which explore how people actually define success for IT projects and how successful we really were. We found that when people define success in their own terms that Agile has a 71% success rate compared with 63% for traditional approaches. Although it's nice to that Agile appears to be lower risk than traditional approaches, a 71% success rate still implies a 29% failure rate. The point is that it behooves us to actively monitor development projects to determine if they're on track, and if not either help them to get back on track or cancel them as soon as we possibly can. Hence the importance of occasional milestone reviews where you make go/no-go decisions. If you're interested in the details behind the project, they can be found at http://www.ambysoft.com/surveys/success2007.html .

Done right, phases are critical to your project success, particularly at scale. Yes, the traditional community seems to have gone overboard with phase-based approaches, but that doesn't mean that we need to make the same mistakes. Let's keep the benefit without the cost of needless bureaucracy.[Read More]

• Examining the Agile Manifesto 
• Reworking the Agile Manifesto for Enterprise Agile
• Scaling Agile: Start with a Disciplined Foundation
• Scaling Agile: The Software Development Context Framework
• Disciplined Agile Delivery (DAD) home page
• Generalizing Specialists: Improving Your IT Skills

A common question that I keep running into with customers is whether you can take an agile approach to service oriented architecture (SOA). The quick answer is yes, because Agile is orthogonal to the implementation technologies used. You can take an agile approach developing COBOL applications running on mainframes, fat-client Java applications, multi-tier J2EE applications, and yes, even services. Granted, it's easier to do with some technologies than others, either because of the nature of the technology or because of the supporting tools.

The long answer is "yes, but". You don't adopt an SOA approach for the sheer joy of doing so, instead you very likely want to improve the level of reuse within your organization. To succeed at SOA-driven reuse you need an enterprise focus, something that doesn't appear to be very common on many agile teams. Therein lies the challenge. Several strategies for improving your chances with Agile SOA, and SOA in general, follows:1. Invest in some initial enterprise architecture modeling. You don't need to identify all of the details up front, that would take too long and actually put the effort at risk, but you do need to set a starting point to guide development teams. Identifying the technical architecture is critical, and identifying a few basic services which would provide immediate business value to one or more teams is critical. Involve people from several application project teams to ensure that you get a wide range of input. See http://www.agiledata.org/essays/enterpriseArchitecture.html for a streamlined approach to enterprise architecture modeling. Creating big, detailed models often proves to be a waste of time because development teams are rarely motivated to read mounds of documentation.2. Build out the initial infrastructure on a real application development project. This proves that your SOA strategy actually works and puts the technical foundation in place for future teams. During this period you'll be tempted to try to support several development teams, which is feasible but dramatically increases your risk. It's also tempting to focus simply on getting the infrastructure in place without delivering any business functionality, but this risks producing an ivory-tower architecture that nobody is interested in.3. Spread the service architects out onto application development teams. The people that formulated and then proved your SOA should be actively involved on the development teams that are working with it to ensure that the teams use it appropriately and to ensure that the architects get concrete feedback which they can use to evolve the architecture. When working on agile teams, these people will need to work in a collaborative and evolutionary approach just like other team members.4. Fund reuse separately. I've lost track of the number of organizations that I've run into that fail at reuse because their development teams never have the resources to develop reusable assets. That's simply the nature of the beast -- project teams will always be more interested in addressing their own specific requirements than they are in investing the time and effort to make something reusable. The real problem here is that you expect them to act differently. A better strategy is to have a separate reuse engineering team that has the resources to monitor existing projects to look for potentially reusable assets. When they find said assets this team does the work to harvest the asset, to reengineer it to make it reusable, and then to integrate back into the original source project. The goal is to make it as painless as possible to produce reusable assets such as services. If you expect project teams to do this work out of the goodness of their hearts then you're effectively punishing them when they do the right thing. That's not a very good governance strategy, IMHO.5. The reuse team now owns the asset. Any reusable asset, including services, will need to be maintained, evolved over time, and supported. This isn't free nor is it viable for project teams to do so.

If you're interested, I provide agile strategies for both enterprise architecture and strategic reuse in the book "Enterprise Unified Process". Although written under the assumption that you're taking a RUP-based approach to development, the reality is that the EUP can extend any evolutionary/agile software development process so that it addresses the larger-scale needs of modern IT organizations.

- Scott[Read More]

There is a distinct rhythm, or cadence, at different levels of the agile process. We call this the agile 3C rhythm, for coordinate, collaborate, and conclude (which is sometimes called stabilize). The agile 3C rhythm occurs at three levels in Disciplined Agile Delivery (DAD):

1. Day. A typical day begins with a short coordination meeting, called a Scrum meeting in the Scrum method. After the daily coordination meeting the team collaborates throughout most of the day to perform their work. The day concludes with a working build, hopefully you had several working builds throughout the day, which depending on your situation may require a bit of stabilization work to achieve.
2. Iteration. DAD construction iterations begin with an iteration planning session (coordinate) where the team identifies a detailed task list of what needs to be done that iteration. Note that iteration modeling is often part of this effort. Throughout the iteration they collaborate to perform the implementation work. They conclude the iteration by producing a potentially consumable solution, a demo of that solution to key stakeholders, and a retrospective to identify potential improvements in the way that they work.
3. Release. The DAD lifecycle calls out three explicit phases - Inception, Construction, and Transition – which map directly to coordinate, collaborate, and conclude respectfully.

The agile 3C rhythm is similar conceptually to Deming’s Plan, Do, Check, Act (PDCA) cycle:

• Coordinate maps to plan
• Collaborate maps to do
• Conclude maps to check and act

1. You adopt "standard" agile discipline.   Aspects of agile which require discipline include adopting practices such as test-driven development (TDD), active stakeholder participation, working collaboratively, shortening the feedback cycle, and many more.  These strategies are a great start to becoming disciplined IT professionals. 
2. You take a goal-driven approach.  When we first started working on the DAD framework I didn't want to create yet another prescriptive framework, particularly given Rational's track record with the Rational Unified Process (RUP) framework.  Rational has been pilloried for years for the prescriptive nature of RUP, which is unfortunate because there are a lot of great ideas in RUP that agile teams can benefit from, some of which we adopted in DAD and many of which are being actively reinvented with the agile community even as you read this. Furthermore, there are many prescriptive elements of the Scrum method that can get teams in trouble.  For example, Scrum prescribes that you hold a daily stand up meeting, often called a Scrum meeting, where everyone should answer three questions.  That's a great approach for teams new to agile, but it proves problematic in many situations due to it's prescriptive nature. Do you really need to do this once a day?  I've been on teams where we held coordination meetings twice a day and others only once a week.  Do you really need to stand up?  I've been on geographically distrubited agile teams where many of us were sitting down during coordination calls.  Do you really need to answer three questions, two of which are clearly focused on status regardless of claims otherwise?  I've been on lean teams where we met around our Kanban board and focused on potential blockers.  The answers to these questions depends on the context of the situation you find yourself in.  The challenge, at least from the point of view of a process framework, is how do you avoid falling into the trap of being overly prescriptive.  The strategy we adopted in DAD is to take a goal-driven approach.  The observation is that regardless of the situation you find yourself in there are common goals your team will need to fulfill.  For example, at the beginning of a project common goals include developing an initial plan, initially exploring the scope, initially identifying a technical strategy, and securing initial funding (amongst others).  Throughout construction you should coordinate your activities, improve the quality of your ecosystem, and produce a potentially consumable solution on a regular basis (more on this below).  So, instead of prescribing a daily stand up meeting the DAD framework instead indicates you should coordinate your activities, and gives several options for doing so (one of which is a Scrum meeting).  More importantly DAD describes the advantages and disadvantages of your options so that you can make the choice that's best suited for the situation your team finds itself in (see this blog posting for a detailed example of the types of tables included in the DAD book to help you through such process tailoring decisions).  In short, our experience is that it requires discipline to take a goal driven approach to agile delivery over the prescriptive strategies in other agile processes. 
3. You take a context-driven approach.  There are many tailoring factors, which I describe in the Software Development Context Framework (SDCF), which you need to consider when making process, tooling, and team structure decisions.  For example, a large team will adopt a different collection of practices and tools than a small team.  A geographically distributed team will adopt a different strategy than a team that is co-located.  You get the idea.  Other tailoring factors include compliance, team culture, organization culture, technical complexity, domain complexity, and project type.  It requires discipline to recognize the context of the situation you find yourself in and then act accordingly.
4. You deliver potentially consumable solutions.  One of the observations that we made early in the development of the DAD framework was that disciplined agile teams produce potentially consumable solutions, not just potentially shippable software.  Although delivery of high-quality, working software is important it is even more important that we deliver high-quality working solutions to our stakeholders. For example, not only are we writing software but we may also be updating the hardware on which it runs, writing supporting documentation, evolving the business processes around the usage of the system, and even evolving the organizational structure of the people working with the system.  In other words, disciplined agilists focus on solutions over software.  Furthermore, "potentially shippable" isn't sufficient: not only should it be shippable but it should also be usable and should be something people want to use.  In other words it should be consumable (a concept DAD adopted from IBM's Outside In Development).  Minimally IT professionals should have the skills and desire to produce good software, but what they really need are the skills and desire to provide good solutions. We need strong technical skills, but we also need strong "soft skills" such as user interface design and process design to name just two. The incremental delivery of potentially consumable solutions on an incremental basis requires discipline to do successfully.  DAD teams focus on repeatable results not repeatable processes.
5. You are enterprise aware.   Whether you like it or not, as you adopt agile you will constrained by the organizational ecosystem, and you will need to act accordingly. It takes discipline to work with enterprise professionals such as enterprise architects, data admistrators, portfolio managers, or IT governance people who may not be completely agile yet, and have the patience to help them. It takes discipline to work with your operations and support staff in a DevOps manner throughout the lifecycle, particularly when they may not be motivated to do so.  It requires discipline to accept and potentially enhance existing corporate development conventions (programming guidelines, data guidelines, UI guidelines, ...).  It requires discipline to accept that your organization has an existing technology roadmap that you should be leveraging, building out, and in some cases improving as you go.  In short, enterprise awareness requires a level of discipline not typically seen on many agile teams.
6. You adopt a full delivery lifecycle.  Empirically it is very easy to observe that at the beginning of an agile project there are some activities that you need to perform to initiate the project.  Similarly at the end of the project there are activities that you need to perform to release the solution into production or the marketplace.  The DAD process framework addresses the effort required for the full delivery effort, including project initiation, construction, and deployment.  Our experience is that it requires discipline on the part of IT professionals to include explicit phases for Inception/Initation, Construction, and Transition/Deployment and more importantly to focus the appropriate amount of effort on each.  One danger of explicit phases is that you run the risk of taking what's known as a Water-Scrum-Fall approach, a term coined by Dave West the person who wrote the forward for the DAD book, where you take an overly heavy/traditional approach to inception and transition in combination with a lighter agile approach to construction.  Water-Scrum-Fall occurs because many organizations haven't made a full transition to agile, often because they think it's only applicable to construction.  Our experience is that you can be very agile in your approach to inception and transition, experience we've built into the DAD framework.  Having said that it clearly requires discipline to keep inception activities short and similarly it requires discipline to reduce the "transition phase" to an activity. 
7. You adopt a wider range of roles.  An interesting side effect of adopting a full delivery lifecycle is that you also need to adopt a more robust set of roles.  For example, the Scrum method suggests three roles - Scrum Master, Product Owner, and Team Member - a reflection of the Scrum lifecycle's construction focus.  DAD suggests three primary roles - Team Lead, Product Owner, Team Member, Architecture Owner, and Stakeholder - as well as five secondary roles which may appear at scale.   
8. You embrace agile governance.  Governance establishes chains of responsibil­ity, authority and communication in support of the overall enterprise’s goals and strategy. It also establishes measurements, policies, standards and control mechanisms to enable people to carry out their roles and responsibilities effectively. You do this by balancing risk versus return on investment (ROI), setting in place effective processes and practices, defining the direction and goals for the department, and defining the roles that people play with and within the department.  It requires discipline to adopt an agile approach to governance, and that's something built right into the DAD framework.

Recently I have been asked by several customer organizations to help them to understand how to account for the expense of agile software development. In particular, incremental delivery of solutions into production or the marketplace seem to be causing confusion with the financial people within these organizations. The details of accounting rules vary between countries, but the fundamentals are common.  In order to get properly account for the costs incurred by software development teams you need to keep track of the amount of work performed and the type of work performed to develop a given solution. Time tracking doesn't have to be complex: at one customer developers spend less than five minutes a week capturing such information.

 

Why is Time Tracking Potentially Valuable?

There are several financial issues to be aware of:

• Capitalization. For public companies capital expenses (CapEx) can potentially boost book value through the increase in assets (in this case a software-based solution) and increase in net income (due to lower operating expenses that year). On the other hand, operational expenses (OpEx) are accounted for in the year that they occur and thereby reduce net income which in turn reduces your organization's taxes for that year.
• Matching. One of the goals of good accounting is to accurately reflect the net income of the enterprise and to prevent income manipulation or "smoothing". As such a key tenet is the principle of matching revenues with the appropriate expenses. For software this means that we expense the cost of the software over the lifetime of the asset against the income at that time. An implication of this is that capitalizing software development, when appropriate, before the software goes into production clearly violates the matching principle since there is no benefit of the asset until such time.
• Tax Credits.  In some countries you can even get tax credits for forms of software development that are research and development (R&D) in nature.

The point is that the way that a software developer's work is accounted for can have a non-trivial impact upon your organization's financial position.

 

What Do Agilists Think of Time Tracking?

So, I thought I'd run a simple test. Last week on LinkedIn's Agile and Lean Software Development group I ran a poll to see what people thought about time tracking. The poll provided five options (a limitation of LinkedIn Polls) to choose from:

• Yes, this is a valuable activity (33% of responses)
• Yes, this is a waste of time (39% of responses)
• No, but we're thinking about doing so (2% of responses)
• No, we've never considered this (18% of responses)
• I don't understand what you're asking (5% of responses, one of which was mine so that I could test the poll)

The poll results reveal that we have a long way to go. Of the people inputting their time more of them believed it was a waste of time than understood it to be a valuable activity. When you stop and think about it, the investment of five minutes a week to track your time could potentially save or even earn your organization many hundreds of dollars. Looking at it from a dollar per minute point of view, it could be the highest value activity that a developer performs in a given week.

The discussion that ensued regarding the poll was truly interesting. Although there were several positive postings, and several neutral ones, many more were negative when it came to time tracking. Some comments that stood out for me included:

• It's a colossal waste of time unless you're billing a customer by the hour.
• We record time spent on new development work (as distinct from other tasks such as bug fixing in legacy code and so on) as this is capitalised as an asset and depreciated.
• I think the *most* pointless example was where the managers told us what we should be putting in.
• One day we will move past the "just do it" mentality and have some meaningful conversations and the reasons for what we do.
• In my experience time tracking is a massive waste... of time. It's a poor substitute for management.
• Why do you need to know more than the info available through Sprint Backlog, Sprint burndown and the daily standup?
• Some of my teams (I am SM for three teams) are skeptical about this. They do not think that keeping track of task hours this way will be any more useful than the daily standup reports. And they do not believe that Management can resist the temptation to use task hours as a measure.

I think that there are several interesting implications from this discussion:

1. Agilists need to become more enterprise aware. It's clear to be really effective that agile delivery teams need a better understanding of the bigger picture, including mundane things such as tax implications of what they're doing.  In Disciplined Agile Delivery (DAD) this is something that we refer to as being enterprise aware.  There's far more to enterprise awareness than understanding pertinent accounting principles, for interest disciplined agile teams work towards a common technology roadmap and common business roadmap, but appreciating why time tracking is a potentially valuable activity would be a good start.
2. Management needs to communicate better.  It's also clear that management needs to communicate more effectively regarding why they're asking people to track their time. To be fair, management themselves might not be aware of the tax implications themselves so may not be making effective use of the time data they're asking for.
3. Management needs to govern more effectively. Several people were clearly concerned about how management was going to use the time data (by definition they are measures) which could be a symptom of both poor communication as well as poor governance (unfortunately many developers have experiences where measures have been used against them, a failure of governance, and no longer trust their management teams to do the right thing as a result).
4. Time tracking should be streamlined.  It was obvious from the conversation that several people worked in organizations where the time tracking effort had gotten completely out of hand.  Spending 5 minutes a week is ok, and to be quite blunt should be more than sufficient, but spending fifteen minutes or more a day doing so is far too much.  Over the years I've helped organizations design measurement programs and I've seen a lot of well-intention efforts become incredibly onerous and expensive for the people they were inflicted upon.  I suspect it's time for a reality check in some of these organizations people were alluding to.  A good heuristic is that for any measurement you should be able to indicate the real cost of collecting it, the use(s) that the information is being put to, and the value resulting from those uses.  If you can't quickly and coherently do that then you need to take a hard look at why you continue to collect that metric.  The lament "we might need it one day" is a symptom that you're wasting time and money.
5. Agile rhetoric is getting in the way.  Some of the team-focused agile practices, such as burndown charts (or better yet ranged burndown charts) and stand up meetings may be preventing people from becoming enterprise aware because they believe that all of their management needs are being met by them.
6. You may be missing out on the benefits of time tracking.  Many organizations are potentially leaving money on the table by not being aware of the implications of how to expense software development.

 

Parting Thoughts

Disciplined agilists are enterprise aware. This is important for two reasons: First, you want to optimize your organizational whole instead of sub-optimize on project-related efforts; second, you can completely miss opportunities to add real value for your organization. In the anecdote I provided it was clear that many agile developers believe that an activity such as time tracking is a waste when that clearly doesn't have to be the case. Worse yet, although someone brought up the issues around capitalizing software development expenses early in the conversation a group of very smart and very experienced people still missed this easy opportunity to see how they could add value to their organization.

Granted, time tracking on an agile project team is nowhere near as sexy as topics such as continuous integration (CI), TDD, the definition of done, continous architecture, or many more.  But you know what?  Although it's a mind-numbingly mundane issue it is still an important one. 'Nuff said (I hope).

 

Related Reading

• Time Tracking and Agile Software Development
• Strategies for Tracking Time on Agile Teams