Agility@Scale: Strategies for Scaling Agile Software Development

And now, a brief message from our sponsor. ;-)

Just wanted to let you know that Agility@Scale will be a major theme at this year's IBM Rational Software Conference being held in Orlando May 31 through to June 4. I'll be giving two presentations, Agility at Scale: Mitigating the Percieved Risks of Agile Adoption and Distributed Agile Software Development: Best Practices, facilitating a panel entitled Experience Reports from the Agile Trenches: What's Really Working, What's Not Working, and Why, and running a half-day workshop entitled Become as Agile as You Need to Be: Scaling Agile Software Development to Meet Your Real-World Situation. As usual, it will be a busy conference for me. ;-)

There are of course other sessions and workshops specific to agile topics. Some highlights include:

• Agility at Scale: Scrum and Rational Team Concert (a half-day workshop)
• Achieving Business Agility with Enterprise Architecture by Chris Armstrong
• Agile Enterprise Development with Groovy and Grails by Bjorn Beskow
• A Scalable "Just Enough" Approach to Requirements by Kurt Bittner
• Collaboration Strategies for Agile Quality Management by Anne Burke
• Agility at Scale: Agile Planning and Best Practices with IBM Rational Team Concert by Erich Gamma
• Agile Model-Driven Development for Embedded Systems by Bruce Douglass
• What they don''t teach you about software at school: Be Smart! by Ivar Jacobson
• Social Networking for Agile Developers by Anthony Kesterton
• Agile Modeling: No, It's Not an Oxymoron by Terry Quatrani
• Collaboration Technologies for Globally Distributed Agile Development Teams by Brad Sandler
• Long Distance Running - Sprints and Iterations for Large Projects by Ian Spence
• IBM(R) Rational(R) Quality Manager in a Globally Distributed World by Peter Sun
• Can Agility and IBM(R) Rational(R) Unified Process(R) Coexist in the Workplace? Yes! by Kim Werner
• Agile Requirements at Scale: Putting User Stories in Context by Tina Zhuo, Cherifa Mansoura, and myself.

Of course there's a lot more going on at the conference when it comes to scaling agile software development than just these sessions, but I thought I'd give you a feel for what's going on. Hope to see you at the conference. There's a "March Madness" sale going on right now at the registration page where if you purchase one pass you get the second for half price.


Further reading:

• IBM Rational Software Conference 2009
• Rational's Agile Software Development Home Page (for webcasts, whitepapers, ...)

[Read More]

I'm happy to announce Rational's Agility at Scale poster which is currently available free of charge from the Rational poster order page.

Although I'm obviously biased as I was involved with its creation, the poster includes some really good information about how to scale agile software development practices effectively. It summarizes the scaling factors, such as large teams, distribution, regulatory compliance, application complexity, and so on that you should be concerned with. It also presents a full agile software development lifecycle that goes beyond the construction focused lifecycles of common "agile in the small" processes. It also summarizes the practices of lean development governance, giving you some insight into how to govern your IT environment more effectively.

I think it's a pretty good poster that's worth checking out. It's free, so it's hard to argue with the price. Most importantly, it would look pretty good hanging on your office wall.

[Read More]

I recently wrote an "e-book" for Internet Evolution overviewing agile software development at scale. The goal of the Agility at Scale: Become as Agile as You Can Be ebook is to get people thinking outside of the box a bit when it comes to agile development strategies and see that they really are ready for primetime.





[Read More]

I recently did the voice over for our Rational.Everyware Agility@Scale whiteboard video, which is a bit less than 4 minutes in length.  As I narrate a whiteboard sketch evolves, sometimes using sticky-notes and index cards, to help explain what I'm talking about. 

In the video, I describe:

• The history of agile
• Some of the challenges surrounding traditional development
• Some of the basics of agile, such as increased collaboration with stakeholders and on delivery of consumable solutions
• The benefits of agile, including increased quality, time to value, stakeholder satisfaction, and ROI
• Domains where agile is being applied successfully
• What IBM agility@scale is all about
• How some of the scaling factors change the way that you'll work and approach tooling
  
• Rational Team Concert (RTC), what it is and why you'd be interested in it
• Benefits that customers are seeing with RTC
• How you can download a fully-functional version of RTC for a team of up to 10, with no time limit, free of charge

So, it's basically a cool marketing video for a bunch of free stuff.

On April 25, 2013 I gave a webcast for the Global Rational User Community entitled Disciplined Agile Delivery: Going beyond Scrum .  During the webcast a large number of questions were asked but unfortunately I couldn’t get to all of them.  So I’ve taken the opportunity to write up the answers in this blog posting.

1. Fequently asked questions. Many of the questions are addressed in the DAD FAQ.
2. DAD elevator pitch - I will be starting work in a couple of weeks for a company that has just started its Agile journey this year by implementing Scrum. What would Scott put in an elevator chat as to why they should be moving towards DAD.  The Disciplined Agile Delivery (DAD) decision process framework is a people-first, learning-oriented hybrid agile approach to IT solution delivery. It has a risk-value delivery lifecycle, is goal-driven, is enterprise aware, and is scalable.  There are clearly some interesting aspects to the DAD framework. DAD is a hybrid approach which extends Scrum with proven strategies from Agile Modeling (AM), Extreme Programming (XP), Unified Process (UP), Kanban, Lean Software Development, Outside In Development (OID) and several other methods. DAD is a non-proprietary, freely available framework. DAD extends the construction-focused lifecycle of Scrum to address the full, end-to-end delivery lifecycle from project initiation all the way to delivering the solution to its end users. It also supports lean and continuous delivery versions of the lifecycle: unlike other agile methods, DAD doesn’t prescribe a single lifecycle because it recognizes that one process size does not fit all. DAD includes advice about the technical practices such as those from Extreme Programming (XP) as well as the modeling, documentation, and governance strategies missing from both Scrum and XP. But, instead of the prescriptive approach seen in other agile methods, including Scrum, the DAD framework takes a goals-driven approach. In doing so DAD provides contextual advice regarding viable alternatives and their trade-offs, enabling you to tailor DAD to effectively address the situation in which you find yourself. By describing what works, what doesn’t work, and more importantly why, DAD helps you to increase your chance of adopting strategies that will work for you.  The article Introduction to Disciplined Agile Delivery provides a more detailed description.
3. The book: Could you please repeat the name of the book that Scott is talking about?  The book is Disciplined Agile Delivery: A Practitioner’s Guide to Agile Software Delivery in the Enterprise published by IBM Press, June 2012.  The Disciplined Agile Delivery (DAD) website  and the Disciplined Agile Consortium website are also good DAD resources.
4. The Surveys: Where is the data published about geographic distribution and effectiveness?  All of my survey data, the original questions as asked, and my analysis can be downloaded free of charge from my IT Surveys page.  If you ever see a request from me to fill out a survey, please take a few minutes and do so.  I think you'll agree that my surveys page is in fact a very useful resource, so please contribute when you can.
5. Project success criteria: Where would a goal of defining success criteria fall?  In the DAD book we describe how the success criteria for the project should be initially identified during Inception. Success criteria, like other things, could evolve throughout the project.  You might find the IT Process Success Surveys to be of interest as several of them explored what success criteria projects actually have.  Interestingly, its rarely “on time, on budget, to specification”.
6. Transitioning to DAD: How would you typically phase a DAD implementation, let's say in a project pilot where they haven't been exposed yet to Agile?  Same question, any variance if the development team has already started Scrum?  This is a fairly complex question to answer.  The short story is that you need to invest some time to understand what your strengths and weaknesses are so that you can identify what you need to focus on.  You will then likely need to pilot strategies/techniques which are new to your organization before rolling them out widely.  You may also need to invest in training and coaching/mentoring depending on your needs.  My company, Scott Ambler + Associates, offers these sorts of services and more for organizations interesting in adopting disciplined agile strategies. 
7. Skills: I am aware that ideally developer should also be able to test but in reality theses are usually separate roles. Is it in line with your beliefs\suggestions?  Many organizations that are new to agile still have roles that reflect their existing strategy.  Non-agile approaches often have people in specific roles such as programmer, tester, designer, and so on. In DAD we promote a different set of agile roles that reflect agile thinking.  The implication is that you’re going to have to help individuals transition over to the new way of thinking, something we cover in Chapter 4 of the DAD book.  You might also find Mark Lines' blog, No role in DAD for an Analyst?  to provide some insights into issues surrounding the transition from traditional to agile roles.
8. Teaming: What happens with the Product Owner and the Architecture Owner don’t Agree?  See my blog What Happens When People Don’t Agree 
9. Skillsets: Different people of the team have different skills, experience, and time horizons.  We can't all be generalists, can we?  There are several agile roles in DAD, each of which have different rights, responsibilities, and skillsets.  So we’re not promoting the idea that everyone have the same skillset.  However, we do promote the philosophy that people should strive to be T-skilled generalizing specialists so as to improve their productivity.
10. Teaming: What are your thoughts on team cohesion?  Teams will gel over time.  Being co-located helps.  Having people who are dedicated 100% to the team helps.  Building a team of people who want to be there helps.  Self organization helps. 
11. Tool support: Is there an RMC plug-in for DAD + Is there any software behind DAD... or some software that supports it... such as Jazz (RTC) for Agile/Scrum?  Yes, IBM Rational does in fact have an RMC plug in for DAD.  There is also a template for Microsoft TFS from RDA Corp, Software Development Expert’s Practices Advisor supports DAD comprehensively, and I’m currently working with MethodPark to do so too.    Stay tuned to the Disciplined Agile Delivery website for information about tool support.
12. Governance: Regarding enterprise governance and enterprise IT, how do you start taking a culture that imposes common-process, common-tools, central (often outsourced) IT services, and heavyweight stage/gate across all organizations, and get that evolving toward a (still enterprise aware) lean/agile approach?  This is a hard one.  I’m often called into organizations to help with this very issue.  The challenge is that you need to have a deep understanding of IT governance techniques as well as how to govern agile teams.  Governance is something we discussed in detail in the DAD book as well as on the DAD site, see Adopting Agile Governance Requires Discipline. 
13. Executable specifications: TDD was mentioned, can you comment on Behavior Driven Development (BDD) and any impact on this lifecycle?  BDD is a slight nuance to acceptance test-driven development (ATDD).  BDD/ATDD and TDD are both potential practices that you might choose to follow on a DAD team.  I’ve written a fair bit about agile testing and quality strategies and about TDD in detail.
14. Architecture: When does the first version of architecture gets established and what would require to establish it? You typically start thinking about architecture early in a DAD project during Inception following a practice called architecture envisioning.  I’ve also written a fair bit about agile architecture techniques and the potential misconceptions about agile architecture that you may find interesting.
15. Travel plans: Will you be coming to South Africa at some point?  The good news is that Mark Lines, my co-author, will be in South Africa the week of May 20 in Joburg and the following week in Capetown.  Contact us for details.  I hope to be visiting SA later this year but exact dates haven’t been set yet.  So, please stay tuned on Twitter at @scottwambler for further announcements. 
16. Other travel plans: If you'd like me to speak at a local event, including corporate conferences or training events, please contact me. 

 

 

 

 

The Disciplined Agile Consortium recently launched a certification programme for practitioners of Disciplined Agile Delivery (DAD).   There are three practitioner certifications:

1. Disciplined Agile Yellow Belt. This beginner certification indicates to colleagues and employers that you are eager to learn Disciplined Agile Delivery (DAD) strategies that enable you to increase your skills and abilities as a software professional.
2. Disciplined Agile Green Belt. This intermediate certification indicates that you are experienced at DAD and are on your way to becoming a generalizing specialist. You have the potential to be a “junior coach” under the guidance of a senior coach (someone who is likely a Disciplined Agile Black Belt).
3. Disciplined Agile Black Belt. This expert certification indicates that you are a trusted expert with significant proficiency at DAD. You can coach other people in disciplined agile strategies and advise organizations in the adoption and tailoring of the DAD framework.

Differentiate yourself in the marketplace. Certification in Disciplined Agile Delivery (DAD) means something to clients and employers because it needs to be earned. Certification in DAD tells the marketplace you understand how to deliver an agile solution from end-to-end with experience in enterprise-class development.

Again and again I've seen IT organizations suffering from what I call the "Bureaucracy is Discipline" antipattern. For example, filling out forms and reviewing documents are both bureaucratic activities, neither of which require significant skill nor discipline to accomplish. However, agile practices such as developing potentially shippable software every iteration is easy to say but requires great discipline to accomplish. Respecting the decisions of your stakeholders, particularly those pertaining to requirements prioritization, is easy to talk about but proves to require great discipline in practice (particularly when you don't agree with a decision). It's easy to talk about taking a test-driven approach to development, but in practice it requires significant skill and discipline to actually do.

A "process smell" which indicates that your organization is suffering from this antipattern is a focus on following repeatable processes instead of focusing on repeatable results. An example of repeatable processes is following the same route to work every day regardless of driving conditions. An example of repeatable results is getting to work on time every day, but being willing to change your route as required, bicycling into work instead of driving, taking public transit, and so on. Nobody really cares how you get to work each day (the process), what they really care about is that you got to work on time (the result). Sadly, we've been told for decades now that repeatable processes are critical to our success in IT, yet when you step back and think about that's really a reflection of a bureaucratic approach. On the other hand, a focus on repeatable results is a reflection of a more disciplined approach. Interestingly, the DDJ 2008 Process Framework survey found that given the choice that people would much rather have repeatable results over repeatable processes when it comes to IT.

Mistaking bureaucracy for discipline, or rigour if you prefer that term, is a reflection of the cultural damage that has occurred over the years in IT organizations as the result of traditional philosophies and techniques. Unfortunately, this mistaken belief is a significant inhibitor to software process improvement (SPI) efforts, in particular agile adoption efforts, which must be addressed if you're to be successful. Overcoming this challenge will require a significant cultural shift in some organizations, and many people (particularly the bureaucrats) will find this uncomfortable.

Further reading:

• The Discipline of Agile
• Measured Capability Improvement Framework (MCIF)
• Repeatable Results Over Repeatable Processes

I'd like to leave you with this parting thought: Bureaucracy is bureaucracy and discipline is discipline, please don't confuse the two.[Read More]

 Timo Tenhunen has recently published his master's thesis, Challenges in Scaling Agile Software Development , and has been kind enough to make it available online.  I suspect you'll find it to be an interesting read.

I recently recorded an audio podcast about Collaborative Development and Operations (DevOps) and how it relates to Disciplined Agile Delivery (DAD).  The podcast is 17 minutes long and covers a range of topics including:

• What are the challenges typically faced by Disciplined Agile Development teams as they begin to transition their working solutions into production?
• How best could this gap between Development and Operations be closed so that they worked collaboratively rather than in silos?
  
• If this DevOps gap was decreased or even closed, what would the measurable value be to Agile projects and their business stakeholders?

On a related note, IBM's Collaborative Development and Operations landing page has some great resources if you're interested in how to implement.

For some reason, it seems as if everyone's grandfather at one point in time recommended to use the right tools for the job.  That's practical wisdom from my point of view, one that is certainly an issue for agile development. 


One of the primary messages, I hope, of the Agile Scaling Model (ASM) is that context counts.  Although the focus of the ASM is on describing a contextual framework for tailoring your process to meet the needs of the situation that you find yourself in, it's also applicable to your tooling selection.  For example, the tool choices of a co-located team will be much different than that of a geographically distributed team.  A co-located team will likely use a whiteboard or paper for their agile modeling efforts, whereas distributed team members may need to capture their diagrams using a more sophisticated tool such as Rational Requirements Composer (RRC) so that their work can be shared electronically.  Having said that, RRC would be overkill for a co-located team (unless they had regulatory compliance issues).  Different teams, different situations, therefore different tooling choices.


One of the concerns that I run into from customers is that some of our legacy products don't support agile very well.  Once again, it's a matter of context because many of our legacy products reflect the realities faced by more traditional teams.  The challenge occurs when you try to take a legacy product which is well suited for traditional development, such as Rational ClearCase, and try to apply it on agile projects.  Although ClearCase makes sense in certain scaling situations, particularly very large teams that are geographically distributed, you'd be better advised to use something like Rational Team Concert (RTC) for configuration management on most agile teams (note that RTC does far more than just SCM).


So, if you're taking an agile approach you should consider Rational tools such as RTC, RRC, Rational BuildForge, Rational AppScan, and others which support agile development.  Granted, some you would only use at scale -- for example Buildforge is a good option in really complex environments, but if you don't face that complexity then you'll likely find that RTC's build engine is sufficient.  Similarly, if you're taking a traditional approach to development then you'll likely consider products such as ClearCase, Appscan, RTC, and Rational Software Architect (RSA) instead.  Different situations, different tooling choices. 


What's even more confusing is that some products support a range of process paradigms.  For example, RTC supports agile, lean, iterative, and traditional approaches to development.  The same can be said of Appscan and several other products.  Notice how I listed RTC and Appscan for both agile and traditional development above. 


So, if anyone tells you that Rational tools don't support agile development don't believe them.  Ask them which tools that they're talking about, and ask them if they're aware of the Rational products that do support agile development.  Context counts.

Although it might not be obvious, and important success factor in adopting agile techniques is to be able to determine whether a team is agile or not. The challenge that many organizations face is that many teams will claim to be agile, yet management, who often has little or no experience with agile approaches, cannot tell which claims are true and which are over zealous (I'm being polite). The following are the criteria that I suggest you look for in a disciplined agile team:1. Produce working software on a regular basis. This is one of the 12 principles behind the Agile Manifesto, and in my experience is a critical differentiator between the teams that are agile and those that are merely claiming it. Ideally the team should produce potentially shippable software each iteration. That doesn't mean that they'll deploy the system into production, or the marketplace, each iteration but they could if required to do so. Typically the team will deploy into a pre-production testing environment or a demo enviroment at the end of each iteration (or more often for that matter).2. Do continuous regression testing, and better yet take a Test-Driven Development (TDD) approach. Agile developers test their work to the best of their ability, minimally doing developer regression testing via a continuous integration (CI) strategy and better yet do developer-level TDD. This approach enables development teams to find defects early, thereby reducing the average cost of addressing the defects, it also helps them to deliver higher quality code and to move forward safely when adding or changing functionality.3. Work closely with their stakeholders, ideally on a daily basis. A common practice of agile teams is to have an on-site customer or product owner who prioritizes requirements and provides information on a timely manner to the team. Disciplined agile teams take it one step further and follow the practice active stakeholder participation where the stakeholders get actively involved with modeling and sometimes even development.4. Are self-organizing within a governance framework. Agile teams are self-organizing, which means that the people doing the work determines how the work will be done, they're not told by a manager who may not even be directly involved with the work how it will be done. In other words the team does its own planning, including scheduling and estimation. Disciplined agile teams are self governing within an effective governance framework.5. Regularly reflect on how they work together and then act to improve on their findings. Most agile teams hold a short meeting at the end of each iteration to reflect upon how well things are working and how they could potentially improve the way that they are working together. Sometimes this is done in a more formalized manner in the form of a retrospective, but often it's done informally. The team then acts on one or more of their suggested improvements the next iteration. Disciplined agile teams take this one step further and measure their software process improvement (SPI) progress over time: the act of taking these measures, perhaps via a product such as Rational Self Check, helps to keep the team on track in their SPI efforts.

I have yet to discover an ad-hoc development team which met all five criteria, and most of them rarely meet two or three.

Further reading:

• Lean Development Governance
• Examining the Agile Manifesto
• Norm Kerth's Retrospectives Home Page
• Measured Capability Improvement Framework (MCIF)

[Read More]

When adopting agile software development techniques across a large number of teams within your organization it is important to provide a definition for what agile software development is, in addition to criteria for what it means to be agile. Many people will point to the four values of the Agile Manifesto and claim that's a good definition. Well... it might be a good definition for the visionaries and early adopters among us, but for people on the right-hand side of the technology adoption chasm (the early majority, late majority and the laggards) this isn't enough. Don't get me wrong, I'm a firm believer in the agile values but I like to cast them as philosophies instead of as a definition.

At IBM Software Group, the definition of disciplined agile software delivery which we have been sharing with our customers is:Disciplined agile software delivery is an evolutionary (iterative and incremental) approach to delivery which regularly produces high quality software in a cost effective and timely manner. It is performed in a highly collaborative and self-organizing manner, with active stakeholder participation to ensure that the team understand and addresses the changing needs of its stakeholders. Disciplined agile delivery teams provide repeatable results by adopting just the right amount of ceremony for the situation which they face.

I think that this is a pretty good definition, although I have no doubt that we'll evolve it over time.

I also suspect that the agile community will never settle on a common definition for what agile is and more than likely are smart enough not to even try. ;-)
Further reading:

• Agile Criteria
• Agile Manifesto
• Rational's Agile Software Development Home Page (for webcasts, whitepapers, ...)

[Read More]

At IBM Rational we define disciplined agile delivery as:

Disciplined agile delivery is an evolutionary (iterative and incremental) approach which regularly produces high quality solutions in a cost effective and timely manner via a risk and value driven life cycle. It is performed in a highly collaborative, disciplined, and self-organizing manner within an appropriate governance framework, with active stakeholder participation to ensure that the team understands and addresses the changing needs of its stakeholders to maximize business value provided. Disciplined agile delivery teams provide repeatable results by adopting just the right amount of ceremony for the situation which they face.

Let’s explore the key points in this definition:

1. Full delivery life cycle.  Disciplined agile delivery processes have life cycles which are serial in the large and iterative in the small.  Minimally they have a release rhythm which recognizes the need for start up/inception activities, construction activities, and deployment/transition activities.  Better yet, they include explicit phases as well. It is very important to note that these are not the traditional waterfall phases – requirements, analysis, design, and so on – but instead different “seasons” of a project. The point is that we need to look beyond agile software development and consider the full complexities of solution delivery.  Adopting a full delivery life cycle, not just a construction life cycle, is arguably the “zeroth” agile scaling factor. 
2. Evolutionary.  Agile strategies are both iterative and incremental in nature.  Iterative means that you are working in a non-serial manner, on any given day you may do some requirements analysis, some testing, some programming, some design, some more testing, and so on.  Incremental means that you add new functionality and working code to the most recent build, until such time as the stakeholder determines there is enough value to release the product.
3. Regularly produces high quality solutions. Agilists are said to be quality focused.  They prefer to test often and early, and the more disciplined ones even take a test-first approach where they will write a single test and the just enough production code to fulfill that test (then they iterate). Many agile developers have adopted the practice of refactoring, which is a technique where you make simple changes to your code or schema which improves its quality without changing its semantics.  Adoption of these sorts of quality techniques seems to work – it appears that agile teams are more likely to deliver high quality systems than traditional teams (according to the DDJ 2008 Project Success survey). Within IBM we take it one step further and focus on consumability, which encompasses quality and other features such as ease of deployment and system performance.  Furthermore, although some agile methods promote the concept of producing “potentially shippable software” on a regular basis, disciplined agile delivery teams produce solutions: a portion of which may be software, a portion of which may be hardware, and a portion of which will be the manner in which the system is used.
4. Cost effective and timely manner.  Agile teams prefer to implement functionality in priority order [http://www.agilemodeling.com/essays/prioritizedRequirements.htm], with the priority being defined by their stakeholders (or a representative thereof).  Working in priority order enables agile teams to maximize the return on investment (ROI) because they are working on the high-value functionality as defined by their stakeholders, thereby increasing cost effectiveness. Agile teams also prefer to produce potentially shippable solutions each iteration (an iteration is a time-box, typically 2-4 weeks in length), enabling their stakeholders to determine when they wish to have a release delivered to them and thereby improving timeliness.  Short iterations reduce the feedback cycle, improving the chance that agile teams will discover problems early (they “fail fast”) and thereby enable them to address the problems when they’re still reasonably inexpensive to do so.  The DDJ 2008 Project Success survey found that agile teams are in fact more likely to deliver good ROI than traditional teams and more likely to deliver in a timely manner.
5. Value driven life cycle. One result of building a potentially shippable solution every iteration is that agile teams produce concrete value in a consistent and visible manner throughout the life cycle. 
6. Risk and value driven life cycle.  Core agile processes are very clear about the need to produce visible value in the form of working software on a regular basis throughout the life cycle.  Disciplined agile delivery processes take it one step further and actively mitigate risk early in the life cycle – during project start up you should come to stakeholder concurrence regarding the project’s scope, thereby reducing significant business risk, and prove the architecture by building a working skeleton of your system, thereby significantly reducing technical risk.  They also help with transition to agile, allowing traditional funding models to use these milestones before moving to the finer grained iteration based funding that agile allows.
7. Highly collaborative.  People build systems, and the primary determinant of success on a development project is the individuals and the way that they work together.  Agile teams strive to work closely together and effectively as possible.  This is a characteristic that applies to both engineers on the team, as well as their leadership.
8. Disciplined.  Agile software development requires greater discipline on the part of practitioners that what is typically required by traditional approaches. 
9. Self organizing. This means that the people who do the work also plan and estimate the work.
10. Self-organization within an appropriate governance framework. Self-organization leads to more realistic plans and estimates which are more acceptable to the people implementing them.  At the same time these self-organizing teams must work within an appropriate governance framework which reflects the needs of their overall organizational environment.  An “appropriate governance framework” explicitly enables disciplined agile delivery teams to effectively leverage a common infrastructure, to follow organizational conventions, and to work towards organizational goals.  The point is that project teams, regardless of the delivery paradigm they are following, need to work within the governance framework of their organization.  More importantly, effective governance programs should make it desirable to do so.  Our experience is that traditional, command-and-control approaches to governance where senior management explicitly tells teams what to do and how to do it don’t work very well with agile delivery teams.  We’ve also found that lean development governance, an approach which is based on collaboration and enablement, is far more effective in practice.  Good governance increases the chance that agile delivery teams will build systems which fit into your overall organizational environment, instead of yet another stand-alone system which increases your overall maintenance burden and data quality problems.
11. Active stakeholder participation.  Agile teams work closely with their stakeholders, who include end users, managers of end users, the people paying for the project, enterprise architects, support staff, operations stuff, and many more.  Within IBM we distinguish between four categories of stakeholder: principles/sponsors, partners (business partners and others), end users, and insiders These stakeholders, or their representatives (product owners in Scrum, or on-site customers in Extreme Programming, or a resident stakeholder in scaling situations), are expected to provide information and make decisions in a timely manner.
12. Changing needs of stakeholders.  As a project progresses your stakeholders will gain a better understanding of what they want, particularly if you’re showing them working software on a regular basis, and will change their “requirements” as a result.  Changes in the business environment, or changes in organization priority, will also motivate changes to the requirements.  There is a clear need for agile requirements change management [http://www.agilemodeling.com/essays/changeManagement.htm] on modern IT projects.
13. Repeatable results.  Stakeholders are rarely interested in how you delivered a solution but instead in what you delivered.  In particular, they are often interested in having a solution which meets their actual needs, in spending their money wisely, in a high-quality solution, and in something which is delivered in a timely manner.  In other words, they’re interested in repeatable results, not repeatable processes.
14. Right amount of ceremony for the situation.  Agile approaches minimize ceremony in favor of delivering concrete value in the form of working software, but that doesn’t mean they do away with ceremony completely. Agile teams will still hold reviews, when it makes sense to do so.  DDJ’s 2008 Modeling and Documentation Survey found that agile teams will still produce deliverable documentation, such as operations manuals and user manuals, and furthermore are just as likely to do so as traditional teams.  The DDJ September 2009 State of the IT Union survey found that the quality of the documentation delivered by agile teams was just as good as that delivered by traditional teams, although iterative teams (e.g. RUP teams) did better than both agile and traditional.

 Suggested reading:

• The Agile System Delivery Life Cycle (SDLC)
• The Agile Scaling Model (ASM): Adapting Agile Methods for Complex Environments (an IBM Rational white paper)
• The Disciplined Agile Delivery (DAD) home page
  

This blog posting has been replaced by the more detailed article: Full Agile Delivery Lifecycles.  

Thank you for your patience.

 

 

The basic idea behind DevOps is that your development strategy and operations strategy should reflect one another, that you should strive to optimize the whole IT process.  This implies that development teams should work closely with your operations staff to deliver new releases smoothly into production and that your operations staff should work closely with development teams to streamline critical production issues.

DevOps has its source in agile software development, and it is an explicit aspect of the Disciplined Agile Delivery (DAD) process framework.  As a result there is a collection of agile development strategies which enable effective DevOps throughout the agile delivery lifecycle.  These strategies include:

1. Initial requirements envisioning.  Disciplined agile teams invest time at the beginning of the project to identify the high-level scope in a light-weight, collaborative manner.  This includes common operations requirements such as the need to backup and restore data sources, to instrument the solution so that it can be monitored in real time by operations staff, or to architect the solution in a modular manner to enable easier deployment. 
2. Initial architecture envisioning. Disciplined agile teams will also identify a viable architectural strategy which reflects the requirements of their stakeholders and your organization’s overall architectural strategy (hence the need to work closely with your enterprise architects and operations staff).  One goal is to ensure that the team is building (or buying) a solution which will work well with the existing operational infrastructure and to begin negotiating any infrastructural changes (such as deploying new technologies) early in the project.   Another goal is to ensure that operations-oriented requirements are addressed by the architecture from the very start. 
3. Initial release planning.  As part of release planning the disciplined agile team works closely with their operations group to identify potential release windows to aim for, any release blackout periods to avoid, and the need for operations-oriented milestone reviews later in the lifecycle (if appropriate).
4. Active stakeholder participation.  Disciplined agile teams work closely with their stakeholders, including both operations and support staff, all the way through the lifecycle to ensure that their evolving needs are understood.   
5. Continuous integration (CI).  This is a common technical agile practice where the solution is built/compiled, regression tested, and maybe even run through code analysis tools.  CI promotes greater quality which in turn enables easier releases into production.  
6. Parallel independent testing. For enterprise-class development or at scale, particularly when the domain or technology is very complex or in regulatory environments, disciplined agile team will find they need to support their whole team testing efforts with an independent test team running in parallel to the development team.  These testing issues often include validation of non-functional requirements – such as security, performance, and availability concerns – and around production system integration.  All of these issues are of clear importance for operations departments.   
7. Continuous deployment.  With this practice you automate the promotion of your working solution between environments.  By automating as much of the deployment effort as possible, and by running it often, the development team increases the chance of a successful deployment and thereby reduces the risk to the operations environment. Note that deployment into production is generally not automatic, as this is an important decision to be made by your operations/release manager(s). 
8. Continuous documentation.  With this practice supporting documentation, including operations and support documentation, is evolved throughout the lifecycle in concert with the development of new functionality. 
9. Production release planning.  This is the subset of your release planning efforts which focuses on the activities required to deploy into production. 
10. Production readiness reviews.  There should be at least one review, performed by the person(s) responsible for your operations environment, before the solution is deployed into production.  The more critical the system, the more product readiness reviews may be required. 
11. End-of-lifecycle testing.  Minimally you will need to run your full automated regression test suite against your baselined code once construction ends.  There may also be manual acceptance reviews or testing to be performed, and any appropriate fixing and retesting required to ensure that the solution is truly ready for production. 

For more detail about this topic, I think that you will find the article I wrote for the December 2011 issue of Cutter IT Journal entitled “Disciplined Agile Delivery and Collaborative DevOps” to be of value.

My new white paper, Disciplined Agile Delivery: An Introduction, is now available free of charge from IBM.com. The paper overviews the Disciplined Agile Delivery (DAD) process framework, a hybrid comprised of strategies from Scrum, XP, Agile Modeling, and other agile methods which is people first, learning oriented, and enterprise aware. DAD is the basis from which you can scale agile.

Topics covered:

• Context counts - The Agile Scaling Model
• People first - People, and the way they interact with each other, are the primary determinant of success for a solution delivery project.
• Learning-oriented - The DAD process framework promotes the ideas that team members should collaborate closely and learn from each other, that the team should invest effort to learn from their experiences and evolve their approach, and that individuals should do so as well.
• Hybrid - DAD adopts and tailors strategies from Scrum, XP, Agile Modeling, UP, Kanban, and many others. It addresses many of the issues Mark Kennaley discusses in SDLC 3.0.
  
• IT solution focused - DAD teams produce potentially consumable solutions every construction iteration.  This extends Scrum's "potentially shippable" strategy to explicitly address usability/consumability plus the fact that we're really delivering full solutions not just software.
• Goal-driven delivery life cycle - The DAD lifecycle is focused on delivery, not just construction.  Furthermore it is goals-driven, the DAD process framework suggests various strategies to fulfill those goals but does not prescribe specific practices.
  
• Risk and value driven - The DAD lifecycle is risk and value driven.  It extends Scrum's value-driven lifecycle which produces potentially shippable software each sprint/iteration so that it explicitly includes light-weight milesstones such as ensuring stakeholder consensus as to the scope of the project early in the lifecycle, proving the architecture with working code early in the lifecycle, ensuring sufficient functionality exists before transition, and ensuring production readiness before actual release of the solution.
  
• Enterprise aware - The DAD process framework promotes the ideas that DAD teams should work closely with their enterprise architecture groups to ensure they leverage and evolve the existing infrastructure, adopt and follow corporate guidelines, and work to the overall organizational vision.  DAD teams are self organizing with appropriate governance.

On Twitter one of the people that I follow recently tweeted in jest "I am waste".  He had been tweeting about the problem that people will often pay consultants for their advice yet will not follow that advice once it's been provided (yet will often still keep paying for more advice).  Perhaps in this situation his efforts were a waste, or perhaps the real issue was that the customer had a long learning process and hadn't yet come to the point where they were willing to act on the advice, we may never know.  But I have to think that there are other situations where this person isn't a waste, regardless of his claims.

This got me thinking that something is a waste in one context yet in another context may be quite valuable.  Or, as the old saying goes, one man's trash is another man's treasure.

For example, consider the following simple value stream: 

[Activity A: 10 min] [Wait time: 20 min] [Activity B: 10 min]

Without considering the context, the wait time of 20 min represents 50% waste in the overall process that we should try to eliminate.

What if that wait time provides people with a much needed rest?  With time to contemplate?  With time to destress?  Eliminating that wait time, or even reducing it, could result a degradation of performance.  In this case, one person's waste (the wait time) is another person's treasure (rest time).  The implication is that we need to work closely with the people intimately involved in a process if we're to help them to improve it.

Context counts.

I've been getting a lot of questions lately about applying the acceleration metric in practice. So, here's some answers to frequently asked questions:
1. How do I monetize acceleration? This is fairly straightforward to do. For example, assume your acceleration is 0.007 (0.7%), there are five people on the team, your annual burdened cost per person is $150,000, and you have two week iterations. All these numbers are made up, but you know how to calculate acceleration now and IT management had darn well better know the average burdened cost (salary plus overhead) of their staff. So, per iteration the average burdened cost per person must be $150,000/26 = $5,770. Productivity improvement per iteration for this team must be $5,770 * 5 * .007 = $202. If the acceleration stayed constant at 0.7% the overall productivity improvement for the year would be (1.007)^26 (assuming the team works all 52 weeks of the year) which would be 1.198 or 19.8%. This would be a savings of $148,500 (pretty much the equivalent of one new person). Of course a 20% productivity increase over an entire year is a really aggressive improvement, regardless of some of the claims made by the agile snake oil salesman out there, although at 10-15% increase is a reasonable expectation. What I'd really want to do is calculate the acceleration for the year by comparing the velocity from the beginning of the year to the end of the year (in Western cultures I'd want to avoid comparing iterations near to the holidays). So, if the team velocity the first week of February 2008 was 20 points, now the same team's velocity the first week of February 2009 was 23 points, that's an acceleration of (23-20)/20 = 15% over a one year period, for a savings of $112,500.
2. Is acceleration really unitless? For the sake of comparison it is. The "units" are % change in points per iteration, or % change in points per time period depending on the way that you want to look at it. Because it's a percentage I can easily monetize it, as you see above, and use it as a basis of comparison.
3. How do I convince teams to share their data? This can be difficult. Because acceleration is easy to calculate for agile teams, and because it's easy to use to compare teams (my team has .7% acceleration whereas other teams down the hall from mine have accelerations of .3% and -.2% of teams), people are concerned that this metric will be used against them. OK, to be fair, my team might be OK with this. ;-) Seriously though, this is a valid fear that will only be addressed by an effective governance program based on enablement, collaboration, and trust instead of the traditional command-and-control approach. Management's track record regarding how they've used measurements in the past, and how they've governed in general, have a great effect on people's willingness to trust them with new metrics such as acceleration. The implication is that you need to build up trust, something that could take years if it's possible at all.
4. Why does this work for agile teams? Agile teams are self organizing, and an implication of that is that they will be held accountable for their estimates. Because of this accountability, and because velocity is a vital input into their planning and estimation efforts, agile teams are motivated to calculate their velocity accurately and to track it over time. Because they're eager to get their velocity right, and because acceleration is based on velocity, there's an exceptionally good chance that it's accurate.
5. What about function points or similar productivity measures? Function points can be calculated for projects being developed via an agile approach, or other approaches for that matter, but it's a very expensive endeavor compared to calculating acceleration (which is essentially free) and likely will be seen as a bureaucratic overhead by the development team. My rule of thumb is that if you're not being explicitly paid to count function points (for example the US DoD will often pay contracting companies to create estimates based on function point counts) then I wouldn't bother with them.
6. What about calculating acceleration for iterative project teams? Iterative project teams, perhaps following Rational Unified Process (RUP), can choose to calculate and track their velocity and thereby their acceleration. The key is to allow the team to be self organizing and accountable for their estimates, which in turn motivates them to get their velocity right just like agile teams (RUP can be as agile as you want to make it, don't let anyone tell you differently).
7. What about calculating acceleration for traditional project teams? In theory this should work, in practice it is incredibly unlikely. Traditional teams don't work in iterations where working software is produced on a regular basis, they're typically not self organizing, and therefore there really isn't any motivate to calculate velocity (even if they do, there is little motivation to get it right). Without knowing the velocity you can't calculate acceleration. If you can't trust the velocity estimate, and I certainly wouldn't trust a traditional team's velocity estimate, then you can't trust your acceleration calculation. So, my fall back position to calculate productivity improvement would be to do something like function point counting (which is expensive and difficult to compare between teams due to different fudge factors used by different FP counters) and then looking at change in FPs delivered over time.
8. How can I apply this across a department? It is fairly straightforward to roll up the acceleration of project teams into an overall acceleration measure for a portfolio of teams simply by taking a weighted average based on team size. However, this is only applicable to teams that are in a position to report an accurate acceleration (the agile and iterative teams) and of course are willing to do so.
9. What does a negative acceleration tell me? If the acceleration is negative then productivity on the team is going down, likely an indicator of quality and/or team work problems. However, you don't want to manage by the numbers so you should talk to the team to see what's actually going on.
10. What does a zero acceleration tell me? This is an indication that the team's productivity is not increasing, and that perhaps they should consider doing retrospectives at the end of each iteration and then acting on the results from those retrospectives. Better yet they can "dial up" their process improvement efforts by adopting something along the lines of IBM Rational Self Check.
Further reading:

• Acceleration: An Agile Productivity Measure
• Lean Development Governance
• Agile and Rational Unified Process (RUP)
• Introducing IBM Rational Self Check for Software Teams
  
  

[Read More]

As part of the Agile/Jazz event In Milwaukee on the 29th I'll be giving a talk about the realities of agile software development. The URL for the event is http://www.iconatg.com/lp/events/jazz/jazz_milwaukee.html and I hope to see you there.

- Scott[Read More]