Agility@Scale: Strategies for Scaling Agile Software Development

This article has been replaced by an official "Disciplined Agile Manifesto".

 

The text of the original article remains below.

---------------------------------------------------------------------------------------

I've recently been working with Mark Lines of UPMentors and we've had some interesting discussions around evolving the Agile Manifesto which I thought I would share here to obtain feedback.  Note that this is not any sort of official position of IBM, nothing in my blog is by the way (unless explicitly stated so), nor is it some sort of devious plot to take over the agile world (although if we did have some sort of devious plot, we'd make the exact same claim).  What we hope to accomplish is to put some ideas out there in the hopes of getting an interesting conversation going.

Over the past decade we’ve applied the ideas captured in the Agile Manifesto and have learned from our experiences doing so.  What we’ve learned has motivated us to suggest changes to the manifesto to reflect the enterprise situations which we have applied agile and lean strategies in.  We believe that the changes we’re suggesting are straightforward:

1. Where the original manifesto focused on software development, a term which too many people have understood to mean only software development, we suggest that it should focus on solution delivery.
2. Where the original focused on customers, a word that for too many people appears to imply only the end users, we suggest that it focus on the full range of stakeholders instead.
3. Where the original manifesto focused on development teams, we suggest that the overall IT ecosystem and its improvement be taken into consideration.
4. Where the original manifesto focused on the understanding of, and observations about, software development at the time there has been some very interesting work done within the lean community since then (and to be fair there was very interesting work done within that community long before the Agile Manifesto was written).  We believe that the Agile Manifesto can benefit from lean principles.

Our suggested rewording of the Agile Manifesto follows, with our suggested changes in italics.

Updating the Values of the Agile Manifesto

We are uncovering better ways of developing software by doing it and helping others do it.  Through this work we have come to value:

1. Individuals and interactions over processes and tools
2. Working solutions over comprehensive documentation
3. Stakeholder collaboration over contract negotiation
4. Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Updating the Principles behind the Agile Manifesto

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable solutions.
2. Welcome changing requirements, even late in the solution delivery lifecycle. Agile processes harness change for the stakeholder’s competitive advantage.
3. Deliver working solutions frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
4. Stakeholders and developers must work together daily throughout the project.
5. Build projects around motivated individuals.  Give them the environment and support they need, and trust them to get the job done.
6. The most efficient and effective method of conveying information to and within a delivery team is face-to-face conversation.
7. Quantified business value is the primary measure of progress.
8. Agile processes promote sustainable development.  The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
9. Continuous attention to technical excellence and good design enhances agility.
10. Simplicity--the art of maximizing the amount of work not done--is essential.
11. The best architectures, requirements, and designs emerge from self-organizing teams.
12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
13. Leverage and evolve the assets within your organizational ecosystem, and collaborate with the people responsible for those assets to do so.
14. Visualize workflow to help achieve a smooth flow of delivery while keeping work in progress to a minimum.
15. The organizational ecosystem must evolve to reflect and enhance the efforts of agile teams, yet be sufficiently flexible to still support non-agile or hybrid teams. 

We’re agile – things evolve, including manifestos.  Looking forward to your feedback (add a comment).

 

Updates Since this Was First Published:

• February 2011: At the 10th Anniversary of the Agile Manifesto get together several people, including myself, brought up the idea of updating the Agile Manifesto.  Unfortunately the decision by the original writers of the manifesto was to adopt a very strict, non-agile change management approach where any changes to the manifesto would need the approval of all 17 of them.  So, pretty much zero chance of there ever being any changes (hence the lack of changes so far). 
• June 2012: This extension to the manifesto was retitled "A Disciplined Agile Manifesto" and published as in the book Disciplined Agile Delivery written by myself and Mark Lines.
• November 2012: Mark and I formed Scott Ambler+Associates.
• February 2013: The Disciplined Agile Consortium (DAC) is formed.  The mission of DAC is to evolve and support the Disciplined Agile Delivery (DAD) framework.
• April 2014: Official Disciplined Agile Manifesto posted on the Disciplined Agile Delivery (DAD) site.

Further reading:

• The Declaration of Interdependence (some other thoughts about, arguably, evolving the manifesto)
• Examining the Agile Manifesto
• Disciplined Agile Delivery website

 

 

 

Contrary to popular belief, agile development teams do in fact model and yes, they even do some up front requirements and architecture modeling. Two of the best practices of Agile Modeling are Requirements Envisioning and Architecture Envisioning where you spend a bit of time at the beginning of the project doing enough initial modeling to get you going in the right direction. The strategy is to take advantage of modeling, which is to communicate and think things through without taking on the risks associated with detailed specifications written early in the lifecycle. In this blog posting I will focus on requirements envisioning, in a future posting I'll cover architecture envisioning.

The goal of initial requirements envisioning is to identify the scope of your effort. You need to do just enough modeling early in the project to come to stakeholder concurrence and answer questions such as what you're going to build, roughly how long it's going to take (give a range), and roughly how much it's likely to cost (once again, give a range). If you can get the right people together in the room, which can sometimes be a logistics challenge but not one that you couldn't choose to overcome, there are very few systems (I suspect less than 5%) that you couldn't initially scope out in a few days or a week. I also suspect that most of the remaining systems could be scoped out with less than 2 weeks of modeling, and if not then I'd take that as an indication that you're taking on too large of a project. I'm not saying that you'll be able to create big detailed specifications during this period, and quite frankly given the problems associated with "Big Requirements Up Front (BRUF)" you really don't want to, but I am saying that you could gain a pretty good understanding of what you need to do. The details, which you'll eventually need, can be elicited throughout the lifecycle when you actually need the information. A common saying in the agile community is that requirements analysis is so important for us that we do it every single day, not just during an initial phase. I'll discuss just in time (JIT) approaches to requirements modeling in a future posting.

To envision the requirements for a business application, you might want to consider creating the following models:

• High-level use cases (or user stories). The most detail that I would capture right now would be point form notes for some of the more complex use cases, but the majority just might have a name. The details are best captured on a just-in-time (JIT) basis during construction.
• User interface flow diagram. This provides an overview of screens and reports and how they're inter-related. You just need the major screens and reports for now.
• User interface sketches. You'll likely want to sketch out a few of the critical screens and reports to give your stakeholders a good gut feeling that you understand what they need. Sketches, not detailed screen specifications, are what's needed at this point in time.
• Domain model. A high-level domain model, perhaps using UML or a data modeling notation, which shows major business entities and the relationships between them, can also be incredibly valuable. Listing responsibilities, both data attributes and behaviors, can be left until later iterations.
• Process diagrams. A high-level process diagram, plus a few diagrams overviewing some of the critical processes, are likely needed to understand the business flow.
• Use-case diagram. Instead of a high-level process diagram you might want to do a high-level use case diagram instead. This is a matter of preference, I likely wouldn't do both.
• Glossary definitions. You might want to start identify key business terms now, although I wouldn't put much effort into settling on exact definitions. I've seen too many teams run aground on "analysis paralysis" because they try to define exact terminology before moving forward. Don't fall into this trap.

For small teams simple tools such as whiteboards and paper are usually sufficient for requirements envisioning. But what happens at scale? What if you're working on a large agile team, say of 50 people, 200 people (IBM has delivered software into the marketplace with agile teams of this size), or even 500 people (IBM currently has teams of this size applying agile techniques)? What if your team is distributed? Even if you have people working on different floors of the same building, let alone working from home or working in different cities or countries, then you're distributed (see my postings about distributed agile development). Suddenly whiteboards and paper-based tools (index cards, sticky notes, ...) aren't sufficient. You're still likely to use these sorts of tools in modeling sessions with stakeholders, but because of one or more scaling factors you need to capture your requirements models electronically.

In January Theresa Kratschmer and I gave a webcast entitled Agile Requirements: Collaborative, Contextual, and Correct which overviewed agile approaches to requirements elicitation and management, including requirements envisioning. We also showed how Rational Requirements Composer (RRC) can be used to electronically capture critical requirements information, enabling you to address the needs of large and/or distributed agile teams, while still remaining lightweight and flexible. I suspect that you'll find the webcast to be very illuminating and RRC something that you want to take a look at (the link leads to a trial version). Of course RRC can be used in other situations as well, but that's not what I'm focused on right now.

Teams which find themselves in regulatory environments will likely need to do more than just use RRC, as might very large teams. Regulatory compliance often requires more complex requirements documentation, which in turn requires more sophisticated tools such as DOORS or Requisite Pro, and I would consider using those tools in the types of situations that warrant it. One of the things that people often struggle to understand about agile approaches is that you need to tailor your strategy to reflect the situation at handle. One process size does not fit all, so you will end up using different tools and creating different artifacts to different extents in different situations. Repeatable results, not repeatable processes, is the rule of the day.


Further reading:

• Agile Requirements: Collaborative, Contextual, and Correct Webcast
• Rational Requirements Composer
• Rational's Agile Software Development Home Page (for webcasts, whitepapers, ...)
• Agile Requirements Modeling
• Requirements Envisioning: An Agile Best Practice
• Examining the Big Requirements Up Front (BRUF) Approach
• DDJ's 2008 Modeling and Documentation Survey, which found that agile teams are more likely to model than traditional teams

[Read More]

Again and again I've seen IT organizations suffering from what I call the "Bureaucracy is Discipline" antipattern. For example, filling out forms and reviewing documents are both bureaucratic activities, neither of which require significant skill nor discipline to accomplish. However, agile practices such as developing potentially shippable software every iteration is easy to say but requires great discipline to accomplish. Respecting the decisions of your stakeholders, particularly those pertaining to requirements prioritization, is easy to talk about but proves to require great discipline in practice (particularly when you don't agree with a decision). It's easy to talk about taking a test-driven approach to development, but in practice it requires significant skill and discipline to actually do.

A "process smell" which indicates that your organization is suffering from this antipattern is a focus on following repeatable processes instead of focusing on repeatable results. An example of repeatable processes is following the same route to work every day regardless of driving conditions. An example of repeatable results is getting to work on time every day, but being willing to change your route as required, bicycling into work instead of driving, taking public transit, and so on. Nobody really cares how you get to work each day (the process), what they really care about is that you got to work on time (the result). Sadly, we've been told for decades now that repeatable processes are critical to our success in IT, yet when you step back and think about that's really a reflection of a bureaucratic approach. On the other hand, a focus on repeatable results is a reflection of a more disciplined approach. Interestingly, the DDJ 2008 Process Framework survey found that given the choice that people would much rather have repeatable results over repeatable processes when it comes to IT.

Mistaking bureaucracy for discipline, or rigour if you prefer that term, is a reflection of the cultural damage that has occurred over the years in IT organizations as the result of traditional philosophies and techniques. Unfortunately, this mistaken belief is a significant inhibitor to software process improvement (SPI) efforts, in particular agile adoption efforts, which must be addressed if you're to be successful. Overcoming this challenge will require a significant cultural shift in some organizations, and many people (particularly the bureaucrats) will find this uncomfortable.

Further reading:

• The Discipline of Agile
• Measured Capability Improvement Framework (MCIF)
• Repeatable Results Over Repeatable Processes

I'd like to leave you with this parting thought: Bureaucracy is bureaucracy and discipline is discipline, please don't confuse the two.[Read More]

The explicit phases of the Unified Process -- Inception, Elaboration, Construction, and Transition -- and their milestones are important strategies for scaling agile software development to meet the real-world needs of modern organizations. Yes, I realize that this is heresy for hard-core agilists who can expound upon the evils of serial development, yet these very same people also take a phased approach to development although are loathe to admit it. The issue is that the UP phases are like seasons of a project: although you'll do the same types of activities all throughout a project, the extent to which you do them and the way in which you do them change depending on your goals. For example, at the beginning of a development project if you want to be effective you need to do basic things like identify the scope of the project, identify a viable architecture strategy, start putting together your team, and obtain support for the project. Towards the end of a project your focus is on the activities surrounding the deployment of your system into production, including end-of-lifecycle testing efforts, training, cleaning up of documentation, piloting the system with a subset of users, and so on. In between you focus on building the system, including analysis, design, testing, and coding of it. Your project clearly progresses through different phases, or call them seasons if the term phase doesn't suit you, whether your team is agile or not.

The UP defines four phases, each of which address a different kind of risk:1. Inception. This phase focuses on addressing business risk by having you drive to scope concurrence amongst your stakeholders. Most projects have a wide range of stakeholdres, and if they don't agree to the scope of the project and recognize that others have conflicting or higher priority needs you project risks getting mired in political infighting. In the Eclipse Way this is called the "Warm Up" iteration and in other agile processes "Iteration 0".2. Elaboration. The goal of this phase is to address technical risk by proving the architecture through code. You do this by building and end-to-end skeleton of your system which implements the highest-risk requirements. Some people will say that this approach isn't agile, that your stakeholders should by the only ones to prioritize requirements. Yes, I agree with that, but I also recognize that there are a wide range of stakeholders, including operations people and enterprise architects who are interested in the technical viability of your approach. I've also noticed that the high-risk requirements are often the high-business-value ones anyway, so you usually need to do very little reorganization of your requirements stack.3. Construction. This phase focuses on implementation risk, addressing it through the creation of working software each iteration. This phase is where you put the flesh onto the skeleton.4. Transition. The goal of this phase is to address deployment risk. There is usually a lot more to deploying software than simply copying a few files onto a server, as I indicated above. Deployment is often a complex and difficult task, one which you often need good guidance to succeed at.

Each phase ends with a milestone review, which could be as simple as a short meeting, where you meet with prime stakeholders who will make a "go/no-go" decision regarding your project. They should consider whether the project still makes sense, perhaps the situation has changed, and that you're addressing the project risks appropriately. This is important for "agile in the small" but also for "agile in the large" because at scale your risks are often much greater. Your prime stakeholders should also verify that you have in fact met the criteria for exiting the phase. For example, if you don't have an end-to-end working skeleton of your system then you're not ready to enter the Construction phase. Holding these sorts of milestone reviews improves your IT governance efforts by giving senior management valuable visibility at the level that they actually need: when you have dozens or hundreds of projects underway, you can't attend all of the daily stand up meetings of each team, nor do you even want to read summary status reports.

These milestone reviews enable you to lower project risk. Last Autumn I ran a survey via Dr. Dobb's Journal (www.ddj.com) which explore how people actually define success for IT projects and how successful we really were. We found that when people define success in their own terms that Agile has a 71% success rate compared with 63% for traditional approaches. Although it's nice to that Agile appears to be lower risk than traditional approaches, a 71% success rate still implies a 29% failure rate. The point is that it behooves us to actively monitor development projects to determine if they're on track, and if not either help them to get back on track or cancel them as soon as we possibly can. Hence the importance of occasional milestone reviews where you make go/no-go decisions. If you're interested in the details behind the project, they can be found at http://www.ambysoft.com/surveys/success2007.html .

Done right, phases are critical to your project success, particularly at scale. Yes, the traditional community seems to have gone overboard with phase-based approaches, but that doesn't mean that we need to make the same mistakes. Let's keep the benefit without the cost of needless bureaucracy.[Read More]

I recently wrote an "e-book" for Internet Evolution overviewing agile software development at scale. The goal of the Agility at Scale: Become as Agile as You Can Be ebook is to get people thinking outside of the box a bit when it comes to agile development strategies and see that they really are ready for primetime.





[Read More]

Although it might not be obvious, and important success factor in adopting agile techniques is to be able to determine whether a team is agile or not. The challenge that many organizations face is that many teams will claim to be agile, yet management, who often has little or no experience with agile approaches, cannot tell which claims are true and which are over zealous (I'm being polite). The following are the criteria that I suggest you look for in a disciplined agile team:1. Produce working software on a regular basis. This is one of the 12 principles behind the Agile Manifesto, and in my experience is a critical differentiator between the teams that are agile and those that are merely claiming it. Ideally the team should produce potentially shippable software each iteration. That doesn't mean that they'll deploy the system into production, or the marketplace, each iteration but they could if required to do so. Typically the team will deploy into a pre-production testing environment or a demo enviroment at the end of each iteration (or more often for that matter).2. Do continuous regression testing, and better yet take a Test-Driven Development (TDD) approach. Agile developers test their work to the best of their ability, minimally doing developer regression testing via a continuous integration (CI) strategy and better yet do developer-level TDD. This approach enables development teams to find defects early, thereby reducing the average cost of addressing the defects, it also helps them to deliver higher quality code and to move forward safely when adding or changing functionality.3. Work closely with their stakeholders, ideally on a daily basis. A common practice of agile teams is to have an on-site customer or product owner who prioritizes requirements and provides information on a timely manner to the team. Disciplined agile teams take it one step further and follow the practice active stakeholder participation where the stakeholders get actively involved with modeling and sometimes even development.4. Are self-organizing within a governance framework. Agile teams are self-organizing, which means that the people doing the work determines how the work will be done, they're not told by a manager who may not even be directly involved with the work how it will be done. In other words the team does its own planning, including scheduling and estimation. Disciplined agile teams are self governing within an effective governance framework.5. Regularly reflect on how they work together and then act to improve on their findings. Most agile teams hold a short meeting at the end of each iteration to reflect upon how well things are working and how they could potentially improve the way that they are working together. Sometimes this is done in a more formalized manner in the form of a retrospective, but often it's done informally. The team then acts on one or more of their suggested improvements the next iteration. Disciplined agile teams take this one step further and measure their software process improvement (SPI) progress over time: the act of taking these measures, perhaps via a product such as Rational Self Check, helps to keep the team on track in their SPI efforts.

I have yet to discover an ad-hoc development team which met all five criteria, and most of them rarely meet two or three.

Further reading:

• Lean Development Governance
• Examining the Agile Manifesto
• Norm Kerth's Retrospectives Home Page
• Measured Capability Improvement Framework (MCIF)

[Read More]

A common goal of IT governance is to determine the productivity of various techniques, tools, and people as part of the overall effort to improve said productivity. If you can easily measure productivity you can easily identify what is working for you in given situations, or what is not working for you, and adjust accordingly. A common question that customers ask me is how do you measure productivity on agile teams. Although you could use traditional strategies such as function point (FP) counting, or another similar strategy, this can require a lot of effort in practice. Remember that we don't only want to measure productivity, we want to do so easily. Ideally it would be nice to do so using information already being generated by the team and therefore we won't add any additional bureaucratic overhead.

A common metric captured by agile teams is their velocity. Velocity is an agile measure of how much work a team can do during a given iteration. At the beginning of an iteration a team will estimate the work that they're about to do in terms of points. At the beginning of a project the team will formulate a point system, which typically takes a few iterations to stabilize, so that they can consistently estimate the work each iteration. Although the point system is arbitrary, my team might estimate that a given work item is two points worth of effort whereas your team might think that it's seven points of effort, the important thing is that it's consistent. So if there is another work item requiring similar effort, my team should estimate that it's two points and your team seven points. With a consistent point system in place, each team can accurately estimate the amount of work that they can do in the current iteration by assuming that they can achieve the same amount of work as last iteration (an XP concept called "yesterday's weather"). So, if my team delivered 27 points of functionality last iteration we would reasonably assume that we can do the same this iteration.

So, is it possible to use velocity as a measure of productivity? The answer is not directly. For example, we have two teams, A and B, each of 5 people and each working on a web site and each having two-week long iterations. Team A reports a velocity of 17 points for their current iteration and team B a velocity of 51 points. They're both comprised of 5 people, therefore team B must be three times (51/17) as productive as team A. No! You can't compare the velocity of the two teams because they're measuring in different units. Team A is reporting in their points and B in their points, so you can't compare them directly, The traditional strategy would be to ask the teams to use the same unit of points, which might be a viable strategy with two teams although likely not if you have twenty agile teams and particularly not if you have two hundred teams. Regardless of the number of teams that you have it would minimally require some coordination to normalize the units and perhaps even some training and development and support of velocity calculation guidelines. Sounds like unnecessary bureaucracy that I would prefer to avoid. Worse yet, so-called "consistent" measurements such as FPs are anything but because there's always some sort of fudge factor involved in the process which will vary by individual estimator.

An easier solution exists. Instead of comparing velocities you instead calculate the acceleration of each team. For example, consider the reported velocities of each team below. Team A's velocity is increasing over time whereas team B's velocity is trending downwards. All things being equal, you can assume that team A's productivity is increasing whereas B's is decreasing. Of course it's not wise to manage simply by the numbers, so instead of assuming what is going on I would rather go and talk with the people on the two teams. Doing so I might find out that team A has adopted quality-oriented practices such as continuous integration and static code analysis which team B has not, indicating that I might want to help team B adopt these practices and hopefully increase their productivity.

Team A: 17 18 17 18 19 20 21 22 22 ...Team B: 51 49 50 47 48 45 44 44 41 ...

There are several advantages to using acceleration as an indicator of productivity over traditional techniques such as FP counting:1. It's easy to calculate. For example, the acceleration of team A from iteration 1 to iteration 6 is (20-17)/17 = 0.176 whereas for team B it is (45-51)/51 = -.118. Of course, you don't need to calculate the acceleration over such a long period of time, you could do it iteration by iteration, although I find that doing it over several iterations gives a more accurate value. You'll need to experiment to determine what works for you.2. It is inexpensive. Acceleration is based on information already being collected by the team, their velocity, so there is no extra work to be done by the team. 3. It is unlikely to be gamed. Teams aren't motivated to fake their velocity because it provides them with important information required to manage themselves effectively. 4. It is easy to automate. For example, Rational Team Concert (RTC) calculates velocity automatically from its work item list (an extension of Scrum's product backlog) and does trend reporting via it's web-based project reporting functionality, providing a visual representation of the team's acceleration (or deceleration as the case may be).5. It offers the opportunity for more effective governance. This approach reflects three of the practices of Lean Development Governance: Simple and Relevant Metrics, Continuous Project Monitoring, and Integrated Lifecycle Environment.6. You can easily adjust for changing team size. If the size of a team varies over time, and it will, this metric falls apart the way that I've described it. To address this issue you need to normalize it by dividing by the number of people on the team to get the average acceleration per team member.7. You can easily monetize this metric. By knowing the acceleration of the project team and knowing how much they're spending each iteration, you can estimate the amount of money you're saving through process improvement. For example, if you're spending $100,000 per iteration and your acceleration is 2%, your cost savings is $2,000 per iteration.

Of course, nothing is perfect, and there are a few potential disadvantages:1. It is an indirect measure of productivity. Truth be told velocity really is a productivity measure, it's just that because it's measured in different units it's difficult to compare between teams. Acceleration is merely an indicator of the change in productivity.2. You actually need to measure what you're interested in. When you step back and think about it, you're not really interested in measuring your productivity, regardless of what the metrics wonks have been telling you the past few decades. In this case what you really want to know is your change in productivity because your real goal is to improve your productivity over time, which is what acceleration actually measures.3. Management must be flexible. For this to be acceptable senior management must be willing to think outside the "traditional metrics box". Using a non-standard, simple metric to calculate productivity? Preposterous! Directly measuring what you're truly interested in instead of calculating trends over long periods of time? Doubly preposterous!4. Your existing measurement program may be questioned. Once management learns how easy it can be to obtain metrics which enables them to truly govern software development projects they may begin to question the investment that they've made in the past in overly complex and expensive metrics schemes. This can be dangerous for the metrics professionals in your organization, particularly if your metrics group doesn't have valid measurements around the value of their own work. Ummmmm....5. The terminology sounds scientific. Terms such as velocity and acceleration can motivate some of us to start believing that we understand the "laws of IT physics", something which I doubt very highly that as an industry we understand. All it would take is for someone to start throwing around terms like "standard theory" and "unified model" and we'd really be in trouble. Wait a minute..... ;-)

In summary, measuring the acceleration of development teams is an easy to collect, straightforward measure of team productivity. I hope that I've given you some food for thought, and would be eager to hear about your experiences applying this metric in practice.

[Read More]

For some reason, it seems as if everyone's grandfather at one point in time recommended to use the right tools for the job.  That's practical wisdom from my point of view, one that is certainly an issue for agile development. 


One of the primary messages, I hope, of the Agile Scaling Model (ASM) is that context counts.  Although the focus of the ASM is on describing a contextual framework for tailoring your process to meet the needs of the situation that you find yourself in, it's also applicable to your tooling selection.  For example, the tool choices of a co-located team will be much different than that of a geographically distributed team.  A co-located team will likely use a whiteboard or paper for their agile modeling efforts, whereas distributed team members may need to capture their diagrams using a more sophisticated tool such as Rational Requirements Composer (RRC) so that their work can be shared electronically.  Having said that, RRC would be overkill for a co-located team (unless they had regulatory compliance issues).  Different teams, different situations, therefore different tooling choices.


One of the concerns that I run into from customers is that some of our legacy products don't support agile very well.  Once again, it's a matter of context because many of our legacy products reflect the realities faced by more traditional teams.  The challenge occurs when you try to take a legacy product which is well suited for traditional development, such as Rational ClearCase, and try to apply it on agile projects.  Although ClearCase makes sense in certain scaling situations, particularly very large teams that are geographically distributed, you'd be better advised to use something like Rational Team Concert (RTC) for configuration management on most agile teams (note that RTC does far more than just SCM).


So, if you're taking an agile approach you should consider Rational tools such as RTC, RRC, Rational BuildForge, Rational AppScan, and others which support agile development.  Granted, some you would only use at scale -- for example Buildforge is a good option in really complex environments, but if you don't face that complexity then you'll likely find that RTC's build engine is sufficient.  Similarly, if you're taking a traditional approach to development then you'll likely consider products such as ClearCase, Appscan, RTC, and Rational Software Architect (RSA) instead.  Different situations, different tooling choices. 


What's even more confusing is that some products support a range of process paradigms.  For example, RTC supports agile, lean, iterative, and traditional approaches to development.  The same can be said of Appscan and several other products.  Notice how I listed RTC and Appscan for both agile and traditional development above. 


So, if anyone tells you that Rational tools don't support agile development don't believe them.  Ask them which tools that they're talking about, and ask them if they're aware of the Rational products that do support agile development.  Context counts.

When you are first adopting agile techniques in your organization a common strategy is to run one or more pilot projects.  When organizing these projects you typically do as much as you can to make them successful, such as finding:

1. Projects where the stakeholders are willing to actively work with you.
2. IT people who are flexible, willing to try new things, and willing to collaborate with one another.
3. IT people who are generalizing specialists, or at least willing to become so.
4. Finding a project which is of medium complexity (therefore it's "real" in the sense that it's significant to your organization) but not one where it can make or break your organization (therefore it's safe to experiment with).

In North America we refer to this as "cherry picking" because you're picking the cherry/best situation that you can find.

Some thoughts:

1. Being agile may not have been the primary determinant of success.  You set up an environment where you have a good relationship with your stakeholders, where you have good people who want to work together, and the project is challenging but not impossible.  Oh, and by the way you adopted a few agile techniques as well.  Sounds to me that situation you could have adopted a few not-so-agile techniques instead and still succeed.  Although my various project success surveys, see my IT surveys page for details, have shown time and again that agile project teams are more successful than traditional project teams I haven't been able to tease out (yet) whether this success is attributable to agile or just attributable to improved project initiation efforts.
   
2. When adopting agile/lean widely across your organization, you can't cherry pick any more.  For the past few years I've been working with IT organizations that are in the process of adopting agile/lean strategies across their entire organization, not just across a few pilot projects.  What these organizations are finding is that they need to find ways to adopt agile where the business isn't as willing to work with IT, where some of the people aren't so flexible or collaborative, where some of the people are narrowly specialized and not as willing to expand their skills, or where the project exhibits scaling factors which motivates you to tailor your agile approach.  It's harder to succeed with agile in these situations because they're not as "cherry" as what you've experienced previously.  Luckily, if you've been successful previously then you now have some agile experienced people, you have successes to reference, and you've likely overcome some problems even in the cherry situations which you have learned from.  So, your cherry successes will hopefully improve your ability to succeed even in "non cherry" situations.
   
3. You need to work smarter, not harder. If the source of your success was actually from improved project initiation practices and not from agile, then recognize that and act accordingly.  Realistically part of your success was from that and part was from agile, and the organizations that adopt a measured improvement approach potentially have the data to determine which practices lead to success and which didn't.  Without the metrics you're effectively flying blind when it comes to deciding how to improve.  There is clearly a mandate for smarter work practices within IT, within your organization as a whole for that matter.
   

If you want to gain more insight into some of the issues that you'll face when adopting agile across your organization, I suspect that you'll find my recent paper Scaling Agile: An Executive Guide to be interesting.  I've got a more detailed paper in the works, so stay tuned to this blog.

Recently I spent some time in the UK with Julian Holmes of Unified Process Mentors. In one of our conversations we deplored what we were seeing in the agile community around certification, in particular what the Scrum community was doing, and he coined the term “integrity debt” to describe the impact it was having on us as IT professionals. Integrity debt is similar to technical debt which refers to the concept that poor quality (either in your code, your user interface, or your data) is a debt that must eventually be paid off through rework. Integrity debt refers to the concept that questionable or unprofessional behavior builds up a debt which must eventually be paid off through the rebuilding of trust with the people that we interact with.
The agile community has been actively increasing their integrity debt through the continuing popularity of Scrum Certification, in particular the program around becoming a Certified Scrum Master (CSM). To become a CSM you currently need to attend, and hopefully pay attention during, a two-day Scrum Master Certification workshop taught by a Certified Scrum Trainer (CST). That’s it. Granted, some CSTs will hold one or more quizzes which you need to pass, an optional practice which isn’t done consistently, to ensure that you pay attention in the workshop. 
Scrum Masters, as you know, take the leadership position on a Scrum team. The idea that someone can master team leadership skills after two entire days of training is absurd. Don’t get me wrong, I’m a firm supporter of people increasing their skillset and have no doubt that many of the CSTs deliver really valuable training. However, there is no possible way that you can master a topic, unless it is truly trivial, in only two days of training. From what I can tell the only thing that is being certified here is that your check didn’t bounce.
The CSM scheme increases the integrity debt of the IT industry by undermining the value of certification. When someone claims that they’re certified there’s an assumption that they had to do something meaningful to earn that certification. Attending a two-day course, and perhaps taking a few quizzes where you parrot back what you’ve heard, clearly isn’t very meaningful. The problem with the term Certified Scrum Master is two-fold: not only does the term Certified imply that the holder of the certification did something to earn it, the term Master implies that they have significant knowledge and expertise gained over years of work. It is very clear that people are falling for the Scrum certification scheme.
A quick search of the web will find job ads requiring that candidates be CSMs, undoubtedly because they don’t realize that there’s no substance behind the certification. Whenever I run into an organization that requires people to be CSMs I walk them through the onerous process of earning the designation and suggest that they investigate the situation themselves. Invariably, once they recognize the level of deception, the customer drops the requirement that people be CSMs.
Another quick search of the web will find people bragging about being a CSM, presumably being motivated by the employment opportunities within the organizations gullible enough to accept Scrum certification at face value. My experience is that the people claiming to be CSMs are for the most part decent, intelligent people who 99.99% of the time have far more impressive credentials to brag about than taking a two-day course. Yet, for some reason they choose to park their integrity at the door when it comes to Scrum certification. I suspect that this happens in part because they see so many other people doing it, in part because they’re a bit desperate to obtain or retain employment in these tough economic times, and in part because the IT industry doesn’t have a widely accepted code of ethical conduct. These people not only embarrass themselves when they indicate on their business cards or in their email signatures that they’re Certified Scrum Masters they also increase the integrity debt of the agile community as a whole.
Yet another search of the web will find people bragging about being Certified Scrum Trainers (CSTs), the people whom have been blessed by the Scrum Alliance to deliver Scrum master certification courses. Once again, my experience is that these are intelligent, skilled people, albeit ones who have also parked their integrity at the door in the pursuit of a quick buck. Surely these people could make a decent living via more ethical means? I know that many of them have done so in the past, so I would presume that they could do so in the future. The actions of the CSTs increase our integrity debt even further.
The group of people who have most embarrassed themselves, in my opinion, are those whom we consider thought leaders within the agile community. Leaving aside the handful who are directly involved with the Scrum certification industry, the real problem lies with those who have turned a blind eye to all of this. The Scrum certification scheme was allowed to fester within our community because few of our thought leaders had the courage to stand up and publicly state what they were talking about in private. This of course is all the more galling when you consider how much rhetoric there is around the importance of courage on software development projects. As Edmund Burke once observed, all that is necessary for evil to triumph is for good men to do nothing.

There are several things that we can do today to start paying off some of our integrity debt:

1. Be discerning, not deceptive. If you’re going to list credentials on your email signature or business card then only choose to list the ones that actually mean something.
2. Educate human resources people. Make them aware of what “Certified Scrum Master” really means and let them think for themselves. I highly suspect that if HR people realized what was going on the demand for CSMs would plummet, and in turn people wouldn’t be tempted by Scrum certification.
3. Act professional, don’t just claim to be certified. Instead of signing up for every easy certification that comes your way why not simply do a good job and let the people you work with be your claim to fame? The good news is that for the past few years the agile community has tried to pay down some of the IT industry’s integrity debt that we have with our stakeholders by providing better return on investment (ROI), delivering systems which are more effective at addressing the needs of your stakeholders, by working in a more timely manner, and by producing greater quality work. All of these claims are borne out by the 2008 Software Development Project Success Rate Survey by the way.
4. Recognize that adding a test doesn’t address the underlying problems. For the past year there’s been a move afoot to have people pass a test as part of earning their CSM (apparently it’s been a challenge to create a non-trivial test to validate your understanding of a topic that you can master by taking a two-day training course). This is something that should have been done from the very beginning, along with some sort of peer review, not years later when the damage has been done. Adding a test at this late date isn’t going to remove the stink that’s built up over the years, but sadly it will fool a few people into believing that they’ve covered it up.
5. Recognize that there is a demand for certification. The agile community needs to put together a decent certification program, something that the Scrum Alliance has clearly failed at doing. My article Coming Soon: Agile Certification provides some thoughts as to what we need to do. The good news is that people such as Ron Jeffries and Chet Hendrickson, and others, are putting together a developer certification program. The really good news is that these are the right people to do this. The really bad news is that they’re doing it under the aegis of the Scrum Alliance, so whatever they accomplish will unfortunately be tainted by the fallout of the CSM debacle.

If we're going to scale agile software development strategies to meet the range of challenges faced by modern organizations, we need to be trustworthy.  Is claiming to be a certified master after taking a two-day course an act which engenders trust?  I don't think so.  As individuals we can choose to do better.  As a community we need to.

Suggested Reading

1. Agile Certification -- A humorous look at certification.
2. IT Surveys -- A great resource for statistics about what IT people are actually doing in practice.

I was recently in Bangalore speaking at the Rational Software Conference, which was really well done this year, and visiting customers.  In addition to discussing how to scale agile software development approaches, particularly when the team is distributed geographically and organizationally, I was also asked about what I thought about a software factory approach to development.  My instinctual reaction was negative, software factories can result in lower overall productivity as the result of over specialization of staff (I prefer generalizing specialists), too many hand-offs between these specialists (I find close collaboration to be far more effective), and too much bureaucratic overhead to coordinate these activities.  I initially chalked it up to these people still believing that software development was mostly a science, or perhaps an engineering domain, whereas my experiences had made me come to believe that software development is really more art than it is a science.  Yet, the consistent belief in this strategy by very smart and experienced people started me thinking about my position. 

Just let me begin by saying that this blog posting isn't meant to be yet another round in the age old, and relatively inane, "art vs. science" debate within the software development community.   That debate is a symptom of versusitis, a dread disease which particularly plagues the IT industry and which can any of us at any time.  There is no known cure, although the combination of experience, open-mindedness, and critical thought are the best inoculation against versusitis that we have so far. In that vein, let me explore the issues as I see them and I will let you think for yourself.

On the one hand software development has aspects of being an art for several reasons.  First, the problem definition is never precise, nor accurate, and even when we have detailed specifications the requirements invariably evolve anyway.  The lack of defined, firm requirements requires us to be flexible and to adjust to the situation that we find ourselves in.  Second, teams typically find themselves in unique situations, necessitating a unique process and tool environment to reflect this (assuming that you want to be effective, otherwise there's nothing stopping you from having a "repeatable process" and consistent tool environment).  Third, software is built by people for people, requiring that the development team have the ability to build a system with a user interface which meets the unique needs of their end users.  One has only to look at the myriad UI designs out there to see that surely there is a bit of art going on.  Fourth, if software development wasn't at least partially art then why hasn't anyone succeeded at building tools which take requirements as inputs and produce a viable solution that we can easily deploy?  It's been over four decades now, so there's been sufficient time and resources available to build such tooling.  Fifth, regardless of how much of a scientific/business facade we put over it, our success rate at producing up front detailed cost estimates and schedules speak for itself (see Funding Agile Projects for links to articles).

On the other hand software development has aspects of being a science for several reasons.  First, some aspects of software development have in fact  been automated to a significant extent.  Second, there is some mathematical basis to certain aspects of software development (although in the case of data-oriented activities the importance of relational theory often gets blown way out of proportion and I have yet to see a situation where formal methods proved to be of practical value). 

What does this have to do with Agility@Scale.  As you know, one of the agile scaling factors is Organizational Complexity, and cultural issues are the hardest to overcome.  Whether your organization believes that software development is mostly an art or mostly a science is a cultural issue which will be a major driver in you choice of methods and practices.  Organizations which believe that software development is more of a science will prefer strategies such as software factories, model-driven architecture (MDA), and master data management (MDM).  And there is ample evidence to support the claims that some organizations are succeeding at these strategies.  Although you may not agree with these strategies, you need to respect the fact that many organizations are making them work in their environments.  Similarly, organizations which believe that software development is more of an art will find that agile and lean strategies are a better fit, and once again there is ample evidence that organizations are succeeding with these approaches (there's also evidence that agile projects are more successful than traditional projects, on average).  Once again, you may not agree with these strategies but you need to respect the fact that other people are making them work in practice. 

Trying to apply agile approaches within an organization that believes software development is mostly a science will find it difficult at best, and will likely need to embark on a multi-year program to shift their culture (likely an expensive endeavor which won't be worth the investment).  Similarly, trying to apply a software factory strategy in an organization that believes that software development is mostly an art will also run aground.  The bottom line is that one size does not fit all, that one strategy is not right for all situations and that you need to understand the trade-offs of various strategies, methodologies, techniques, and practices and apply them appropriately given the situation that you face.  In other words, it depends!  If you are embarking on a software process initiative, and you don't have the broad experience required to effective choose between strategies (very few organizations do, although many believe otherwise), then you should consider Measured Capability Improvement Framework (MCIF) to help increase your chance of success.

In the early days of agile, the applications where agile development was applied were smaller in scope and relatively straightforward. Today, the picture has changed significantly and organizations want to apply agile development to a broader set of projects. Agile hence needs to adapt to deal with the many business, organization, and technical complexities today’s software development organizations are facing. This is what Agility@Scale is all about – explicitly addressing the complexities which disciplined agile delivery teams face in the real world.These agile scaling factors which we've found to be important are:

1. Team size. Mainstream agile processes work very well for smaller teams of ten to fifteen people, but what if the team is much larger? What if it’s fifty people? One hundred people? One thousand people? Paper-based, face-to-face strategies start to fall apart as the team size grows.
2. Geographical distribution. What happens when the team is distributed, perhaps on floors within the same building, different locations within the same city, or even in different countries? Suddenly effective collaboration becomes more challenging and disconnects are more likely to occur.
3. Compliance requirement. What if regulatory issues – such as Sarbanes Oxley, ISO 9000, or FDA CFR 21 – are applicable? These issues bring requirements of their own that may be imposed from outside your organization in addition to the customer-driven product requirements.
4. Enterprise discipline. Most organizations want to leverage common infrastructure platforms to lower cost, reduce time to market, and to improve consistency. To accomplish this they need effective enterprise architecture, enterprise business modeling, strategic reuse, and portfolio management disciplines. These disciplines must work in concert with, and better yet enhance, your disciplined agile delivery processes.
5. Organizational complexity. Your existing organization structure and culture may reflect traditional values, increasing the complexity of adopting and scaling agile strategies within your organization. To make matters worse different subgroups within your organization may have different visions as to how they should work. Individually the strategies can be quite effective, but as a whole they simply don’t work together effectively.
6. Organization distribution. Sometimes a project team includes members from different divisions, different partner companies, or from external services firms. This lack of organizational cohesion can greatly increase the risk to your project.
7. Technical complexity. Some applications are more complex than others. It’s fairly straightforward to achieve high-levels of quality if you’re building a new system from scratch, but not so easy if you’re working with existing legacy systems and legacy data sources which are less than perfect. It’s straightforward to build a system using a single platform, not so easy if you’re building a system running on several platforms or built using several disparate technologies. Sometimes the nature of the problem that your team is trying to address is very complex in its own right.

Each factor has a range of complexities, and each team will have a different combination and therefore will need a process, team structure, and tooling environment tailored to meet their unique situation.


Further reading:

• Rational's Agile Software Development Home Page (for webcasts, whitepapers, ...)

[Read More]

I recently ran into an interesting issue at a customer organization.  This customer is in the process of transitioning to Disciplined Agile Delivery (DAD) and part of that effort is to train, mentor, and coach their people in these new ideas and techniques.  The challenge is that some of "their people" are full time employees (FTEs) and some are contractors/consultants.   When we were planning an upcoming DAD workshop with them, part of the planning effort was to identify who should get that training, which we're delivering in a just-in-time (JIT) basis on a team-by-team basis.  The only people invited to take the training were FTEs because the customer has a policy of not training contractors.   I pushed back a bit on this, but they were adamant about not training contractors because their view was that contractors should either have the skills required to do their jobs or be willing to get those skills on their own time.  Fair enough, but from an agile team building point of view this isn't ideal. 

This situation got me thinking a bit.  One issue is that not all contractors are the same.  Some are short term contractors that are brought in for a specific purpose, they're paid well, and then they move on.  Other contractors stay much longer, sometimes months or even years, and as a result gain deeper knowledge and understanding of your business.  For these longer term contractors it seems to me that there is little difference between them and FTEs, perhaps only in the way that they're remunerated.  Some countries such as the United States now have laws in place limiting how long someone is allowed to remain a contractor because these similarities lead to interesting legal questions around extending benefits to them.

Another issue is that if you intend to build teams from both FTEs and contractors, it behooves you to ensure that these people get similar training, coaching and mentoring to streamline the transition effort. 

Here's the logic I would suggest to address the issue of whether or not to train a contractor:

1. Is the contractor going to be assigned to a key project/product for the organization?  If not, don't train them.
2. Are they someone you want to keep long term?  If not, don't train them and consider not putting them on the new agile team at all.
3. Does the contractor work for a large service provider?  If yes, ask the service provider to cover the costs of training.
4. Is the contractor an independent or working for a smaller service provider?  If yes, include the person in the training if there's room but don't pay their wage during the training period (so you effectively share the investment/cost of training).

As always, let the context of the situation drive your strategy.

A fair question to ask is why should your organization consider adopting the Disciplined Agile Delivery (DAD) process framework.  I believe that there are several clear benefits to doing so:

1. DAD shows how agile techniques fit together.  DAD is a hybrid that adopts strategies from a variety of sources, including Scrum, Extreme Programming (XP), Agile Modeling, Kanban, Outside In Development (OID) and many more.  More importantly DAD's process-goal driven approach shows how this all fits together, providing advice for when (and when not) to use each technique and the advantages and disadvantages of doing so.  In doing so DAD enables you to streamline your efforts to tailor agile to reflect the context of the situation you find yourself in.  Furthermore, it provides this advice in the context of a full agile delivery lifecycle, explicitly showing how to initiate a project, construct a solution, and then deploy into production.  Instead of starting with a small agile method such as Scrum and doing all the work to figure out how to tailor ideas from other methods to actually make it work, why not start with a framework that's already done all that work for you?
2. DAD isn't prescriptive.  DAD is far less prescriptive than other agile methods.  For example, where Scrum prescribes a single strategy for managing changing requirements, a strategy called a Product Backlog, DAD suggests several strategies and provides advice for choosing the right one.  Where other agile methods define a single lifecycle, DAD instead describes several lifecycles (an agile Scrum-based one, a lean lifecycle, and a continuous delivery lifecycle to name just three) and once again describes how to choose the right one for your situation.  Your agile team is in a unique situation, and as a result needs a flexible process framework that provides coherent, easy-to-follow tailoring advice.   Isn't it better to adopt strategies that reflect the context that you actually face?
3. DAD explicitly addresses architecture.  Even after a decade of agile software development it still seems that the topic of how agile teams address architecture is a mystery for many people.  As a result DAD builds agile architecture strategies right in, starting with initial architecture envisioning, to proving the architecture with working code, to evolutionary design strategies during construction. 
4. DAD explicitly addresses DevOps.  DevOps is the art of combining development and operations approaches in such a way as to streamline your overall efforts.  DAD "bakes in" DevOps through explicit support for common DevOps practices as well as its robust stakeholder definition which explicitly includes operations and support people.
5. DAD explicitly addresses governance.  Although governance is often considered a dirty word within some agile circles, the reality is that software development teams can and should be governed.  Sadly, many agile teams have traditional governance strategies inflicted upon them, strategies which invariably increase schedule, cost and risk on the project.  But is doesn't have to be this way.  It is possible, and very desirable to adopt agile goverance strategies, strategies which are built right into the DAD framework.
6. DAD explicitly addresses many other important development activities.  Architecture, DevOps, and governance are just the tip of the iceberg.  DAD also shows how critical activities such as analysis, design, testing, quality, technical writing, and many more are addressed in an agile and streamlined manner throughout the delivery lifecycle.  As described earlier, this is done in a non-prescriptive and tailorable manner, thereby removing a lot of the mystery regarding how this agile stuff all fits together into a coherent whole.
7. DAD is solution focused, not software focused.  Although the rhetoric around "potentially shippable software" resonates well with developers it observably and empirically misses the mark.  DAD promotes the more robust idea of "potentially consumable solutions".  Yes, shipping is nice but shipping something that people actually want to use/buy, something that is consumable, is much nicer.  Yes, software is part of the equation but that software runs on hardware, we often also need to develop supporting documentation, we often evolve the business process, and even evolve the organization structure around the usage of the system.  In other words, we're really producing solutions, not just software.  Isn't it better to adopt rhetoric that actually reflects what we're doing in practice?
8. DAD promotes enterprise awareness over team awareness.  One of the great benefits of an agile approach to software development is its focus on the team.  This can also be a bit of a problem, because a team-focused strategy can result in suboptimal decisions for your overall organization.  DAD promotes the idea that disciplined agilists should be enterprise aware, working towards common business and technical goals while leveraging and enhancing the existing infrastructure whenever possible.
9. DAD provides a foundation from which to scale.  The starting point for scaling agile is to understand how agile strategies fit together from project initiation to delivery into production.  If you don't know how to succeed with agile in a straightforward situation then it will prove very difficult to do so in an agility @ scale situation.  DAD not only provides this tailorable foundation from which to scale agile it also takes a robust view of what it means to scale agile (hint: large or distributed teams are only a start).
10. DAD provides a basis for enterprise agile.  As organizations move towards a true "enterprise agile" approach they often find that they need to adopt either DAD as a foundation or invest a fair bit of effort inventing something similar.  They are also starting to adopt strategies from the SAFe framework, or reinventing such, as well as ideas from sources such as Enterprise Unified Process (EUP) (sadly, poorly named in hindsight), ITIL, and even CoBIT.  More on this in a future blog posting.

In short, DAD provides a lot of proven advice culled from years of experience applying agile software techniques in enterprise-class environments.  Instead of figuring all of this stuff out on your own, why not jump ahead and leverage the hard-won lessons learned from other organizations that have already dealt with the challenges that you're struggling with today?

The primary shortcoming of the DAD framework is it makes it very clear that software development, oops I mean solution delivery, is quite complex in practice.  As IT practitioners we inherently know this, but it seems that we need to be reminded of this fact every so often.  DAD doesn't provide a simplistic, feel-good strategy that you can learn in a few hours of training.  Instead it defines a coherent, tailorable strategy that reflects the realities of enterprise IT. 

There is a wealth of information at DAD posted at the  Disciplined Agile Delivery (DAD) web site and great discussions occuring on the DAD LinkedIn discussion forum.  For those of you interested in agile certification, the Disciplined Agile Consortium site will prove valuable too, in particular the list of upcoming DAD workshops provided by several IBM partners.  And of course the book Disciplined Agile Delivery: A Practitioner's Guide to Agile Software Delivery in the Enterprise (IBM Press, 2012) written by Mark Lines and myself is a very good read.