Modified by ScottAmbler
An imporant step in scaling your agile strategy is to adopt a Disciplined Agile Delivery (DAD)
approach instead of one which is just focused on agile construction. One aspect of adopting a DAD approach it to mature your focus from just producing software to instead providing a solution which meets the needs of its stakeholders within the appropriate economic, cultural, and technical constraints. The fundamental observation is that as IT professionals we do far more than just develop software. Yes, this is clearly important, but in addressing the needs of our stakeholders we will often:
Provide new or upgraded hardware
Change the business/operational processes which stakeholders follow
Change the organizational structure in which our stakeholders work
Update supporting documentation
And yes, develop high-quality software
Although delivery of high-quality, working software is important it is even more important that we deliver high-quality working solutions to our stakeholders. Minimally IT professionals should have the skills and desire to produce good software, but what they really need are the skills and desire to provide good solutions. We need strong technical skills, but we also need strong "soft skills" such as user interface design and process design to name just two.
The shift to a solution-oriented focus from a software-oriented focus requires your agile teams to address some of the software-oriented prejudices which crept into the Agile Manifesto
. The people who wrote the manifesto (which I fully endorse) were for the most part software developers, consultants, and in many cases both. It is little wonder that this group would allow a bias towards software development creep into the language of their manifesto.
Just like there are 5Ps of marketing, there are also “5 Ps” of IT:
- People. People and the way they work together have a greater effect on the outcomes of a project than the processes they’re following or the products (tools and technologies) that they’re using. People issues include having visible executive sponsorship, building an environment of trust, empowering staff, focusing on leadership as well as management, recognizing that the primary gating factor when improving processes is people’s ability to absorb change, and promoting a cross-discipline strategy at both the team and individual levels.
- Principles/philosophies. We’ve found both internally within IBM as well as with many of our customers that there is a need to define a common set of principles to provide a consistent foundation to enable effective teamwork and continuous process improvement. These principles help to guide people’s decisions when their processes and practices don’t directly address the situation which they find themselves in.
- Practices/patterns. A practice is a self-contained, deployable component of a process. You might find the IBM Practices interesting.
- Products. This includes the technologies – such as databases, application servers, networks, and client platforms – and tools such as integrated development environments, testing tools, and project planning tools used to create solutions for stakeholders.
- Processes. The previous 4Ps do not exist in a vacuum, we need some sort of glue to help piece all of this together. Minimally this glue is a lifecycle although more often than not it is a full process or method.
My experience is that to be successful at software process improvement (SPI)
across your entire IT department that you must address these 5Ps. How you address each issue, and to what extent, will vary based on your situation.
My January 2010 DDJ Agile Update, Tragic Mistakes When Adopting Test Driven Development (TDD)
, is now online. In the article I summarize what I consider to be common, and tragic, mistakes that I'm seeing organizations make when they attempt to adopt TDD.
These mistakes include:
The article also goes into potential benefits of TDD as well as potential challenges that you're face when adopting it.
As you may know I write the agile update newsletter for Dr. Dobb's Journal (DDJ)
. One of the things that I do for DDJ is run the "State of the IT Union" surveys to find out what IT professionals are actually doing in practice. I invite you to fill out the January 2010 edition of the survey at http://www.surveymonkey.com/s/StateOfITUnion
. The survey should take you about 5-7 minutes to complete, and your privacy will be completely protected.
At the end of the survey you will be given the chance to be entered into a draw for one of ten copies of "The Art of Scalability: Scalable Web Architecture, Processes, and Organizations for the Modern Enterprise" by Martin Abbott and Michael Fisher published in December 2009 by Addison Wesley.
The results of this survey will be summarized in a forthcoming newsletter, most likely my March 2010 agile update. Furthermore, this is an open survey, so the source data (without identifying information to protect your privacy), a summary slide deck, and the original source questions will be posted at www.ambysoft.com/surveys/
so that others may analyze the data for their own purposes. Data from previous surveys have been used by university students and professors for their research papers, and hopefully the same will be true of the data from this survey. The results from several other surveys are already posted there, so please feel free to take advantage of this resource.
activities are evolutionary (iterative and incremental) and highly collaborative in nature. Initially requirements are explored at a high level via requirements envisioning
at the beginning of the project and the details are explored on a just-in-time (JIT) basis via iteration modeling
and model storming
activities. The way that you perform these agile practices, and the extent to which you do so, depends on the situation in which a project team finds itself. The Agile Scaling Model (ASM)
is a contextual framework for effective adoption and tailoring of agile practices to meet the unique challenges faced by a system delivery team of any size. To see how this works, let's apply the concepts of the ASM to see how we would scale our agile approach to requirements.
First, let's consider how a small, co-located team would work. The first two categories of the ASM are core agile development and disciplined agile delivery
, the focus of both are small co-located teams in a fairly straightforward situation. In these situations simple techniques such as user stories
written on index cards and sketches on whiteboards
work very well, so the best advice that I can give is to stick with them. Some teams will take a test-driven development
(TDD) approach where they capture their requirements and design in the form of executable specifications
, although this sort of strategy isn't as common as it should be (yet!), likely because of the greater skill and discipline that it requires. Traditionalists often balk at this approach, believing that they need to document the requirements in some manner. But, for a small co-located team working in a collaborative manner, requirements documentation proves to be little more than busy work, often doing nothing more than justifying the existence of a business analyst who hasn't made the jump to agile yet. Don't get me wrong, there are good reasons to write some requirements documentation, and we'll see this in a minute, but you should always question any request for written specifications and try to find more effective ways to address the actual goal(s) motivating the request. Never forget that written documentation
is the least effective communication
option available to you.
Although inclusive tools
such as whiteboards and paper work well for requirements, for development activities you will need electronic tools. You will either put together an environment from point-specific tools or adopt something more sophisticated such as IBM Rational Team Concert (RTC)
which is already fully integrated and instrumented. RTC is a commercial tool, but luckily you can download a 10-license environment free of charge, which is just perfect for a small team. Larger teams, of course, will need to purchase licenses. One of the things that a disciplined agile delivery approach adds to core agile development is it addresses the full delivery life cycle, which is important because it explicitly includes pre-construction activities such as requirements envisioning. The first step in scaling agile techniques is to adopt a full delivery life cycle which covers the full range of activities required to initiate a project, produce the solution, and then release to solution to your end users.
More interesting is the third category of the ASM, Agility@Scale, and how its eight agile scaling factors
affect the way that you tailor your process and tooling strategy. Let's explore how each one could potentially affect your agile requirements strategy:
- Geographical distribution. The majority of agile teams are distributed in some manner -- some people are working in cubicles or private offices, on different floors, in different buildings, or even in different countries -- and when this happens your communication and coordination risks goes up. To counter this risk you will need to perform a bit more requirements envisioning up front to help ensure that everyone is working to the same vision, although this doesn't imply that you need to write detailed requirements speculations which would dramatically increase the risk to your project. Remember, agilists do just barely enough modeling and are prepared to iteratively elicit the details when they need to do so. The more distributed the team is the more likely they will need to adopt software-based requirements modeling tools such as IBM Rational Requirements Composer (RRC) which supports streamlined, agile requirements elicitation throughout the delivery life cycle. Index cards and whiteboards are great, but they're difficult to see if you're outside the room where they're posted. I've written a fair bit about distributed agile development in this blog.
- Team size. Some organizations, including IBM, are successfully applying agile techniques with teams of hundreds of people. A team of one hundred people will naturally work much differently than a team of ten people, or of one thousand people. Large teams are organized into collections of smaller teams, and the requirements for the overall project must be divvied up somehow between those teams. The implications are that as the team size grows you will need to invest a bit more time in initial requirements envisioning, and in initial architecture envisioning for that matter; you will need to use more sophisticated tools; and may need to use more sophisticated modeling techniques such as use cases and functional user interface prototypes. See large agile teams for more advice.
- Compliance requirement. When regulatory issues – such as Sarbanes Oxley, ISO 9000, or FDA CFR 21 – are applicable you are likely going to be required to capture requirements specifications in some manner and to enact traceability between those requirements. However, I highly recommend that you read the actual regulations yourself and don't let bureaucrats interpret them for you (doesn't it always seem that their interpretation always results in an onerous, documentation heavy solution?) because I have yet to run into a regulation which required you to work in an ineffective manner. Managing your requirements as work items in RTC can often more than meet your regulatory requirements for documentation and traceability, although you may want to consider a tool such as IBM Rational RequisitePro for complex regulatory situations.
- Domain complexity. The manner in which you elicit requirements for a data entry application or an informational web site will likely be much simpler than for a bio-chemical process monitoring or air traffic control system. More complex domains will require greater emphasis on exploration and experimentation, including but not limited to prototyping, modeling, and simulation. Although user stories may be effective as a primary requirements artifact in simple domains, in more complex domains you are likely to find that you need to drive your requirements effort with more sophisticated modeling techniques.
- Organization distribution. Sometimes a project team includes members from different divisions, different partner companies, or from external services firms. In these cases, particularly where the work is strictly organized between the various organizations (perhaps for security concerns), you may need a more sophisticated approach to managing the requirements. RTC enables you to organize the requirements between teams, and then to automatically track progress in real time via the RTC project dashboard.
- Technical complexity. The technical complexity of a solution can vary widely, from a single platform silo application to a multi-platform application working with legacy systems and data to a full-blown systems engineering effort. Complex technical domains, just like complex business domains, require more complex strategies for requirements elicitation and management. The requirements for your legacy systems are likely to have been captured using tools and techniques appropriate for that platform, for example the requirements for your COBOL application may have been captured using data flow diagrams and data models, whereas the requirements for your Java legacy application where captured using UML diagrams. The subteam working on the COBOL system might be using IBM Rational Application Developer (RAD) and RTC for Z whereas the Java subteam may use Eclipse with RTC. Because systems engineering projects can stretch on for years, particularly when the hardware is being developed in parallel to the software, sophisticated tooling such as IBM Rational DOORS is often used in these situations. For more information about systems engineering, see the IBM Rational Harmony process.
- Organizational complexity. Your approach to requirements elicitation and management will be affected by a host of organizational complexities, including your corporate culture. When the culture is flexible and collaborative you can be very agile in your approach to requirements, but as it becomes more rigid you become more constrained in what is considered acceptable and thus take on greater project risk. For example, many organizations still struggle with their approach to funding projects, often demanding that the project team provides an "accurate" estimate up front to which they will be held to. This in turn motivates risky behavior on the part of the development, including a "big requirements up front (BRUF)" approach where a detailed requirements speculation is developed early in the project. This is just one example of how questionable corporate culture can impact the way in which an agile team works.
- Enterprise discipline. Some organizations have enterprise-level disciplines, such as enterprise architecture, enterprise business modeling, strategic reuse, and portfolio management in place. These disciplines can easily be agile and from what I can tell the more successful efforts appear to lean more towards the agile end of the spectrum rather than the traditional end. Having an enterprise business modeling effort underway will affect your project-level requirements strategy -- you'll be able to leverage existing models, have access to people who understand the domain at an enterprise level, and will likely need to map your project efforts back to your enterprise models. The enterprise modelers will likely be using tools such as IBM Rational System Architect or IBM Websphere Business Modeler.
It is important to note that the way that you tailor the agile practices that you follow, and the tools that you use, will reflect the situation that you find yourself in. In other words, you need to right size your process and the Agile Scaling Model (ASM) provides the context to help you do so. As you saw above, in simpler situations you will use the simpler tools and techniques which are commonly promoted within the core agile development community. But, when things become a bit more complex and one or more of the scaling factors applies you need to modify your approach -- just don't forget that you should strive to be as agile as you can be given the situation that you find yourself in.
On Tuesday, Dec 1, 2009 Philippe Kruchten
, Bruce MacIsaac
, and myself participated on two virtual panels about the future of the Unified Process (we did two to support callers from around the globe) for the Global Rational User's Group (GRUG)
. During the panel sessions we discussed a bit of the history of the Unified Process, some of the misconceptions people have with it, some of the common mistakes people made implementing it (instantiating it to be documentation heavy and/or serial) due to those misconceptions, how it can be very agile if you choose to instantiate it that way, the OpenUP
, the AUP
, how UP relates to the IBM Practices
, and other topics.
The links to the recordings are:
Hope you find it interesting. As I've written in the past, the RUP can be as agile as you want to make it. Furthermore, there are a lot of really good ideas in the RUP that the agile community can and should choose to mine, although sadly I see far too many teams doing things the hard way and reinventing the process wheel on their own. I hope they're enjoying themselves, because it clearly isn't a very efficient way for them to go about process improvement.
The Agile Scaling Model (ASM) is a contextual framework for effective adoption and tailoring of agile practices to meet the unique challenges faced by a system delivery team of any size.
The ASM distinguishes between three scaling
- Core agile development. Core agile methods, such as Scrum and Agile Modeling, are self governing, have a value-driven system development lifecycle (SDLC), and address a portion of the development lifecycle. These methods, and their practices, such as daily stand up meetings and requirements envisioning, are optimized for small, co-located teams developing fairly straightforward systems.
- Disciplined agile delivery. Disciplined agile delivery processes, which include Dynamic System Development Method (DSDM) and Open Unified Process (OpenUP), go further by covering the full software development lifecycle from project inception to transitioning the system into your production environment (or into the marketplace as the case may be). Disciplined agile delivery processes are self organizing within an appropriate governance framework and take both a risk and value driven approach to the lifecycle. Like the core agile development category, this category is also focused on small, co-located teams delivering fairly straightforward systems. To address the full delivery lifecycle you need to combine practices from several core methods, or adopt a method which has already done so.
- Agility at Scale. This category focuses on disciplined agile delivery where one or more scaling factors are applicable. The eight scaling factors are team size, geographical distribution, regulatory compliance, organizational complexity, technical complexity, organizational distribution, domain complexity, and enterprise discipline. All of these scaling factors are ranges, and not all of them will likely be applicable to any given project, so you need to be flexible when scaling agile approaches to meet the needs of your unique situation. To address these scaling factors you will need to tailor your disciplined agile delivery practices and in some situations adopt a handful of new practices to address the additional risks that you face at scale.
The first step in scaling agile approaches is to move from partial methods to a full-fledged, disciplined agile delivery process. Mainstream agile development processes and practices, of which there are many, have certainly garnered a lot of attention in recent years. They’ve motivated the IT community to pause and consider new ways of working, and many organizations have adopted and been successful with them. However, these mainstream strategies (such as Extreme Programming (XP) or Scrum, which the ASM refers to as core agile development strategies) are never sufficient on their own; as a result organizations must combine and tailor them to address the full delivery life cycle. When doing so the smarter organizations also bring a bit more discipline to the table, even more so than what is required by core agile processes themselves, to address governance and risk.
The second step to scaling agile is to recognize your degree of complexity. A lot of the mainstream agile advice is oriented towards small, co-located teams developing relatively straightforward systems. But once your team grows, or becomes distributed, or you find yourself working on a system that isn’t so straightforward, you find that the mainstream agile advice doesn’t work quite so well – at least not without sometimes significant modification. Each of the scaling factors introduces their own risks, and when addressed effectively can actually reduce project risk, and for your project team to succeed you will want to identify the scaling factors applicable to the situation that you face and act accordingly. Unfortunately, this is a lot easier said (OK, in this case blogged about) than done.
IBM Rational advocates disciplined agile delivery as the minimum that your organization should consider if it wants to succeed with agile techniques. You may not be there yet, still in the learning stages. But our experience is that you will quickly discover how one or more of the scaling factors is applicable, and as a result need to change the way you work.
When you’re inside, safe in the warmth of your home watching snow fall on your driveway outside, all snowflakes look the same. But, when you look at a snowflake up close, particularly when you do so under a microscope, you quickly discover that all snowflakes are in fact unique.
It’s the same with IT projects.
When you look at them from afar, particularly
from a very high level, they all look the same.
However, when you look at them up close, you quickly discover that they too
The agile scaling factors
, which are really just general scaling factors applicable to all types of IT
project regardless of paradigm, help to make this very clear.
For example, when it comes to team size some teams
are small, less than ten people perhaps, some are medium sized, and some are very
large (with hundreds of people).
comes to distribution some teams are co-located in the same room, some teams
have team members in different cubicles in the same building, some have people
working in different buildings, and some even have people working in different
Many agile teams work in
regulatory environments, in fact the July 2009 DDJ State of the IT Union survey reports that one third of agile teams must
comply to industry regulations, although clearly many agile teams do not have
this as a concern. That’s only three
scaling factors. The point is that a
small, co-located team working in a non-regulatory environment will work much
differently than a fifty-person team working in three different locations,
which in turn works differently than a two hundred person team in the same
building working in a regulatory situation.
Different teams, facing different scaling issues will work in different
ways – unique snowflakes from a process point of view.
At IBM Rational we define disciplined agile delivery as:
Disciplined agile delivery is an evolutionary (iterative and incremental) approach which regularly produces high quality solutions in a cost effective and timely manner via a risk and value driven life cycle. It is performed in a highly collaborative, disciplined, and self-organizing manner within an appropriate governance framework, with active stakeholder participation to ensure that the team understands and addresses the changing needs of its stakeholders to maximize business value provided. Disciplined agile delivery teams provide repeatable results by adopting just the right amount of ceremony for the situation which they face.
Let’s explore the key points in this definition:
- Full delivery life cycle. Disciplined agile delivery processes have life cycles which are serial in the large and iterative in the small. Minimally they have a release rhythm which recognizes the need for start up/inception activities, construction activities, and deployment/transition activities. Better yet, they include explicit phases as well. It is very important to note that these are not the traditional waterfall phases – requirements, analysis, design, and so on – but instead different “seasons” of a project. The point is that we need to look beyond agile software development and consider the full complexities of solution delivery. Adopting a full delivery life cycle, not just a construction life cycle, is arguably the “zeroth” agile scaling factor.
- Evolutionary. Agile strategies are both iterative and incremental in nature. Iterative means that you are working in a non-serial manner, on any given day you may do some requirements analysis, some testing, some programming, some design, some more testing, and so on. Incremental means that you add new functionality and working code to the most recent build, until such time as the stakeholder determines there is enough value to release the product.
- Regularly produces high quality solutions. Agilists are said to be quality focused. They prefer to test often and early, and the more disciplined ones even take a test-first approach where they will write a single test and the just enough production code to fulfill that test (then they iterate). Many agile developers have adopted the practice of refactoring, which is a technique where you make simple changes to your code or schema which improves its quality without changing its semantics. Adoption of these sorts of quality techniques seems to work – it appears that agile teams are more likely to deliver high quality systems than traditional teams (according to the DDJ 2008 Project Success survey). Within IBM we take it one step further and focus on consumability, which encompasses quality and other features such as ease of deployment and system performance. Furthermore, although some agile methods promote the concept of producing “potentially shippable software” on a regular basis, disciplined agile delivery teams produce solutions: a portion of which may be software, a portion of which may be hardware, and a portion of which will be the manner in which the system is used.
- Cost effective and timely manner. Agile teams prefer to implement functionality in priority order [http://www.agilemodeling.com/essays/prioritizedRequirements.htm], with the priority being defined by their stakeholders (or a representative thereof). Working in priority order enables agile teams to maximize the return on investment (ROI) because they are working on the high-value functionality as defined by their stakeholders, thereby increasing cost effectiveness. Agile teams also prefer to produce potentially shippable solutions each iteration (an iteration is a time-box, typically 2-4 weeks in length), enabling their stakeholders to determine when they wish to have a release delivered to them and thereby improving timeliness. Short iterations reduce the feedback cycle, improving the chance that agile teams will discover problems early (they “fail fast”) and thereby enable them to address the problems when they’re still reasonably inexpensive to do so. The DDJ 2008 Project Success survey found that agile teams are in fact more likely to deliver good ROI than traditional teams and more likely to deliver in a timely manner.
- Value driven life cycle. One result of building a potentially shippable solution every iteration is that agile teams produce concrete value in a consistent and visible manner throughout the life cycle.
- Risk and value driven life cycle. Core agile processes are very clear about the need to produce visible value in the form of working software on a regular basis throughout the life cycle. Disciplined agile delivery processes take it one step further and actively mitigate risk early in the life cycle – during project start up you should come to stakeholder concurrence regarding the project’s scope, thereby reducing significant business risk, and prove the architecture by building a working skeleton of your system, thereby significantly reducing technical risk. They also help with transition to agile, allowing traditional funding models to use these milestones before moving to the finer grained iteration based funding that agile allows.
- Highly collaborative. People build systems, and the primary determinant of success on a development project is the individuals and the way that they work together. Agile teams strive to work closely together and effectively as possible. This is a characteristic that applies to both engineers on the team, as well as their leadership.
- Disciplined. Agile software development requires greater discipline on the part of practitioners that what is typically required by traditional approaches.
- Self organizing. This means that the people who do the work also plan and estimate the work.
- Self-organization within an appropriate governance framework. Self-organization leads to more realistic plans and estimates which are more acceptable to the people implementing them. At the same time these self-organizing teams must work within an appropriate governance framework which reflects the needs of their overall organizational environment. An “appropriate governance framework” explicitly enables disciplined agile delivery teams to effectively leverage a common infrastructure, to follow organizational conventions, and to work towards organizational goals. The point is that project teams, regardless of the delivery paradigm they are following, need to work within the governance framework of their organization. More importantly, effective governance programs should make it desirable to do so. Our experience is that traditional, command-and-control approaches to governance where senior management explicitly tells teams what to do and how to do it don’t work very well with agile delivery teams. We’ve also found that lean development governance, an approach which is based on collaboration and enablement, is far more effective in practice. Good governance increases the chance that agile delivery teams will build systems which fit into your overall organizational environment, instead of yet another stand-alone system which increases your overall maintenance burden and data quality problems.
- Active stakeholder participation. Agile teams work closely with their stakeholders, who include end users, managers of end users, the people paying for the project, enterprise architects, support staff, operations stuff, and many more. Within IBM we distinguish between four categories of stakeholder: principles/sponsors, partners (business partners and others), end users, and insiders These stakeholders, or their representatives (product owners in Scrum, or on-site customers in Extreme Programming, or a resident stakeholder in scaling situations), are expected to provide information and make decisions in a timely manner.
- Changing needs of stakeholders. As a project progresses your stakeholders will gain a better understanding of what they want, particularly if you’re showing them working software on a regular basis, and will change their “requirements” as a result. Changes in the business environment, or changes in organization priority, will also motivate changes to the requirements. There is a clear need for agile requirements change management [http://www.agilemodeling.com/essays/changeManagement.htm] on modern IT projects.
- Repeatable results. Stakeholders are rarely interested in how you delivered a solution but instead in what you delivered. In particular, they are often interested in having a solution which meets their actual needs, in spending their money wisely, in a high-quality solution, and in something which is delivered in a timely manner. In other words, they’re interested in repeatable results, not repeatable processes.
- Right amount of ceremony for the situation. Agile approaches minimize ceremony in favor of delivering concrete value in the form of working software, but that doesn’t mean they do away with ceremony completely. Agile teams will still hold reviews, when it makes sense to do so. DDJ’s 2008 Modeling and Documentation Survey found that agile teams will still produce deliverable documentation, such as operations manuals and user manuals, and furthermore are just as likely to do so as traditional teams. The DDJ September 2009 State of the IT Union survey found that the quality of the documentation delivered by agile teams was just as good as that delivered by traditional teams, although iterative teams (e.g. RUP teams) did better than both agile and traditional.
At Agile 2009 in August Sue McKinney, VP of Development Transformation with IBM Software Group, was interviewed by DZone's Nitin Bharti about IBM's experiences adopting agile techniques. There are over 25,000 developers within IBM Software Group alone. Follow the link to the interview
to view it online (there is also a text transcript posted there. There's some great insights into the realities of scaling agile in large teams, in distributed agile development, and in particular how to transform a large organization's development staff.