Modified by ScottAmbler
IBM Rational recently published an update to my Agility@Scale e-book, which can be downloaded free of charge. The e-book is a 21 page, 2.3 meg PDF (sorry about the size, guess the graphics did it) . It overviews the Agile Scaling Model (ASM) (which has since been replaced by the Software Development Context Framework (SDCF) ), Disciplined Agile Delivery (DAD), the scaling factors of agility at scale, and ends with some advice for becoming as agile as you can be. In short it's a light-weight coverage of some of the things I've been writing about in this blog the past couple of years. Could be a good thing to share with the decision makers in your organization if they're considering adoption agile strategies.
Modified by ScottAmbler
One of the scaling factors called out in the Software Development Context Framework (SDCF) is domain complexity. The general idea is that agile teams will find themselves in different situations where some teams are developing fairly straightforward solutions, such as an informational website, whereas others are addressing very complex domains, such as building an air-traffic control system (ATCS). Clearly the team building an ATCS will work in a more sophisticated manner than the one building an informational website. I don't know whether agile techniques have been applied in the development of an ATCS, although I have to think that agile's greater focus on quality and working collaboratively with stakeholders would be very attractive to ATCS delivery teams, I do know that agile is being applied in other complex environments: The 2009 Agility at Scale Survey found that 18% of respondents indicated that their organizations had success at what they perceived to be very complex problem domains,.
Increased domain complexity may affect your strategy in the following ways:
Reaching initial stakeholder consensus becomes difficult. One of the risk reduction techniques called out in Disciplined Agile Delivery (DAD) is to come to (sufficient) stakeholder consensus at the beginning of the project during the Inception phase (called Sprint 0 in Scrum or Iteration 0 in other agile methods). Stakeholder consensus, or perhaps "near concensus" or "reasonable agreement" are better terms, can be difficult to come to the more complex the problem domain is because the stakeholders may not fully understand the implications of what they're making decisions about and because there is likely a greater range of stakeholders with differing goals and opinions. The implication is that your project initiation efforts may stretch out, increasing the chance that you'll fall back on the old habits of big requirements up front (BRUF) and incur the costs and risks associated with doing so.
Increased prototyping during inception. It is very common for disciplined agile teams to do some light-weight requirements envisioning during inception to identify the scope of what they're doing and to help come to stakeholder consensus. The greater the complexity of the domain, and particularly the less your team understands about the domain, the more likely it is that you'll benefit from doing some user interface (UI) prototyping to explore the requirements. UI prototyping is an important requirements exploration technique regardless of paradigm, and it is something that you should consider doing during both initial requirements envisioning as well as throughout the lifecycle to explore detailed issues on a just in time (JIT) manner.
Holding "all-hands reviews". One strategy for getting feedback from a wide range of people is to hold an "all hands review" where you invite a large group of people who aren't working on a regular basis with your team to review your work to date. This should be done occasionally throughout the project to validate that the input that you're getting from your stakeholder represenatives/product owners truly reflects the needs of the stakeholders which they represent. The 2010 How Agile Are You? Survey found that 42% of "agile teams" reported running such reviews.
Increased requirements exploration. Simple modeling techniques work for simple domains. Complex domains call for more complex strategies for exploring requirements. The implication is that you may want to move to usage scenarios or use cases from the simpler format of user stories to capture critical nuances more effectively. A common misunderstanding about agile is that you have to take a "user story driven approach" to development. This is an effective strategy in many situations, but it isn't a requirement for being agile.
The use of simulation. You may want to take your prototyping efforts one step further and simulate the solution. This can be done via concrete, functional prototypes, via simulation software, via play acting, or other strategies.
Addition of agile business analysts to the team. Analysis is so important to agile teams we do it every day. In situations where the domain is complex, or at least portions of the domain is complex, it can make sense to have someone who specializes in exploring the domain so as to increase the chance that your team gets it right. This is what an agile business analyst can do. There are a few caveats. First, even though the domain is complex you should still keep your agile analysis efforts as light, collaborative, and evolutionary as possible. Second, this isn't a reason to organize your team as a collection of specialists and thereby increase overall risk to your project. The agile analyst may be brought on because their specialized skills are required, but the majority of the people on the team should still strive to be generalizing specialists. This is also true of the agile analyst because their may not be eight hours a day of valuable business analysis work on the team, and you don't want the BA filling in their time with needless busy work.
The important thing is to recognize that the strategies which work well when you're dealing with a simple domain will not work well for a complex domain. Conversely, techniques oriented towards exploring complex domains will often be overkill for simple domains. Process and tooling flexiblity is key to your success.
I was recently in Bangalore speaking at the Rational Software Conference, which was really well done this year, and visiting customers. In addition to discussing how to scale agile software development approaches, particularly when the team is distributed geographically and organizationally, I was also asked about what I thought about a software factory approach to development. My instinctual reaction was negative, software factories can result in lower overall productivity as the result of over specialization of staff (I prefer generalizing specialists
), too many hand-offs between these specialists (I find close collaboration to be far more effective), and too much bureaucratic overhead to coordinate these activities. I initially chalked it up to these people still believing that software development was mostly a science, or perhaps an engineering domain, whereas my experiences had made me come to believe that software development is really more art than it is a science. Yet, the consistent belief in this strategy by very smart and experienced people started me thinking about my position.
Just let me begin by saying that this blog posting isn't meant to be yet another round in the age old, and relatively inane, "art vs. science" debate within the software development community. That debate is a symptom of versusitis
, a dread disease which particularly plagues the IT industry and which can any of us at any time. There is no known cure, although the combination of experience, open-mindedness, and critical thought are the best inoculation against versusitis that we have so far. In that vein, let me explore the issues as I see them and I will let you think for yourself.
On the one hand software development has aspects of being an art for several reasons. First, the problem definition is never precise, nor accurate, and even when we have detailed specifications the requirements invariably evolve
anyway. The lack of defined, firm requirements requires us to be flexible and to adjust to the situation that we find ourselves in. Second, teams typically find themselves in unique situations, necessitating a unique process and tool environment to reflect this (assuming that you want to be effective, otherwise there's nothing stopping you from having a "repeatable process" and consistent tool environment). Third, software is built by people for people, requiring that the development team have the ability to build a system with a user interface which meets the unique needs of their end users. One has only to look at the myriad UI designs out there to see that surely there is a bit of art going on. Fourth, if software development wasn't at least partially art then why hasn't anyone succeeded at building tools which take requirements as inputs and produce a viable solution that we can easily deploy? It's been over four decades now, so there's been sufficient time and resources available to build such tooling. Fifth, regardless of how much of a scientific/business facade we put over it, our success rate at producing up front detailed cost estimates and schedules speak for itself (see Funding Agile Projects
for links to articles).
On the other hand software development has aspects of being a science for several reasons. First, some aspects of software development have in fact been automated to a significant extent. Second, there is some mathematical basis to certain aspects of software development (although in the case of data-oriented activities the importance of relational theory
often gets blown way out of proportion and I have yet to see a situation where formal methods proved to be of practical value).
What does this have to do with Agility@Scale. As you know, one of the agile scaling factors
is Organizational Complexity, and cultural issues are the hardest to overcome. Whether your organization believes that software development is mostly an art or mostly a science is a cultural issue which will be a major driver in you choice of methods and practices. Organizations which believe that software development is more of a science will prefer strategies such as software factories, model-driven architecture (MDA),
and master data management (MDM)
. And there is ample evidence to support the claims that some organizations are succeeding at these strategies. Although you may not agree with these strategies, you need to respect the fact that many organizations are making them work in their environments. Similarly, organizations which believe that software development is more of an art will find that agile and lean strategies are a better fit, and once again there is ample evidence that organizations are succeeding with these approaches (there's also evidence that agile projects are more successful
than traditional projects, on average). Once again, you may not agree with these strategies but you need to respect the fact that other people are making them work in practice.
Trying to apply agile approaches within an organization that believes software development is mostly a science will find it difficult at best, and will likely need to embark on a multi-year program to shift their culture (likely an expensive endeavor which won't be worth the investment). Similarly, trying to apply a software factory strategy in an organization that believes that software development is mostly an art will also run aground. The bottom line is that one size does not fit all, that one strategy is
not right for all situations and that you need to understand the trade-offs of various strategies, methodologies, techniques, and practices and apply them appropriately given the situation that you face. In other words, it depends! If you are embarking on a software process initiative, and you don't have the broad experience required to effective choose between strategies (very few organizations do, although many believe otherwise), then you should consider Measured Capability Improvement Framework (MCIF)
to help increase your chance of success.
Modified by ScottAmbler
A fair question to ask is why should your organization consider adopting the Disciplined Agile Delivery (DAD) process framework. I believe that there are several clear benefits to doing so:
DAD shows how agile techniques fit together. DAD is a hybrid that adopts strategies from a variety of sources, including Scrum, Extreme Programming (XP), Agile Modeling, Kanban, Outside In Development (OID) and many more. More importantly DAD's process-goal driven approach shows how this all fits together, providing advice for when (and when not) to use each technique and the advantages and disadvantages of doing so. In doing so DAD enables you to streamline your efforts to tailor agile to reflect the context of the situation you find yourself in. Furthermore, it provides this advice in the context of a full agile delivery lifecycle, explicitly showing how to initiate a project, construct a solution, and then deploy into production. Instead of starting with a small agile method such as Scrum and doing all the work to figure out how to tailor ideas from other methods to actually make it work, why not start with a framework that's already done all that work for you?
DAD isn't prescriptive. DAD is far less prescriptive than other agile methods. For example, where Scrum prescribes a single strategy for managing changing requirements, a strategy called a Product Backlog, DAD suggests several strategies and provides advice for choosing the right one. Where other agile methods define a single lifecycle, DAD instead describes several lifecycles (an agile Scrum-based one, a lean lifecycle, and a continuous delivery lifecycle to name just three) and once again describes how to choose the right one for your situation. Your agile team is in a unique situation, and as a result needs a flexible process framework that provides coherent, easy-to-follow tailoring advice. Isn't it better to adopt strategies that reflect the context that you actually face?
DAD explicitly addresses architecture. Even after a decade of agile software development it still seems that the topic of how agile teams address architecture is a mystery for many people. As a result DAD builds agile architecture strategies right in, starting with initial architecture envisioning, to proving the architecture with working code, to evolutionary design strategies during construction.
DAD explicitly addresses DevOps. DevOps is the art of combining development and operations approaches in such a way as to streamline your overall efforts. DAD "bakes in" DevOps through explicit support for common DevOps practices as well as its robust stakeholder definition which explicitly includes operations and support people.
DAD explicitly addresses governance. Although governance is often considered a dirty word within some agile circles, the reality is that software development teams can and should be governed. Sadly, many agile teams have traditional governance strategies inflicted upon them, strategies which invariably increase schedule, cost and risk on the project. But is doesn't have to be this way. It is possible, and very desirable to adopt agile goverance strategies, strategies which are built right into the DAD framework.
DAD explicitly addresses many other important development activities. Architecture, DevOps, and governance are just the tip of the iceberg. DAD also shows how critical activities such as analysis, design, testing, quality, technical writing, and many more are addressed in an agile and streamlined manner throughout the delivery lifecycle. As described earlier, this is done in a non-prescriptive and tailorable manner, thereby removing a lot of the mystery regarding how this agile stuff all fits together into a coherent whole.
DAD is solution focused, not software focused. Although the rhetoric around "potentially shippable software" resonates well with developers it observably and empirically misses the mark. DAD promotes the more robust idea of "potentially consumable solutions". Yes, shipping is nice but shipping something that people actually want to use/buy, something that is consumable, is much nicer. Yes, software is part of the equation but that software runs on hardware, we often also need to develop supporting documentation, we often evolve the business process, and even evolve the organization structure around the usage of the system. In other words, we're really producing solutions, not just software. Isn't it better to adopt rhetoric that actually reflects what we're doing in practice?
DAD promotes enterprise awareness over team awareness. One of the great benefits of an agile approach to software development is its focus on the team. This can also be a bit of a problem, because a team-focused strategy can result in suboptimal decisions for your overall organization. DAD promotes the idea that disciplined agilists should be enterprise aware, working towards common business and technical goals while leveraging and enhancing the existing infrastructure whenever possible.
DAD provides a foundation from which to scale. The starting point for scaling agile is to understand how agile strategies fit together from project initiation to delivery into production. If you don't know how to succeed with agile in a straightforward situation then it will prove very difficult to do so in an agility @ scale situation. DAD not only provides this tailorable foundation from which to scale agile it also takes a robust view of what it means to scale agile (hint: large or distributed teams are only a start).
DAD provides a basis for enterprise agile. As organizations move towards a true "enterprise agile" approach they often find that they need to adopt either DAD as a foundation or invest a fair bit of effort inventing something similar. They are also starting to adopt strategies from the SAFe framework, or reinventing such, as well as ideas from sources such as Enterprise Unified Process (EUP) (sadly, poorly named in hindsight), ITIL, and even CoBIT. More on this in a future blog posting.
In short, DAD provides a lot of proven advice culled from years of experience applying agile software techniques in enterprise-class environments. Instead of figuring all of this stuff out on your own, why not jump ahead and leverage the hard-won lessons learned from other organizations that have already dealt with the challenges that you're struggling with today?
The primary shortcoming of the DAD framework is it makes it very clear that software development, oops I mean solution delivery, is quite complex in practice. As IT practitioners we inherently know this, but it seems that we need to be reminded of this fact every so often. DAD doesn't provide a simplistic, feel-good strategy that you can learn in a few hours of training. Instead it defines a coherent, tailorable strategy that reflects the realities of enterprise IT.
There is a wealth of information at DAD posted at the Disciplined Agile Delivery (DAD) web site and great discussions occuring on the DAD LinkedIn discussion forum. For those of you interested in agile certification, the Disciplined Agile Consortium site will prove valuable too, in particular the list of upcoming DAD workshops provided by several IBM partners. And of course the book Disciplined Agile Delivery: A Practitioner's Guide to Agile Software Delivery in the Enterprise (IBM Press, 2012) written by Mark Lines and myself is a very good read.
activities are evolutionary (iterative and incremental) and highly collaborative in nature. Initially requirements are explored at a high level via requirements envisioning
at the beginning of the project and the details are explored on a just-in-time (JIT) basis via iteration modeling
and model storming
activities. The way that you perform these agile practices, and the extent to which you do so, depends on the situation in which a project team finds itself. The Agile Scaling Model (ASM)
is a contextual framework for effective adoption and tailoring of agile practices to meet the unique challenges faced by a system delivery team of any size. To see how this works, let's apply the concepts of the ASM to see how we would scale our agile approach to requirements.
First, let's consider how a small, co-located team would work. The first two categories of the ASM are core agile development and disciplined agile delivery
, the focus of both are small co-located teams in a fairly straightforward situation. In these situations simple techniques such as user stories
written on index cards and sketches on whiteboards
work very well, so the best advice that I can give is to stick with them. Some teams will take a test-driven development
(TDD) approach where they capture their requirements and design in the form of executable specifications
, although this sort of strategy isn't as common as it should be (yet!), likely because of the greater skill and discipline that it requires. Traditionalists often balk at this approach, believing that they need to document the requirements in some manner. But, for a small co-located team working in a collaborative manner, requirements documentation proves to be little more than busy work, often doing nothing more than justifying the existence of a business analyst who hasn't made the jump to agile yet. Don't get me wrong, there are good reasons to write some requirements documentation, and we'll see this in a minute, but you should always question any request for written specifications and try to find more effective ways to address the actual goal(s) motivating the request. Never forget that written documentation
is the least effective communication
option available to you.
Although inclusive tools
such as whiteboards and paper work well for requirements, for development activities you will need electronic tools. You will either put together an environment from point-specific tools or adopt something more sophisticated such as IBM Rational Team Concert (RTC)
which is already fully integrated and instrumented. RTC is a commercial tool, but luckily you can download a 10-license environment free of charge, which is just perfect for a small team. Larger teams, of course, will need to purchase licenses. One of the things that a disciplined agile delivery approach adds to core agile development is it addresses the full delivery life cycle, which is important because it explicitly includes pre-construction activities such as requirements envisioning. The first step in scaling agile techniques is to adopt a full delivery life cycle which covers the full range of activities required to initiate a project, produce the solution, and then release to solution to your end users.
More interesting is the third category of the ASM, Agility@Scale, and how its eight agile scaling factors
affect the way that you tailor your process and tooling strategy. Let's explore how each one could potentially affect your agile requirements strategy:
- Geographical distribution. The majority of agile teams are distributed in some manner -- some people are working in cubicles or private offices, on different floors, in different buildings, or even in different countries -- and when this happens your communication and coordination risks goes up. To counter this risk you will need to perform a bit more requirements envisioning up front to help ensure that everyone is working to the same vision, although this doesn't imply that you need to write detailed requirements speculations which would dramatically increase the risk to your project. Remember, agilists do just barely enough modeling and are prepared to iteratively elicit the details when they need to do so. The more distributed the team is the more likely they will need to adopt software-based requirements modeling tools such as IBM Rational Requirements Composer (RRC) which supports streamlined, agile requirements elicitation throughout the delivery life cycle. Index cards and whiteboards are great, but they're difficult to see if you're outside the room where they're posted. I've written a fair bit about distributed agile development in this blog.
- Team size. Some organizations, including IBM, are successfully applying agile techniques with teams of hundreds of people. A team of one hundred people will naturally work much differently than a team of ten people, or of one thousand people. Large teams are organized into collections of smaller teams, and the requirements for the overall project must be divvied up somehow between those teams. The implications are that as the team size grows you will need to invest a bit more time in initial requirements envisioning, and in initial architecture envisioning for that matter; you will need to use more sophisticated tools; and may need to use more sophisticated modeling techniques such as use cases and functional user interface prototypes. See large agile teams for more advice.
- Compliance requirement. When regulatory issues – such as Sarbanes Oxley, ISO 9000, or FDA CFR 21 – are applicable you are likely going to be required to capture requirements specifications in some manner and to enact traceability between those requirements. However, I highly recommend that you read the actual regulations yourself and don't let bureaucrats interpret them for you (doesn't it always seem that their interpretation always results in an onerous, documentation heavy solution?) because I have yet to run into a regulation which required you to work in an ineffective manner. Managing your requirements as work items in RTC can often more than meet your regulatory requirements for documentation and traceability, although you may want to consider a tool such as IBM Rational RequisitePro for complex regulatory situations.
- Domain complexity. The manner in which you elicit requirements for a data entry application or an informational web site will likely be much simpler than for a bio-chemical process monitoring or air traffic control system. More complex domains will require greater emphasis on exploration and experimentation, including but not limited to prototyping, modeling, and simulation. Although user stories may be effective as a primary requirements artifact in simple domains, in more complex domains you are likely to find that you need to drive your requirements effort with more sophisticated modeling techniques.
- Organization distribution. Sometimes a project team includes members from different divisions, different partner companies, or from external services firms. In these cases, particularly where the work is strictly organized between the various organizations (perhaps for security concerns), you may need a more sophisticated approach to managing the requirements. RTC enables you to organize the requirements between teams, and then to automatically track progress in real time via the RTC project dashboard.
- Technical complexity. The technical complexity of a solution can vary widely, from a single platform silo application to a multi-platform application working with legacy systems and data to a full-blown systems engineering effort. Complex technical domains, just like complex business domains, require more complex strategies for requirements elicitation and management. The requirements for your legacy systems are likely to have been captured using tools and techniques appropriate for that platform, for example the requirements for your COBOL application may have been captured using data flow diagrams and data models, whereas the requirements for your Java legacy application where captured using UML diagrams. The subteam working on the COBOL system might be using IBM Rational Application Developer (RAD) and RTC for Z whereas the Java subteam may use Eclipse with RTC. Because systems engineering projects can stretch on for years, particularly when the hardware is being developed in parallel to the software, sophisticated tooling such as IBM Rational DOORS is often used in these situations. For more information about systems engineering, see the IBM Rational Harmony process.
- Organizational complexity. Your approach to requirements elicitation and management will be affected by a host of organizational complexities, including your corporate culture. When the culture is flexible and collaborative you can be very agile in your approach to requirements, but as it becomes more rigid you become more constrained in what is considered acceptable and thus take on greater project risk. For example, many organizations still struggle with their approach to funding projects, often demanding that the project team provides an "accurate" estimate up front to which they will be held to. This in turn motivates risky behavior on the part of the development, including a "big requirements up front (BRUF)" approach where a detailed requirements speculation is developed early in the project. This is just one example of how questionable corporate culture can impact the way in which an agile team works.
- Enterprise discipline. Some organizations have enterprise-level disciplines, such as enterprise architecture, enterprise business modeling, strategic reuse, and portfolio management in place. These disciplines can easily be agile and from what I can tell the more successful efforts appear to lean more towards the agile end of the spectrum rather than the traditional end. Having an enterprise business modeling effort underway will affect your project-level requirements strategy -- you'll be able to leverage existing models, have access to people who understand the domain at an enterprise level, and will likely need to map your project efforts back to your enterprise models. The enterprise modelers will likely be using tools such as IBM Rational System Architect or IBM Websphere Business Modeler.
It is important to note that the way that you tailor the agile practices that you follow, and the tools that you use, will reflect the situation that you find yourself in. In other words, you need to right size your process and the Agile Scaling Model (ASM) provides the context to help you do so. As you saw above, in simpler situations you will use the simpler tools and techniques which are commonly promoted within the core agile development community. But, when things become a bit more complex and one or more of the scaling factors applies you need to modify your approach -- just don't forget that you should strive to be as agile as you can be given the situation that you find yourself in.
During the second week of August the Agile 2011 conference was held in Salt Lake City (SLC). As you likely know the Agile Manifesto was formulated 10 years ago in SLC so it was apropos to hold it there. There was some excitement around the 10 year anniversary of the manifesto, with a panel session with the 17 authors of it. Sadly there seemed to be little excitement around the efforts of the 10th anniversay agile workshop
in February which proposed a potential path forward for the agile community. I found the conference to be an evolutionary improvement over the conferences of the past few years, which is a very good thing because the focus since 2008 has moved beyond the "cool" new programming techniques to include the issues that enterprises face.
Starting at the Agile 2008 conference I've seen an uptick in interest in what I would consider some of the more mature topics in agile development, although I'm unfortunately still seeing significant confusion out there too, in part due to over-exuberence of people new to agile. For example, there's people still asking about basic issues about agile architecture
and agile database
techniques, although I was really happy to see more coherent discussions around scaling agile
. My own presentation about the Agile Scaling Model
was well attended and I suspect I opened a few people's eyes regarding the realities that we face (yes, there's a lot more to it than holding a "scrum of scrums", yeesh). We have a long way to go until people really start to understand scaling issues, but we're clearly on the path to getting there.
The conference show floor was interesting, with a wide range of vendors offering services and products focused on agile and lean. One thing that I noticed was many vendors had large monitors showing off their ability to support lean task boards, which for the most part they all looked the same. At the IBM booth we were showing off some of the Jazz tools
, in particular Rational Team Concert (RTC)
. For a long time now we've been giving away fully functional, with no time limit, licenses of RTC for teams of up to 10 people. Something worth checking out.
The Agile 201x conferences hosted by the Agile Alliance are always a good investment of your time and money, and Agile 2011 was no exception. See you at Agile 2012 in the great state of Texas!
The explicit phases of the Unified Process -- Inception, Elaboration, Construction, and Transition -- and their milestones are important strategies for scaling agile software development to meet the real-world needs of modern organizations. Yes, I realize that this is heresy for hard-core agilists who can expound upon the evils of serial development, yet these very same people also take a phased approach to development although are loathe to admit it. The issue is that the UP phases are like seasons of a project: although you'll do the same types of activities all throughout a project, the extent to which you do them and the way in which you do them change depending on your goals. For example, at the beginning of a development project if you want to be effective you need to do basic things like identify the scope of the project, identify a viable architecture strategy, start putting together your team, and obtain support for the project. Towards the end of a project your focus is on the activities surrounding the deployment of your system into production, including end-of-lifecycle testing efforts, training, cleaning up of documentation, piloting the system with a subset of users, and so on. In between you focus on building the system, including analysis, design, testing, and coding of it. Your project clearly progresses through different phases, or call them seasons if the term phase doesn't suit you, whether your team is agile or not.
The UP defines four phases, each of which address a different kind of risk:1. Inception. This phase focuses on addressing business risk by having you drive to scope concurrence amongst your stakeholders. Most projects have a wide range of stakeholdres, and if they don't agree to the scope of the project and recognize that others have conflicting or higher priority needs you project risks getting mired in political infighting. In the Eclipse Way this is called the "Warm Up" iteration and in other agile processes "Iteration 0".2. Elaboration. The goal of this phase is to address technical risk by proving the architecture through code. You do this by building and end-to-end skeleton of your system which implements the highest-risk requirements. Some people will say that this approach isn't agile, that your stakeholders should by the only ones to prioritize requirements. Yes, I agree with that, but I also recognize that there are a wide range of stakeholders, including operations people and enterprise architects who are interested in the technical viability of your approach. I've also noticed that the high-risk requirements are often the high-business-value ones anyway, so you usually need to do very little reorganization of your requirements stack.3. Construction. This phase focuses on implementation risk, addressing it through the creation of working software each iteration. This phase is where you put the flesh onto the skeleton.4. Transition. The goal of this phase is to address deployment risk. There is usually a lot more to deploying software than simply copying a few files onto a server, as I indicated above. Deployment is often a complex and difficult task, one which you often need good guidance to succeed at.
Each phase ends with a milestone review, which could be as simple as a short meeting, where you meet with prime stakeholders who will make a "go/no-go" decision regarding your project. They should consider whether the project still makes sense, perhaps the situation has changed, and that you're addressing the project risks appropriately. This is important for "agile in the small" but also for "agile in the large" because at scale your risks are often much greater. Your prime stakeholders should also verify that you have in fact met the criteria for exiting the phase. For example, if you don't have an end-to-end working skeleton of your system then you're not ready to enter the Construction phase. Holding these sorts of milestone reviews improves your IT governance efforts by giving senior management valuable visibility at the level that they actually need: when you have dozens or hundreds of projects underway, you can't attend all of the daily stand up meetings of each team, nor do you even want to read summary status reports.
These milestone reviews enable you to lower project risk. Last Autumn I ran a survey via Dr. Dobb's Journal (www.ddj.com) which explore how people actually define success for IT projects and how successful we really were. We found that when people define success in their own terms that Agile has a 71% success rate compared with 63% for traditional approaches. Although it's nice to that Agile appears to be lower risk than traditional approaches, a 71% success rate still implies a 29% failure rate. The point is that it behooves us to actively monitor development projects to determine if they're on track, and if not either help them to get back on track or cancel them as soon as we possibly can. Hence the importance of occasional milestone reviews where you make go/no-go decisions. If you're interested in the details behind the project, they can be found at http://www.ambysoft.com/surveys/success2007.html .
Done right, phases are critical to your project success, particularly at scale. Yes, the traditional community seems to have gone overboard with phase-based approaches, but that doesn't mean that we need to make the same mistakes. Let's keep the benefit without the cost of needless bureaucracy.[Read More
A recurring discussion that I have with experienced agile developers is what it means to take a disciplined agile approach. The conversation usually starts off by some saying "but it already requires discipline to do agile", something that I fully agree with, followed by "therefore 'disciplined agile' is merely a marketing term", something which I don't agree with. The challenge with the "standard" agile discipline is that it is often focused on construction activities within a single project team, clearly important but also clearly not the full picture. There's more to an agile project than construction, and there's more to most IT departments than a single development project. In short, there are many opportunities for IT professionals to up their discipline, and thereby up their effectiveness, opportunities which we make explicit in the Disciplined Agile Delivery (DAD) framework.
Let's explore the many aspects to taking a disciplined agile approach:
You adopt "standard" agile discipline
. Aspects of agile which require discipline
include adopting practices such as test-driven development (TDD), active stakeholder participation, working collaboratively, shortening the feedback cycle
, and many more. These strategies are a great start to becoming disciplined IT professionals.
You take a goal-driven approach
. When we first started working on the DAD framework I didn't want to create yet another prescriptive framework, particularly given Rational's track record with the Rational Unified Process (RUP) framework. Rational has been pilloried for years for the prescriptive nature of RUP, which is unfortunate because there are a lot of great ideas in RUP that agile teams can benefit from, some of which we adopted in DAD and many of which are being actively reinvented with the agile community even as you read this. Furthermore, there are many prescriptive elements of the Scrum method that can get teams in trouble. For example, Scrum prescribes that you hold a daily stand up meeting, often called a Scrum meeting, where everyone should answer three questions. That's a great approach for teams new to agile, but it proves problematic in many situations due to it's prescriptive nature. Do you really need to do this once a day? I've been on teams where we held coordination meetings twice a day and others only once a week. Do you really need to stand up? I've been on geographically distrubited agile teams where many of us were sitting down during coordination calls. Do you really need to answer three questions, two of which are clearly focused on status regardless of claims otherwise? I've been on lean teams where we met around our Kanban board and focused on potential blockers. The answers to these questions depends on the context of the situation you find yourself in. The challenge, at least from the point of view of a process framework, is how do you avoid falling into the trap of being overly prescriptive. The strategy we adopted in DAD is to take a goal-driven approach. The observation is that regardless of the situation you find yourself in there are common goals your team will need to fulfill. For example, at the beginning of a project common goals include developing an initial plan, initially exploring the scope, initially identifying a technical strategy, and securing initial funding (amongst others). Throughout construction you should coordinate your activities, improve the quality of your ecosystem, and produce a potentially consumable solution on a regular basis (more on this below). So, instead of prescribing a daily stand up meeting the DAD framework instead indicates you should coordinate your activities, and gives several options for doing so (one of which is a Scrum meeting). More importantly DAD describes the advantages and disadvantages of your options so that you can make the choice that's best suited for the situation your team finds itself in (see this blog posting
for a detailed example of the types of tables included in the DAD book to help you through such process tailoring decisions). In short, our experience is that it requires discipline to take a goal driven approach
to agile delivery over the prescriptive strategies in other agile processes.
You take a context-driven approach
. There are many tailoring factors, which I describe in the Software Development Context Framework (SDCF)
, which you need to consider when making process, tooling, and team structure decisions. For example, a large team will adopt a different collection of practices and tools than a small team. A geographically distributed team will adopt a different strategy than a team that is co-located. You get the idea. Other tailoring factors include compliance, team culture, organization culture, technical complexity, domain complexity, and project type. It requires discipline to recognize the context of the situation you find yourself in and then act accordingly.
You deliver potentially consumable solutions
. One of the observations that we made early in the development of the DAD framework was that disciplined agile teams produce potentially consumable solutions, not just potentially shippable software. Although delivery of high-quality, working software is important it is even more important that we deliver high-quality working solutions to our stakeholders. For example, not only are we writing software but we may also be updating the hardware on which it runs, writing supporting documentation, evolving the business processes around the usage of the system, and even evolving the organizational structure of the people working with the system. In other words, disciplined agilists focus on solutions over software
. Furthermore, "potentially shippable" isn't sufficient: not only should it be shippable but it should also be usable and should be something people want to use. In other words it should be consumable (a concept DAD adopted from IBM's Outside In Development
). Minimally IT professionals should have the skills and desire to produce good software, but what they really need are the skills and desire to provide good solutions. We need strong technical skills, but we also need strong "
such as user interface design and process design to name just two.
The incremental delivery of potentially consumable solutions on an incremental basis requires discipline
to do successfully. DAD teams focus on repeatable results not repeatable processes
You are enterprise aware
. Whether you like it or not, as you adopt agile you will constrained by the organizational ecosystem, and you will need to act accordingly. It takes discipline to work with enterprise professionals such as enterprise architects, data admistrators, portfolio managers, or IT governance people who may not be completely agile yet, and have the patience to help them. It takes discipline to work with your operations and support staff in a DevOps
manner throughout the lifecycle, particularly when they may not be motivated to do so. It requires discipline to accept and potentially enhance existing corporate development conventions (programming guidelines, data guidelines, UI guidelines, ...). It requires discipline to accept that your organization has an existing technology roadmap that you should be leveraging, building out, and in some cases improving as you go. In short, enterprise awareness requires a level of discipline
not typically seen on many agile teams.
You adopt a full delivery lifecycle
. Empirically it is very easy to observe that at the beginning of an agile project there are some activities that you need to perform to initiate the project. Similarly at the end of the project there are activities that you need to perform to release the solution into production or the marketplace. The DAD process framework addresses the effort required for the full delivery effort, including project initiation, construction, and deployment. Our experience is that it requires discipline on the part of IT professionals to include explicit phases
for Inception/Initation, Construction, and Transition/Deployment and more importantly to focus the appropriate amount of effort on each. One danger of explicit phases is that you run the risk of taking what's known as a Water-Scrum-Fall
approach, a term coined by Dave West the person who wrote the forward for the DAD book, where you take an overly heavy/traditional approach to inception and transition in combination with a lighter agile approach to construction. Water-Scrum-Fall occurs because many organizations haven't made a full transition to agile, often because they think it's only applicable to construction. Our experience is that you can be very agile in your approach to inception and transition, experience we've built into the DAD framework. Having said that it clearly requires discipline to keep inception activities short
and similarly it requires discipline to reduce the "transition phase" to an activity
You adopt a wider range of roles
. An interesting side effect of adopting a full delivery lifecycle is that you also need to adopt a more robust set of roles. For example, the Scrum method suggests three roles - Scrum Master, Product Owner, and Team Member - a reflection of the Scrum lifecycle's construction focus. DAD suggests three primary roles - Team Lead, Product Owner, Team Member, Architecture Owner
, and Stakeholder - as well as five secondary roles which may appear at scale.
You embrace agile governance
. Governance establishes chains of responsibility, authority and communication in support of the overall enterprise’s goals and strategy. It also establishes measurements, policies, standards and control mechanisms to enable people to carry out their roles and responsibilities effectively. You do this by balancing risk versus return on investment (ROI), setting in place effective processes and practices, defining the direction and goals for the department, and defining the roles that people play with and within the department. It requires discipline to adopt an agile approach to governance
, and that's something built right into the DAD framework.
- Does the team regularly produce value for their stakeholders?
- Does the team validate its own work to the best of its ability?
- Are stakeholders actively involved?
- Is the team self organizing?
- Does the team strive to improve their process?
Some interesting results include:
94% of teams which are claiming to be agile are providing value to stakeholders on a regular basis.
87% of teams which are claiming to be agile are validating their own work.
95% of teams which are claiming to be agile are working closely with stakeholders.
56% of teams which are claiming to be agile are self organizing.
88% of teams which are claiming to be agile are improving the process that they follow throughout the lifecycle.
Teams which are claiming to be agile often aren't. 53% of "agile teams" meet the five criteria, although 72% meet all but the self-organization criteria.
Teams which are moving towards agile but aren't there yet are reasonably close. 39% of those teams meet all five criteria and 63% meet all but self-organization.
I believe that there are several important implications:
- Whenever someone claims to be on an agile team you may want to explore that claim a bit deeper.
The low level of self organization may be an indicator of cultural challenges with organizations in that their project managers aren't giving up sufficient control. The Agility at Scale survey
in November 2009 found that 59% of respondents who indicated that their organization hadn't adopted agile techniques yet that a rigid culture was hampering their efforts. The IT Governance and Project Management
survey in July 2009 discovered that "questionable behaviors", many of which were ethically questionable (I'm being polite), were far too common within IT project management.
Although "agile teams" may not be as agile as they claim, they're still doing better than traditional V-model teams, as revealed (again) by the 2010 IT Project Success
If there was some sort of consensus within the agile community as to the criteria for determining whether a team is agile, I highly suspect that the agileness ratings would increase over time. What gets measured often improves.
However, how agile you are isn't anywhere near as important as getting better at what you're doing. So perhaps I'm barking up the wrong tree on this issue. ;-)
One of the scaling factors
of the Agile Scaling Model (ASM)
is technical complexity.
The fundamental observation is that the underlying technology of solutions varies and as a result your approach to developing a solution will also need to vary.
It’s fairly straightforward to achieve high-levels of quality if you’re building a new system from scratch on a known technology platform, but not so easy when there are several technologies, the technologies are not well known, or legacy assets are involved.
There are several potential technical complexities which a Disciplined Agile Delivery (DAD) team may face:
- New technology platforms. Your team may choose to work with a technology platform which is either new to the team or sometimes even new to the industry. In the past few years new technology platforms include the Android operating system, Apple’s iPad platform, and various cloud computing (http://www.ibm.com/ibm/cloud/) platforms. Working with these platforms may require you to adopt new development tools and techniques, not to mention the need to train and mentor your staff in their usage. Furthermore, your team may need to allocate time for architectural spikes to explore how to use the new technology and to prove the overall architecture with working code early in the project lifecycle (this is a DAD milestone).
- Multiple technology platforms. IT solutions often run on multiple platforms. For example, a system’s user interface (UI) could run in a browser, access business logic implemented using J2EE on Websphere which in turn invokes web services implemented in COBOL running on a Z-series mainframe, and stores data in an Oracle database, a DB2 database, and in several XML files. Implementing new business functionality, or updating existing functionality, could require changes made on several of these platforms in parallel. The implication is that you’ll need to adopt tools and strategies which enable your team to develop, test, and deploy functionality on all of these platforms. Testing and debugging in particular will become more difficult as the number of technology platforms increases, potentially requiring you to adopt the practice of parallel independent testing. The Agility at Scale survey found that 34% of respondents indicated that their agile teams were working with multiple technology platforms.
- Legacy data. IT solutions should leverage existing, legacy data wherever possible to reduce the number of data sources and thereby increase data quality within your organization. Also, using existing data sources can potentially speed up development, assuming your team has a good relationship with the owners of the legacy data sources (sadly, this often isn’t the case as the Data Management Survey found). Working with legacy data sources may require improved database regression testing, practices, database refactoring practices, and agile approaches to data administration. The Agility at Scale survey found that 42% of respondents indicated that their agile teams were working with legacy data sources (personally, I’m shocked that this figure is so low, and fear that many agile teams are contributing to data quality problems within their organization as a result).
- Legacy systems. There are several potential challenges with legacy systems. First, the code quality may not be the best either because it was never really that good to begin with or because it’s degraded over the years as multiple people worked with it. You know you’ve got a quality problem if you’re either afraid to update the code or if when you do so you have to spend a lot of time debugging and then fixing problems revealed when doing the update. If the legacy system is a true asset for your organization you will want to pay off some of this technical debt by refactoring the code to make it of higher quality. Second, you may not have a full regression test suite in place, making it difficult to find problems when you do update the code let alone when you refactor it. Third, your development tools for your legacy code may be a bit behind the times. For example, I often run across mainframe COBOL developers still working with basic code editors instead of modern IDEs such as Rational Developer for System Z. Some of the strategies to deal effectively with legacy systems are to adopt a modern development toolset if you haven’t already done so (better yet, if possible adopt a common IDE across platforms and thereby reduce overall licensing and support costs) and to adopt agile practices such as static code analysis, dynamic software analysis, and continuous integration (CI). The Agile Project Initiation Survey found that 57% of respondents were integrating their new code with legacy systems and 51% were evolving legacy systems.
- Commercial off-the-shelf (COTS) solutions. COTS solutions, also called package applications, can add in a few complexities for agile teams. The packages rarely come with regression test suites, they often have rules about what you can modify and what you shouldn’t (rules that are ignored at your peril), and they’re often architected with the assumption that they’re the center of the architectural universe (which is a valid assumption if they’re the only major system within your organization). As I describe in my article Agile Package Implementations it is possible to take an agile approach to COTS implementations, although it may require a significant paradigm shift for the people involved. The Agility at Scale survey found that 15% of respondents indicated that their agile teams were working with COTS solutions.
- System/embedded solutions. For the sake of simplicity, if your team is developing a solution with both hardware and software aspects to it then you’re a systems project. Embedded systems are a specialization where the system has a few dedicated functions often with real-time constraints. Bottom line is that systems/embedded projects are typically more challenging than software-only projects – it gets really interesting when laws of physics starts to kick in, such as when you’re building satellites or space probes. I highly suggest Bruce Douglass’s book Real-Time Agility if you are interested in taking an agile approach to systems/embedded solution delivery.
The technical complexity faced by a project team is contextual – Working with four technology platforms is straightforward for someone used to dealing with seven, but difficult for someone used to dealing with just one. Recommended Reading: