Agility@Scale: Strategies for Scaling Agile Software Development
It's been awhile since I've posted anything here, but that will soon change. I'm excited about some ideas that I have for upcoming blog postings around disciplined agile architecture and what it means to take a disciplined approach to agile solution delivery. Organizations around the world are finding that Disciplined Agile Delivery (DAD) provides a solid foundation from which to scale agile, and I'm going to continue sharing my observations and experiences about scaling DAD here in this blog.
As many of you may know I left IBM in June 2012, after a six-year stint as Chief Methodologist for IT within IBM Rational, to form Scott Ambler + Associates with Mark Lines. Mark is my co-author on the IBM Press book Disciplined Agile Delivery, a topic I've written about extensively in this blog over the years. Since I left IBM I've remained in contact with several key people within Rational and have started working with them on several fronts. For example Walker Royce, Alan Brown, and myself are co-authoring a paper about scaling agile which we hope will be accepted at a prestigious academic conference in 2013. I'm also working with IBM Rational at the upcoming Agile Development Conference East in Orlando, November 4-9. Both Mark and I will be presenting on DAD and Rational will have a booth and be throwing a party on Wednesday night which I'm looking forward to. Rational has thrown some pretty good conference parties in the past and I'm guessing that they will do so this time too. Hope to see you there.
In short, stay tuned as there's some exciting stuff coming your way soon.
ScottAmbler 120000HESD Tags:  disciplined-agile-deliver... agileadopt agile manifesto agility-at-scale 19 Comments 35,354 Visits
This article has been replaced by an official "Disciplined Agile Manifesto".
The text of the original article remains below.
I've recently been working with Mark Lines of UPMentors and we've had some interesting discussions around evolving the Agile Manifesto which I thought I would share here to obtain feedback. Note that this is not any sort of official position of IBM, nothing in my blog is by the way (unless explicitly stated so), nor is it some sort of devious plot to take over the agile world (although if we did have some sort of devious plot, we'd make the exact same claim). What we hope to accomplish is to put some ideas out there in the hopes of getting an interesting conversation going.
Updates Since this Was First Published:
ScottAmbler 120000HESD Tags:  disciplined-agile-deliver... agileexec agile agility-at-scale risk rup phases governance 6 Comments 8,528 Visits
The explicit phases of the Unified Process -- Inception, Elaboration, Construction, and Transition -- and their milestones are important strategies for scaling agile software development to meet the real-world needs of modern organizations. Yes, I realize that this is heresy for hard-core agilists who can expound upon the evils of serial development, yet these very same people also take a phased approach to development although are loathe to admit it. The issue is that the UP phases are like seasons of a project: although you'll do the same types of activities all throughout a project, the extent to which you do them and the way in which you do them change depending on your goals. For example, at the beginning of a development project if you want to be effective you need to do basic things like identify the scope of the project, identify a viable architecture strategy, start putting together your team, and obtain support for the project. Towards the end of a project your focus is on the activities surrounding the deployment of your system into production, including end-of-lifecycle testing efforts, training, cleaning up of documentation, piloting the system with a subset of users, and so on. In between you focus on building the system, including analysis, design, testing, and coding of it. Your project clearly progresses through different phases, or call them seasons if the term phase doesn't suit you, whether your team is agile or not.
The UP defines four phases, each of which address a different kind of risk:1. Inception. This phase focuses on addressing business risk by having you drive to scope concurrence amongst your stakeholders. Most projects have a wide range of stakeholdres, and if they don't agree to the scope of the project and recognize that others have conflicting or higher priority needs you project risks getting mired in political infighting. In the Eclipse Way this is called the "Warm Up" iteration and in other agile processes "Iteration 0".2. Elaboration. The goal of this phase is to address technical risk by proving the architecture through code. You do this by building and end-to-end skeleton of your system which implements the highest-risk requirements. Some people will say that this approach isn't agile, that your stakeholders should by the only ones to prioritize requirements. Yes, I agree with that, but I also recognize that there are a wide range of stakeholders, including operations people and enterprise architects who are interested in the technical viability of your approach. I've also noticed that the high-risk requirements are often the high-business-value ones anyway, so you usually need to do very little reorganization of your requirements stack.3. Construction. This phase focuses on implementation risk, addressing it through the creation of working software each iteration. This phase is where you put the flesh onto the skeleton.4. Transition. The goal of this phase is to address deployment risk. There is usually a lot more to deploying software than simply copying a few files onto a server, as I indicated above. Deployment is often a complex and difficult task, one which you often need good guidance to succeed at.
Each phase ends with a milestone review, which could be as simple as a short meeting, where you meet with prime stakeholders who will make a "go/no-go" decision regarding your project. They should consider whether the project still makes sense, perhaps the situation has changed, and that you're addressing the project risks appropriately. This is important for "agile in the small" but also for "agile in the large" because at scale your risks are often much greater. Your prime stakeholders should also verify that you have in fact met the criteria for exiting the phase. For example, if you don't have an end-to-end working skeleton of your system then you're not ready to enter the Construction phase. Holding these sorts of milestone reviews improves your IT governance efforts by giving senior management valuable visibility at the level that they actually need: when you have dozens or hundreds of projects underway, you can't attend all of the daily stand up meetings of each team, nor do you even want to read summary status reports.
These milestone reviews enable you to lower project risk. Last Autumn I ran a survey via Dr. Dobb's Journal (www.ddj.com) which explore how people actually define success for IT projects and how successful we really were. We found that when people define success in their own terms that Agile has a 71% success rate compared with 63% for traditional approaches. Although it's nice to that Agile appears to be lower risk than traditional approaches, a 71% success rate still implies a 29% failure rate. The point is that it behooves us to actively monitor development projects to determine if they're on track, and if not either help them to get back on track or cancel them as soon as we possibly can. Hence the importance of occasional milestone reviews where you make go/no-go decisions. If you're interested in the details behind the project, they can be found at http://www.ambysoft.com/surveys/success2007.html .
Done right, phases are critical to your project success, particularly at scale. Yes, the traditional community seems to have gone overboard with phase-based approaches, but that doesn't mean that we need to make the same mistakes. Let's keep the benefit without the cost of needless bureaucracy.[Read More]
ScottAmbler 120000HESD Tags:  agile-scaling-model agileexec agility-at-scale whitepaper asm agile 6,232 Visits
My new paper Scaling Agile: An Executive Guide is now available. As the title suggests the paper overviews how to scale agile strategies to meet your organization's unique needs.
The executive summary:
Agile software development is a highly collaborative, quality-focused approach to software and systems delivery, which emphasizes potentially shippable working solutions produced at regular intervals for review and course correction. Built upon the shoulders of iterative development techniques, and standing in stark contrast to traditional serial or sequential software engineering methods, agile software delivery techniques hold such promise that IBM has begun to adopt agile processes throughout its Software Group, an organization with over 25,000 developers. But how can practices originally designed for small teams (10-12) be “scaled up” for significantly larger operations? The answer is what IBM calls “agility@scale.”
There are two primary aspects of scaling agile techniques that you need to consider. First is scaling agile techniques at the project level to address the unique challenges individual project teams face. This is the focus of the Agile Scaling Model (ASM). Second is scaling your agile strategy across your entire IT department, as appropriate. It is fairly straightforward to apply agile on a handful of projects, but it can be very difficult to evolve your organizational culture and structure to fully adopt the agile way of working.
The Agile Scaling Model (ASM) defines a roadmap for effective adoption and tailoring of agile strategies to meet the unique challenges faced by a software and systems delivery team. Teams must first adopt a disciplined delivery lifecycle that scales mainstream agile construction techniques to address the full delivery process, from project initiation to deployment into production. Then teams must determine which agile scaling factors – team size, geographical distribution, regulatory compliance, domain complexity, organizational distribution, technical complexity, organizational complexity, or enterprise discipline, if any — are applicable to a project team and then tailor their adopted strategies accordingly to address their specific range of complexities.
When scaling agile strategies across your entire IT organization you must effectively address five strategic categories — the Five Ps of IT: People, principles, practices, process, and products (i.e., technology and tooling). Depending on your organizational environment the level of focus on each area will vary. What we are finding within many organizations, including IBM, is that the primary gating factor for scaling agile across your entire organization is your organization’s ability to absorb change.
Recently I visited a customer who had adopted Scrum. They were a few sprints, what Scrum calls iterations, into the project and were running into some difficulties. Although I was primarily brought in to educate senior management on disciplined agile software development, I was also asked to sit in on the team’s daily stand-up meeting so that I could hopefully provide some suggestions as to how to address the problems they were running into.
Their work area was fairly typical. They had some whiteboards which they were using for project planning and tracking, with sticky notes to indicate what work had been taken on by each team member. The current status of the task (not yet started, in progress, and completed) was indicated by putting each sticky note in a corresponding column for the status and corresponding column for the team member. This allowed everyone on the team to easily share their status and to see the status of everyone else. On the sides were sketches of the architecture as well as some business oriented models. In addition to Scrum the team had adopted several practices from Agile Modeling, in this case they had done some initial requirements envisioning and architecture envisioning, as well as practices from Extreme Programming (XP) for construction. In short, they had followed a fairly common strategy of combining practices from various agile methods.
This would have worked perfectly fine if they had tailored the practices to reflect the situation that they were in, but instead they adopted them "straight out of the book". First, the team was distributed, with most of the team in the location that I was visiting but some people located in two other distant cities. Therein was the source of most of their problems. The people at the other two locations weren’t getting much value out of the daily stand-up meetings, even though they would dial in, because they couldn’t see the project status information. Although people at this location were trying their best to represent these distant people in the daily stand-ups it wasn’t working well – their status information wasn’t being kept up to date and for some people it was a bit of mystery as to what they were actually working on at all.
This team also had 30 people in it, which isn’t a big deal although it can stretch the limits of the simple modeling and planning tools (in this case paper and whiteboards) that they were using. Because the team was larger they were investing a fair bit of time creating burn down charts at both the iteration/sprint and project levels. One of the unfortunate implications of using manual tools for project management is that any associated metric/status reporting in turn becomes manual as well. Considering how the agile community is so concerned with working efficiently, I find it comical that we have a tendency to overlook our own potentially unnecessary bureaucracy such as this.
The problem was that the team was applying strategies, in this case using sticky notes and whiteboards to capture the detailed iteration plan, applying similar strategies to capture key models, and were verbally relaying of status information between sub-teams. There are perfectly fine strategies for smaller co-located teams, but not so good for large or distributed teams. The solution was to recognize that they were in an Agility@Scale situation and needed to tailor their approach to reflect this fact. In this case they needed to forgo some of the manual tools and instead use electronic tooling such as Rational Team Concert (RTC) to share information across disparate locations, in particular the work assignment and corresponding status information. RTC also creates common agile reports such as burn-down charts based on the activities of the developers, providing accurate (nearly) real-time information while removing the burden of status reporting. The RTC project dashboard does more than just this, to see an actual example of one visit www.jazz.net to see the dashboard for the RTC development team itself. You can also see their actual work item list too, a more advanced version of Scrum’s product and sprint backlogs.[Read More]
ScottAmbler 120000HESD Tags:  tdd requirements design specification agileadopt agility-at-scale amdd testing 3 Comments 6,903 Visits
Test-driven development (TDD) is a common agile programming technique which has both specification and validation aspects. With TDD, you specify your software in detail on a just-in-time (JIT) basis via executable tests that are run in a regression manner to confirm that the system works to your current understanding of what your stakeholders require.
TDD is the combination of test-first development (TFD) and refactoring. With TFD, you write a single test (at either the requirements level with customer/acceptance tests or the design level with developer tests) and then you write just enough software to fulfill that test. Refactoring is a technique where you make a small change to your existing code to improve its design without changing its semantics.
TDD offers several benefits:1. It enables you to take small, safe steps during development, increasing programmer productivity.2. It increases quality. Agile developers are doing more testing, and doing it more often, than ever before. We're also fixing the problems that we find right on the spot.3. It helps to push validation activities early in the lifecycle, decreasing the average cost to fix defects (which rises exponentially the longer it takes you to detect them).4. Through single sourcing information, by treating tests as both specifications and as tests, we reduce the work required, increasing productivity.5. We leave behind valuable, up-to-date, detailed specifications for the people who come after us. Have you ever met a maintenance programmer who wouldn't want a full regression test suite for the code that they're working with?
But TDD isn't perfect. Although TDD is great at specifying code at a fine-grain level, tests simply don't scale to address higher level business process and architectural issues. Agile Model Driven Development (AMDD) enables you to scale TDD through initial envisioning of the requirements and architecture as well as just-in-time (JIT) modeling at the beginning and during construction iterations. To scale requirements-level TDD, you must recognize that customer tests are very good at specifying the details, but not so good at providing overall context. High-level business process models, conceptual domain models, and use cases are good at doing so, and these work products are often created as part of your initial requirements envisioning and iteration modeling activities. Similarly, to scale design-level TDD you must recognize that developer tests are very finely grained but once again do not provide overall context. High-level architecture sketches created during envisioning activities help set your initial technical direction. During each construction iteration, you'll do more detailed design modeling to think through critical issues before you implement them via TDD.
You also need to scale the validation aspects of TDD. TDD is in effect an approach to confirmatory testing where you validate the system to the level of your understanding of the requirements. The fundamental challenge with confirmatory testing, and hence TDD, is that it assumes that stakeholders actually know and can describe their requirements. Therefore you need to add investigative testing practices which explore issues that your stakeholders may not have thought of, such as usability issues, system integration issues, production performance issues, security issues, and a multitude of others.
For further reading, I suggest:1. My article "Introduction to TFD/TDD" at http://www.agiledata.org/essays/tdd.html which overviews TDD.2. My February 2008 column in Dr. Dobb's Journal entitled "Scaling TDD" at http://www.ddj.com/architect/205207998 which explores this issue in detail. 3. Andrew Glover's article "In pursuit of code quality: Adventures in behavior-driven development" at http://www.ibm.com/developerworks/java/library/j-cq09187/ which describes a new-and-improved take on TDD called BDD.[Read More]
I'm happy to announce that I've accepted the role of Managing Director of the Scrum Alliance (SA), a part-time position in addition to my duties here at IBM. On the surface this must appear to be a radical and unpredictable departure for me, considering my history of being critical when it comes to some of the past activities of the Scrum Alliance. To be fair, I've actually been critical of the Certified Scrum Master (CSM) scheme, and rightfully so. But I have also actively embraced the good ideas contained in Scrum and have incorporated them, with attribution, in my writings about Disciplined Agile Delivery (DAD) and other agile topics. I believe that I've made this very apparent in this blog and in other sources such as the Agile Modeling site. So, it really isn't such a radical departure for me afterall, although still arguably one that was difficult to predict. In fact, one of the reasons why the Scrum Alliance approached me to be Managing Director is the fact that I have been critical of many of the Scrum community's behaviors.
So, over the next few months you're going to see what I believe to be some welcome changes at the Scrum Alliance. Our first step at serving you better will be to apply agile strategies and principles in the way that we work. Importantly, we'll be taking a three pronged strategy based on respect, clarity, and integrity. We have dubbed this strategy "Scrum Alliance 2.0".
To be more respectful of existing and potential SA members, we will begin executing the following activities:
Clarity through openness and honesty
We are also starting to execute on four activities for improving the clarity of how we operate:
The fundamental idea here is that the Scrum Alliance should have nothing to hide from our membership. We've preached open and honest communication for years, now we're going to start actually living by those words. Yes, it may be a bit painful to work to this level of clarity, but we feel that you deserve this.
Integrity through actions, not words
Finally, we're taking three actions to increase the overall integrity of the Scrum community:
As I hope you have guessed by now this blog is an April Fool's joke. I have no intention of becoming the Managing Director of the Scrum Alliance and my condolences go out to anyone who would take on this position. This blog posting does however reflect what I would do to bring greater clarity, integrity, and respect to the Scrum community. The Scrum Alliance can and should choose to do a lot better. I hope it has been food for thought.
I recently ran into an interesting issue at a customer organization. This customer is in the process of transitioning to Disciplined Agile Delivery (DAD) and part of that effort is to train, mentor, and coach their people in these new ideas and techniques. The challenge is that some of "their people" are full time employees (FTEs) and some are contractors/consultants. When we were planning an upcoming DAD workshop with them, part of the planning effort was to identify who should get that training, which we're delivering in a just-in-time (JIT) basis on a team-by-team basis. The only people invited to take the training were FTEs because the customer has a policy of not training contractors. I pushed back a bit on this, but they were adamant about not training contractors because their view was that contractors should either have the skills required to do their jobs or be willing to get those skills on their own time. Fair enough, but from an agile team building point of view this isn't ideal.
This situation got me thinking a bit. One issue is that not all contractors are the same. Some are short term contractors that are brought in for a specific purpose, they're paid well, and then they move on. Other contractors stay much longer, sometimes months or even years, and as a result gain deeper knowledge and understanding of your business. For these longer term contractors it seems to me that there is little difference between them and FTEs, perhaps only in the way that they're remunerated. Some countries such as the United States now have laws in place limiting how long someone is allowed to remain a contractor because these similarities lead to interesting legal questions around extending benefits to them.
Another issue is that if you intend to build teams from both FTEs and contractors, it behooves you to ensure that these people get similar training, coaching and mentoring to streamline the transition effort.
Here's the logic I would suggest to address the issue of whether or not to train a contractor:
As always, let the context of the situation drive your strategy.
ScottAmbler 120000HESD Tags:  agileadopt disciplined-agile-deliver... agility-at-scale agile solution 14,231 Visits
An imporant step in scaling your agile strategy is to adopt a Disciplined Agile Delivery (DAD) approach instead of one which is just focused on agile construction. One aspect of adopting a DAD approach it to mature your focus from just producing software to instead providing a solution which meets the needs of its stakeholders within the appropriate economic, cultural, and technical constraints. The fundamental observation is that as IT professionals we do far more than just develop software. Yes, this is clearly important, but in addressing the needs of our stakeholders we will often:
Although delivery of high-quality, working software is important it is even more important that we deliver high-quality working solutions to our stakeholders. Minimally IT professionals should have the skills and desire to produce good software, but what they really need are the skills and desire to provide good solutions. We need strong technical skills, but we also need strong "soft skills" such as user interface design and process design to name just two.
The shift to a solution-oriented focus from a software-oriented focus requires your agile teams to address some of the software-oriented prejudices which crept into the Agile Manifesto. The people who wrote the manifesto (which I fully endorse) were for the most part software developers, consultants, and in many cases both. It is little wonder that this group would allow a bias towards software development creep into the language of their manifesto.
As you may know I write the agile update newsletter for Dr. Dobb's Journal (DDJ). One of the things that I do for DDJ is run the "State of the IT Union" surveys to find out what IT professionals are actually doing in practice. I invite you to fill out the January 2010 edition of the survey at http://www.surveymonkey.com/s/StateOfITUnion. The survey should take you about 5-7 minutes to complete, and your privacy will be completely protected.
At the end of the survey you will be given the chance to be entered into a draw for one of ten copies of "The Art of Scalability: Scalable Web Architecture, Processes, and Organizations for the Modern Enterprise" by Martin Abbott and Michael Fisher published in December 2009 by Addison Wesley.
The results of this survey will be summarized in a forthcoming newsletter, most likely my March 2010 agile update. Furthermore, this is an open survey, so the source data (without identifying information to protect your privacy), a summary slide deck, and the original source questions will be posted at www.ambysoft.com/surveys/ so that others may analyze the data for their own purposes. Data from previous surveys have been used by university students and professors for their research papers, and hopefully the same will be true of the data from this survey. The results from several other surveys are already posted there, so please feel free to take advantage of this resource.