Agility@Scale: Strategies for Scaling Agile Software Development
I was recently in Bangalore speaking at the Rational Software Conference, which was really well done this year, and visiting customers. In addition to discussing how to scale agile software development approaches, particularly when the team is distributed geographically and organizationally, I was also asked about what I thought about a software factory approach to development. My instinctual reaction was negative, software factories can result in lower overall productivity as the result of over specialization of staff (I prefer generalizing specialists), too many hand-offs between these specialists (I find close collaboration to be far more effective), and too much bureaucratic overhead to coordinate these activities. I initially chalked it up to these people still believing that software development was mostly a science, or perhaps an engineering domain, whereas my experiences had made me come to believe that software development is really more art than it is a science. Yet, the consistent belief in this strategy by very smart and experienced people started me thinking about my position.
Just let me begin by saying that this blog posting isn't meant to be yet another round in the age old, and relatively inane, "art vs. science" debate within the software development community. That debate is a symptom of versusitis, a dread disease which particularly plagues the IT industry and which can any of us at any time. There is no known cure, although the combination of experience, open-mindedness, and critical thought are the best inoculation against versusitis that we have so far. In that vein, let me explore the issues as I see them and I will let you think for yourself.
On the one hand software development has aspects of being an art for several reasons. First, the problem definition is never precise, nor accurate, and even when we have detailed specifications the requirements invariably evolve anyway. The lack of defined, firm requirements requires us to be flexible and to adjust to the situation that we find ourselves in. Second, teams typically find themselves in unique situations, necessitating a unique process and tool environment to reflect this (assuming that you want to be effective, otherwise there's nothing stopping you from having a "repeatable process" and consistent tool environment). Third, software is built by people for people, requiring that the development team have the ability to build a system with a user interface which meets the unique needs of their end users. One has only to look at the myriad UI designs out there to see that surely there is a bit of art going on. Fourth, if software development wasn't at least partially art then why hasn't anyone succeeded at building tools which take requirements as inputs and produce a viable solution that we can easily deploy? It's been over four decades now, so there's been sufficient time and resources available to build such tooling. Fifth, regardless of how much of a scientific/business facade we put over it, our success rate at producing up front detailed cost estimates and schedules speak for itself (see Funding Agile Projects for links to articles).
On the other hand software development has aspects of being a science for several reasons. First, some aspects of software development have in fact been automated to a significant extent. Second, there is some mathematical basis to certain aspects of software development (although in the case of data-oriented activities the importance of relational theory often gets blown way out of proportion and I have yet to see a situation where formal methods proved to be of practical value).
What does this have to do with Agility@Scale. As you know, one of the agile scaling factors is Organizational Complexity, and cultural issues are the hardest to overcome. Whether your organization believes that software development is mostly an art or mostly a science is a cultural issue which will be a major driver in you choice of methods and practices. Organizations which believe that software development is more of a science will prefer strategies such as software factories, model-driven architecture (MDA), and master data management (MDM). And there is ample evidence to support the claims that some organizations are succeeding at these strategies. Although you may not agree with these strategies, you need to respect the fact that many organizations are making them work in their environments. Similarly, organizations which believe that software development is more of an art will find that agile and lean strategies are a better fit, and once again there is ample evidence that organizations are succeeding with these approaches (there's also evidence that agile projects are more successful than traditional projects, on average). Once again, you may not agree with these strategies but you need to respect the fact that other people are making them work in practice.
Trying to apply agile approaches within an organization that believes software development is mostly a science will find it difficult at best, and will likely need to embark on a multi-year program to shift their culture (likely an expensive endeavor which won't be worth the investment). Similarly, trying to apply a software factory strategy in an organization that believes that software development is mostly an art will also run aground. The bottom line is that one size does not fit all, that one strategy is not right for all situations and that you need to understand the trade-offs of various strategies, methodologies, techniques, and practices and apply them appropriately given the situation that you face. In other words, it depends! If you are embarking on a software process initiative, and you don't have the broad experience required to effective choose between strategies (very few organizations do, although many believe otherwise), then you should consider Measured Capability Improvement Framework (MCIF) to help increase your chance of success.
Recently Gina Poole blogged about IBM saved $300 million by going agile. That's not too bad when you think about it.
A few days later someone asked a series of questions that I thought would make an interesting blog posting, so here goes:
How much of IBM's projects (in percentage) are agile at the moment?
I don’t have exact numbers, but I believe that 90%+ of our teams in SWG are applying agile techniques in practical ways that make sense for their projects. The primary goal is to be effective – in frequent releases, higher quality, and happy customers – not just agile. By the way, there is roughly 30,000 developers in SWG.
Can all of IBM's projects work with an agile methodology?
It’s certainly possible, but it may not always make sense. Products that are in maintenance mode with few bugs or feature requirements may not benefit as much from agile practices -- those teams will likely continue to do whatever it is that they have been doing. Having said that, it's still highly desirable to apply agile techniques on maintenance projects.
Also, agile methods can be harder to use on some projects than others, for example, around hardware development. As a general rule, I believe that the majority of software projects can benefit from agile techniques. The primary determinant of whether a team can adopt agile techniques is culture and skill – not team size, the domain, or the degree of geographic distribution. That notion surprises many people who think that large agile teams or geographically distributed agile teams can’t succeed in adopting agile practices.
Are agile projects sub-parts of large waterfall projects?
In some cases, that may happen. I’m sure it’s also true in reverse. We see many customers who are migrating from waterfall projects to a more agile way of doing things, and they often start this migration with smaller sub-projects. At IBM, we have tens of thousands of developers worldwide on hundreds of teams, so we have examples of pretty much any combination of agile, iterative, and traditional practices that you can imagine. There’s definitely not one size that fits all, which is a key aspect of the Disciplined Agile Delivery (DAD) process framework.
What do you think the impact of these numbers will be on the PM community?
The IBM PM community is embracing agile. And the reality is that a majority of development organizations around the world are moving to agile now as well (as much as 80% in some of the recent studies I’ve seen). I look forward to the increased adoption of agile methods by the PM community in general. The fact that PMI now offers an Agile Certified Practitioner training program certainly underscores the fact that agile practices are being adopted widely in the mainstream which is a great thing to see.
There is a distinct rhythm, or cadence, at different levels of the agile process. We call this the agile 3C rhythm, for coordinate, collaborate, and conclude (which is sometimes called stabilize). The agile 3C rhythm occurs at three levels in Disciplined Agile Delivery (DAD):
The agile 3C rhythm is similar conceptually to Deming’s Plan, Do, Check, Act (PDCA) cycle:
When I talk to people about scaling agile techniques, or about agile software development in general, I often put describe strategies in terms of various risks. I find that this is an effective way for people to understand the trade-offs that they're making when they choose one strategy over another. The challenge with this approach is that you need to understand these risks that you're taking on, and the risks that you're mitigating, with the techniques that you adopt. Therein lies the rub, because the purveyors of the various process religions ( oops I mean methodologies) rarely seem to coherently the discuss the risks which people take on (and there's always risk) when following their dogma (oops, I mean sage advice).
My final advice is to take a look at the Disciplined Agile Delivery (DAD) framework as it provides a robust strategy for addressing the realities of agile software development in enterprise settings.
ScottAmbler 120000HESD Tags:  product-management discipilnedagiledelivery safe agility-at-scale testing agile scaled-agile agileexec agile-exec architecture 18,058 Views
I recently wrote a detailed article about Large Agile Teams that was a detailed walkthrough of how to structure agile teams of various sizes. I suspect that this is the most comprehensive online discussion of this topic. The article addressed the following topics:
I welcome any feedback that you may have about Large Agile Teams.
Recently I spent some time in the UK with Julian Holmes of Unified Process Mentors. In one of our conversations we deplored what we were seeing in the agile community around certification, in particular what the Scrum community was doing, and he coined the term “integrity debt” to describe the impact it was having on us as IT professionals. Integrity debt is similar to technical debt which refers to the concept that poor quality (either in your code, your user interface, or your data) is a debt that must eventually be paid off through rework. Integrity debt refers to the concept that questionable or unprofessional behavior builds up a debt which must eventually be paid off through the rebuilding of trust with the people that we interact with.
The agile community has been actively increasing their integrity debt through the continuing popularity of Scrum Certification, in particular the program around becoming a Certified Scrum Master (CSM). To become a CSM you currently need to attend, and hopefully pay attention during, a two-day Scrum Master Certification workshop taught by a Certified Scrum Trainer (CST). That’s it. Granted, some CSTs will hold one or more quizzes which you need to pass, an optional practice which isn’t done consistently, to ensure that you pay attention in the workshop.
Scrum Masters, as you know, take the leadership position on a Scrum team. The idea that someone can master team leadership skills after two entire days of training is absurd. Don’t get me wrong, I’m a firm supporter of people increasing their skillset and have no doubt that many of the CSTs deliver really valuable training. However, there is no possible way that you can master a topic, unless it is truly trivial, in only two days of training. From what I can tell the only thing that is being certified here is that your check didn’t bounce.
The CSM scheme increases the integrity debt of the IT industry by undermining the value of certification. When someone claims that they’re certified there’s an assumption that they had to do something meaningful to earn that certification. Attending a two-day course, and perhaps taking a few quizzes where you parrot back what you’ve heard, clearly isn’t very meaningful. The problem with the term Certified Scrum Master is two-fold: not only does the term Certified imply that the holder of the certification did something to earn it, the term Master implies that they have significant knowledge and expertise gained over years of work. It is very clear that people are falling for the Scrum certification scheme.
A quick search of the web will find job ads requiring that candidates be CSMs, undoubtedly because they don’t realize that there’s no substance behind the certification. Whenever I run into an organization that requires people to be CSMs I walk them through the onerous process of earning the designation and suggest that they investigate the situation themselves. Invariably, once they recognize the level of deception, the customer drops the requirement that people be CSMs.
Another quick search of the web will find people bragging about being a CSM, presumably being motivated by the employment opportunities within the organizations gullible enough to accept Scrum certification at face value. My experience is that the people claiming to be CSMs are for the most part decent, intelligent people who 99.99% of the time have far more impressive credentials to brag about than taking a two-day course. Yet, for some reason they choose to park their integrity at the door when it comes to Scrum certification. I suspect that this happens in part because they see so many other people doing it, in part because they’re a bit desperate to obtain or retain employment in these tough economic times, and in part because the IT industry doesn’t have a widely accepted code of ethical conduct. These people not only embarrass themselves when they indicate on their business cards or in their email signatures that they’re Certified Scrum Masters they also increase the integrity debt of the agile community as a whole.
Yet another search of the web will find people bragging about being Certified Scrum Trainers (CSTs), the people whom have been blessed by the Scrum Alliance to deliver Scrum master certification courses. Once again, my experience is that these are intelligent, skilled people, albeit ones who have also parked their integrity at the door in the pursuit of a quick buck. Surely these people could make a decent living via more ethical means? I know that many of them have done so in the past, so I would presume that they could do so in the future. The actions of the CSTs increase our integrity debt even further.
The group of people who have most embarrassed themselves, in my opinion, are those whom we consider thought leaders within the agile community. Leaving aside the handful who are directly involved with the Scrum certification industry, the real problem lies with those who have turned a blind eye to all of this. The Scrum certification scheme was allowed to fester within our community because few of our thought leaders had the courage to stand up and publicly state what they were talking about in private. This of course is all the more galling when you consider how much rhetoric there is around the importance of courage on software development projects. As Edmund Burke once observed, all that is necessary for evil to triumph is for good men to do nothing.
There are several things that we can do today to start paying off some of our integrity debt:
ScottAmbler 120000HESD Tags:  modeling agility-at-scale agileadopt requirements 8 Comments 17,751 Views
Contrary to popular belief, agile development teams do in fact model and yes, they even do some up front requirements and architecture modeling. Two of the best practices of Agile Modeling are Requirements Envisioning and Architecture Envisioning where you spend a bit of time at the beginning of the project doing enough initial modeling to get you going in the right direction. The strategy is to take advantage of modeling, which is to communicate and think things through without taking on the risks associated with detailed specifications written early in the lifecycle. In this blog posting I will focus on requirements envisioning, in a future posting I'll cover architecture envisioning.
The goal of initial requirements envisioning is to identify the scope of your effort. You need to do just enough modeling early in the project to come to stakeholder concurrence and answer questions such as what you're going to build, roughly how long it's going to take (give a range), and roughly how much it's likely to cost (once again, give a range). If you can get the right people together in the room, which can sometimes be a logistics challenge but not one that you couldn't choose to overcome, there are very few systems (I suspect less than 5%) that you couldn't initially scope out in a few days or a week. I also suspect that most of the remaining systems could be scoped out with less than 2 weeks of modeling, and if not then I'd take that as an indication that you're taking on too large of a project. I'm not saying that you'll be able to create big detailed specifications during this period, and quite frankly given the problems associated with "Big Requirements Up Front (BRUF)" you really don't want to, but I am saying that you could gain a pretty good understanding of what you need to do. The details, which you'll eventually need, can be elicited throughout the lifecycle when you actually need the information. A common saying in the agile community is that requirements analysis is so important for us that we do it every single day, not just during an initial phase. I'll discuss just in time (JIT) approaches to requirements modeling in a future posting.
To envision the requirements for a business application, you might want to consider creating the following models:
For small teams simple tools such as whiteboards and paper are usually sufficient for requirements envisioning. But what happens at scale? What if you're working on a large agile team, say of 50 people, 200 people (IBM has delivered software into the marketplace with agile teams of this size), or even 500 people (IBM currently has teams of this size applying agile techniques)? What if your team is distributed? Even if you have people working on different floors of the same building, let alone working from home or working in different cities or countries, then you're distributed (see my postings about distributed agile development). Suddenly whiteboards and paper-based tools (index cards, sticky notes, ...) aren't sufficient. You're still likely to use these sorts of tools in modeling sessions with stakeholders, but because of one or more scaling factors you need to capture your requirements models electronically.
In January Theresa Kratschmer and I gave a webcast entitled Agile Requirements: Collaborative, Contextual, and Correct which overviewed agile approaches to requirements elicitation and management, including requirements envisioning. We also showed how Rational Requirements Composer (RRC) can be used to electronically capture critical requirements information, enabling you to address the needs of large and/or distributed agile teams, while still remaining lightweight and flexible. I suspect that you'll find the webcast to be very illuminating and RRC something that you want to take a look at (the link leads to a trial version). Of course RRC can be used in other situations as well, but that's not what I'm focused on right now.
Teams which find themselves in regulatory environments will likely need to do more than just use RRC, as might very large teams. Regulatory compliance often requires more complex requirements documentation, which in turn requires more sophisticated tools such as DOORS or Requisite Pro, and I would consider using those tools in the types of situations that warrant it. One of the things that people often struggle to understand about agile approaches is that you need to tailor your strategy to reflect the situation at handle. One process size does not fit all, so you will end up using different tools and creating different artifacts to different extents in different situations. Repeatable results, not repeatable processes, is the rule of the day.
ScottAmbler 120000HESD Tags:  business-analysis agility-at-scale domain-complexity scaling-agile agile sdcf 17,748 Views
One of the scaling factors called out in the Software Development Context Framework (SDCF) is domain complexity. The general idea is that agile teams will find themselves in different situations where some teams are developing fairly straightforward solutions, such as an informational website, whereas others are addressing very complex domains, such as building an air-traffic control system (ATCS). Clearly the team building an ATCS will work in a more sophisticated manner than the one building an informational website. I don't know whether agile techniques have been applied in the development of an ATCS, although I have to think that agile's greater focus on quality and working collaboratively with stakeholders would be very attractive to ATCS delivery teams, I do know that agile is being applied in other complex environments: The 2009 Agility at Scale Survey found that 18% of respondents indicated that their organizations had success at what they perceived to be very complex problem domains,.
The important thing is to recognize that the strategies which work well when you're dealing with a simple domain will not work well for a complex domain. Conversely, techniques oriented towards exploring complex domains will often be overkill for simple domains. Process and tooling flexiblity is key to your success.
ScottAmbler 120000HESD Tags:  scaling-agile gdd scrum sdcf book-review agility-at-scale 17,624 Views
I'm happy to announce that A Practical Guide to Distributed Scrum by Elizabeth Woodward, Steffan Surdek, and Matthew Ganis is now in print. I've been talking this book up in presentations and with customers the past few months and promised that I would let everyone know once it was available. I was one of several people who wrote forewords for the book, Ken Schwaber, Roman Pichler, and Matthew Wang also did so, and I've modified my foreword below to help you to understand a bit better what the book is about.
ScottAmbler 120000HESD Tags:  agility-at-scale disciplined-agile-deliver... agile e-book sdcf 17,614 Views
IBM Rational recently published an update to my Agility@Scale e-book, which can be downloaded free of charge. The e-book is a 21 page, 2.3 meg PDF (sorry about the size, guess the graphics did it) . It overviews the Agile Scaling Model (ASM) (which has since been replaced by the Software Development Context Framework (SDCF) ), Disciplined Agile Delivery (DAD), the scaling factors of agility at scale, and ends with some advice for becoming as agile as you can be. In short it's a light-weight coverage of some of the things I've been writing about in this blog the past couple of years. Could be a good thing to share with the decision makers in your organization if they're considering adoption agile strategies.