One of the scaling factors
of the Agile Scaling Model (ASM)
is technical complexity.
The fundamental observation is that the underlying technology of solutions varies and as a result your approach to developing a solution will also need to vary.
It’s fairly straightforward to achieve high-levels of quality if you’re building a new system from scratch on a known technology platform, but not so easy when there are several technologies, the technologies are not well known, or legacy assets are involved.
There are several potential technical complexities which a Disciplined Agile Delivery (DAD) team may face:
- New technology platforms. Your team may choose to work with a technology platform which is either new to the team or sometimes even new to the industry. In the past few years new technology platforms include the Android operating system, Apple’s iPad platform, and various cloud computing (http://www.ibm.com/ibm/cloud/) platforms. Working with these platforms may require you to adopt new development tools and techniques, not to mention the need to train and mentor your staff in their usage. Furthermore, your team may need to allocate time for architectural spikes to explore how to use the new technology and to prove the overall architecture with working code early in the project lifecycle (this is a DAD milestone).
- Multiple technology platforms. IT solutions often run on multiple platforms. For example, a system’s user interface (UI) could run in a browser, access business logic implemented using J2EE on Websphere which in turn invokes web services implemented in COBOL running on a Z-series mainframe, and stores data in an Oracle database, a DB2 database, and in several XML files. Implementing new business functionality, or updating existing functionality, could require changes made on several of these platforms in parallel. The implication is that you’ll need to adopt tools and strategies which enable your team to develop, test, and deploy functionality on all of these platforms. Testing and debugging in particular will become more difficult as the number of technology platforms increases, potentially requiring you to adopt the practice of parallel independent testing. The Agility at Scale survey found that 34% of respondents indicated that their agile teams were working with multiple technology platforms.
- Legacy data. IT solutions should leverage existing, legacy data wherever possible to reduce the number of data sources and thereby increase data quality within your organization. Also, using existing data sources can potentially speed up development, assuming your team has a good relationship with the owners of the legacy data sources (sadly, this often isn’t the case as the Data Management Survey found). Working with legacy data sources may require improved database regression testing, practices, database refactoring practices, and agile approaches to data administration. The Agility at Scale survey found that 42% of respondents indicated that their agile teams were working with legacy data sources (personally, I’m shocked that this figure is so low, and fear that many agile teams are contributing to data quality problems within their organization as a result).
- Legacy systems. There are several potential challenges with legacy systems. First, the code quality may not be the best either because it was never really that good to begin with or because it’s degraded over the years as multiple people worked with it. You know you’ve got a quality problem if you’re either afraid to update the code or if when you do so you have to spend a lot of time debugging and then fixing problems revealed when doing the update. If the legacy system is a true asset for your organization you will want to pay off some of this technical debt by refactoring the code to make it of higher quality. Second, you may not have a full regression test suite in place, making it difficult to find problems when you do update the code let alone when you refactor it. Third, your development tools for your legacy code may be a bit behind the times. For example, I often run across mainframe COBOL developers still working with basic code editors instead of modern IDEs such as Rational Developer for System Z. Some of the strategies to deal effectively with legacy systems are to adopt a modern development toolset if you haven’t already done so (better yet, if possible adopt a common IDE across platforms and thereby reduce overall licensing and support costs) and to adopt agile practices such as static code analysis, dynamic software analysis, and continuous integration (CI). The Agile Project Initiation Survey found that 57% of respondents were integrating their new code with legacy systems and 51% were evolving legacy systems.
- Commercial off-the-shelf (COTS) solutions. COTS solutions, also called package applications, can add in a few complexities for agile teams. The packages rarely come with regression test suites, they often have rules about what you can modify and what you shouldn’t (rules that are ignored at your peril), and they’re often architected with the assumption that they’re the center of the architectural universe (which is a valid assumption if they’re the only major system within your organization). As I describe in my article Agile Package Implementations it is possible to take an agile approach to COTS implementations, although it may require a significant paradigm shift for the people involved. The Agility at Scale survey found that 15% of respondents indicated that their agile teams were working with COTS solutions.
- System/embedded solutions. For the sake of simplicity, if your team is developing a solution with both hardware and software aspects to it then you’re a systems project. Embedded systems are a specialization where the system has a few dedicated functions often with real-time constraints. Bottom line is that systems/embedded projects are typically more challenging than software-only projects – it gets really interesting when laws of physics starts to kick in, such as when you’re building satellites or space probes. I highly suggest Bruce Douglass’s book Real-Time Agility if you are interested in taking an agile approach to systems/embedded solution delivery.
The technical complexity faced by a project team is contextual – Working with four technology platforms is straightforward for someone used to dealing with seven, but difficult for someone used to dealing with just one. Recommended Reading:
In Implementing Lean Software Development
, Mary and Tom
Poppendieck show how the seven principles of lean manufacturing can be applied
to optimize the whole IT value stream. These principles are:
- Eliminate waste. Lean thinking advocates regard any activity
that does not directly add value to the finished product as waste. The three
biggest sources of waste in software development are the addition of unrequired
features, project churn and crossing organizational boundaries (particularly
between stakeholders and development teams). To reduce waste it is critical
that development teams be allowed to self organize and operate in a manner that
reflects the work they’re trying to accomplish. Walker Royce argues in “Improving Software Economics” that the primary benefit of modern iterative/agile
techniques is the reduction of scrap and rework late in the lifecycle.
- Build in quality. Your process should not allow defects to
occur in the first place, but when this isn’t possible you should work in such
a way that you do a bit of work, validate it, fix any issues that you find, and
then iterate. Inspecting after the fact,
and queuing up defects to be fixed at some time in the future, isn’t as
effective. Agile practices which build
quality into your process include test driven development (TDD) and non-solo
development practices such as pair programming and modeling with others.
- Create knowledge. Planning is useful, but learning is essential.
You want to promote strategies, such as iterative development, that help teams
discover what stakeholders really want and act on that knowledge. It’s also
important for a team to regularly reflect on what they’re doing and then act to
improve their approach.
- Defer commitment. It’s not necessary to start software
development by defining a complete specification, and in fact that appears to
be a questionable strategy at best. You can support the business effectively
through flexible architectures that are change tolerant and by scheduling
irreversible decisions to the last possible moment. Frequently, deferring
commitment requires the ability to closely couple end-to-end business scenarios
to capabilities developed in multiple applications by multiple projects.
- Deliver quickly. It is possible to deliver high-quality
systems quickly. By limiting the work of a team to its capacity, which is
reflected by the team’s velocity (this is the number of “points” of
functionality which a team delivers each iteration), you can establish a
reliable and repeatable flow of work. An effective organization doesn’t demand
teams do more than they are capable of, but instead asks them to self-organize
and determine what they can accomplish. Constraining these teams to delivering potentially
shippable solutions on a regular basis motivates them to stay focused on
continuously adding value.
- Respect people.
The Poppendiecks also observe that sustainable advantage is gained from
engaged, thinking people. The implication is that you need a lean governance
strategy that focuses on motivating and enabling IT teams—not on controlling
- Optimize the whole. If you want to be effective at a solution you
must look at the bigger picture. You need to understand the high-level business
processes that individual projects support—processes that often cross multiple
systems. You need to manage programs of interrelated systems so you can deliver
a complete product to your stakeholders. Measurements should address how well
you’re delivering business value, because that is the sole reason for your IT
Lean thinking is important for scaling agile in several ways:
- Lean provides an explanation for why many of the agile
practices work. For example, Agile
Modeling’s practices of light weight, initial requirements envisioning followed
by iteration modeling and just-in-time (JIT) model storming work because they
reflect deferment of commitment regarding what needs to be built until it’s
actually needed, and the practices help eliminate waste because you’re only modeling
what needs to be built.
Lean offers insight into strategies for improving your
software process. For example, by
understanding the source of waste in IT you can begin to identify it and then
Lean principles provide a philosophical foundation for
scaling agile approaches.
- It provides techniques for identifying waste. Value stream mapping, a technique common within the lean
community whereby you model a process and then identify how much time is spent
on value-added work versus wait time, helps calculate overall time efficiency
of what you’re doing. Value stream maps are
a straightforward way to illuminate your IT processes, providing insight into
where significant problems exist. I’ve
created value stream maps with several customers around the world where we
analyzed their existing processes which some of their more traditional staff
believed worked well only to discover they had efficiency ratings of
20-30%. You can’t fix problems which you
are blind to.
One of the scaling factors
called out in the Agile Scaling Model (ASM)
is “regulatory compliance”. This name is a bit of a misnomer because this scaling factor really addresses two issues: complying to regulations imposed upon you from external sources and choosing to adhere to internal regulations willingly adopted by your organization. It is relatively common for agile teams to find themselves in such situations. For example, in the 2009 Agile Practices Survey
one third of respondents said that they were applying agile on projects where one or more industry regulations applied.
First let’s consider external regulatory compliance. In these situations you may face the need to undergo an audit by an external regulatory body with consequences for non-compliance ranging from anywhere to a warning to a fine or even to legal action. Sometimes even a warning may be a grave thing. A few years ago I was working with a pharmaceutical company which had discovered that a warning from the FDA for non-compliance with their CFR 21 Part 11 regulation, when reported in major newspapers, resulted on average in a half-billion dollar loss to their market capitalization as the result of a dip in their stock price. There are financial regulations such as Sarbanes-Oxley and Basel II, informational regulations such as HIPAA which focuses on health information privacy, technical regulations such as ISO 27002 for security practices, and even life-critical regulations such as some of the FDA regulations.
External regulations are typically managed by a government organization or industry watchdog will range in complexity and can have a myriad of effects on project teams. For example, you may need to be able to prove that you had a documented process and that you followed it appropriately; you may need to produce extra artifacts, or more detailed artifacts, than you normally would; you may need to add extra features to your solution, such as tracking financial information, that you wouldn’t have normally implemented; you may need to produce specific reports to be submitted to the regulatory body; or you may even need to submit your team to audits, sometimes scheduled and sometimes not, to ensure regulatory compliance. Interestingly, even though many of those requirements go against the agile grain, the 2009 Agility at Scale Survey
found that organizations were successfully applying agile techniques while still conforming to external regulations. So yes, it is possible to scale your agile strategy to address regulatory compliance.
Second, let’s consider compliance to internally adopted, or sometimes even developed, “regulations” which you will be potentially evaluated/appraised against. Perfect examples of these are process improvement frameworks such as CMMI and ISO 900x. Similar to external regulations, the 2009 Agility at Scale Survey
found that some agile teams are succeeding in situations where they have chosen to adopt such frameworks. It’s important to note that frameworks such as CMMI aren’t primarily about ensuring the compliance of development teams to a standard process, regardless of what CMMI detractors may claim, but instead about process improvement. Process improvement at the IT department (or beyond) is an enterprise discipline issue from the point of view of ASM, implying that frameworks such as CMMI affect more than one scaling factor.
When you find yourself in a regulatory situation, whether those regulations are imposed or willingly adopted, the best advice that I can give is to read the regulations and develop a strategy to conform to them in the most agile manner possible. If you let bureaucrats interpret the regulations you’ll likely end up with a bureaucratic strategy, but if you instead choose to take a pragmatic approach you will very likely end up with a very practical strategy. Part of that strategy is to treat the regulatory representative(s) within your organization as important stakeholders whom you interact with regularly throughout the project.
You are invited to participate in my 2010 IT Project Success survey (http://www.surveymonkey.com/s/StateOfITUnion
The goal of this survey series is to find out how we define IT project
success in practice and how successful our projects actually are. The
survey should take you about 5 minutes to complete, and your privacy will
be completely protected.
At the end of the survey you will be given
the chance to be entered into a draw for one of 10 copies of Reflections on
Management: How to Manage Your Software Projects, Your Teams, Your Boss,
and Yourself by Watts Humphrey and William R. Thomas published in April 2010
by Addison Wesley. I'm reading it right now and it's a really great
This is an open survey, so the source data (without
identifying information to protect your privacy), a summary slide deck, and
the original source questions will be posted at www.ambysoft.com/surveys/
that others may analyze the data for their own purposes. Data
from previous surveys have been used by university students and
professors for their research papers, and hopefully the same will be true of
the data from this survey. The results from several other surveys
are already posted there, so please feel free to take advantage of
Thank you very much for taking your valuable time to
fill out this survey.
Rolf Nelson recently recorded a short (5 min) podcast about IBM Rational
(RTC). RTC is a complete agile collaborative development
environment providing agile planning, source code management, work item
management, build management, and project health, along with integrated
reporting and process support. I've worked with RTC for a couple of years now and have been truly impressed with it. What should be of interest to many people is the Express-C version which is a free, fully-featured, 10-license version of RTC which can be easily downloaded from www.jazz.net
The SEMAT vision
has recently been posted online. SEMAT is short for Software Engineering and Method and Theory (SEMAT) and I am one of several signatories and a provider of input into the effort.
There are several reasons why I'm involved with SEMAT:
- The industry clearly needs something like this. Ivar Jacobson has been writing and speaking for awhile now about how our industry behaves in a similar manner to the fashion industry
-- we lurch from one cool idea (the fashion) to another and few of us
observe that the current fashion is often just a rehash of fashion(s) from the
- The right people are involved. The SEMAT initiative has
attracted a wide range of industry experts, including both the old
guard, the new "agile guard", and people in between. Arguably we may
be short a few people, the data community isn't well represented and
I'm not sure we have the systems community covered well, but we can
address those challenges in time.
We'll achieve something. We may not pull of the entire vision, but we
are going to produce something of value and I hope that it has a
positive impact on the industry. Time will tell.
I have several thoughts about the SEMAT vision
which I'd like to share with you:
- The vision is coherent. A lot of reasoned effort went into it's development as you can see when you read it.
- It will be a challenge to identify a non-trivial kernel. Even developing a kernel language will be hard as people will often stick to their preferred terms. For example, is it an iteration, a sprint, or a time box?
- Practitioners may not notice. It will also be a significant challenge to get practitioners interested in the SEMAT effort and more importantly to leverage the material. For example, the patterns community has a long history of producing great work which for the most part is ignored by the vast majority of practitioners, with the exception of a handful of the hundreds of patterns out there. So, although SEMAT is likely to produce some great ideas, will anyone care?
- Academics may not notice. I suspect that this will be less of a problem than practitioners not noticing, but it's still a possibility.
- Position papers from many of the signatories, including myself, will soon be published at the SEMAT site. Several non-signatories have been invited to submit papers as well.
- A workshop is being held in the third week of March in Zurich. At this workshop the people who published position papers will flesh out our ideas.
- I'll blog about these things as they occur.
Just like there are 5Ps of marketing, there are also “5 Ps” of IT:
- People. People and the way they work together have a greater effect on the outcomes of a project than the processes they’re following or the products (tools and technologies) that they’re using. People issues include having visible executive sponsorship, building an environment of trust, empowering staff, focusing on leadership as well as management, recognizing that the primary gating factor when improving processes is people’s ability to absorb change, and promoting a cross-discipline strategy at both the team and individual levels.
- Principles/philosophies. We’ve found both internally within IBM as well as with many of our customers that there is a need to define a common set of principles to provide a consistent foundation to enable effective teamwork and continuous process improvement. These principles help to guide people’s decisions when their processes and practices don’t directly address the situation which they find themselves in.
- Practices/patterns. A practice is a self-contained, deployable component of a process. You might find the IBM Practices interesting.
- Products. This includes the technologies – such as databases, application servers, networks, and client platforms – and tools such as integrated development environments, testing tools, and project planning tools used to create solutions for stakeholders.
- Processes. The previous 4Ps do not exist in a vacuum, we need some sort of glue to help piece all of this together. Minimally this glue is a lifecycle although more often than not it is a full process or method.
My experience is that to be successful at software process improvement (SPI)
across your entire IT department that you must address these 5Ps. How you address each issue, and to what extent, will vary based on your situation.
My January 2010 DDJ Agile Update, Tragic Mistakes When Adopting Test Driven Development (TDD)
, is now online. In the article I summarize what I consider to be common, and tragic, mistakes that I'm seeing organizations make when they attempt to adopt TDD.
These mistakes include:
The article also goes into potential benefits of TDD as well as potential challenges that you're face when adopting it.
As you may know I write the agile update newsletter for Dr. Dobb's Journal (DDJ)
. One of the things that I do for DDJ is run the "State of the IT Union" surveys to find out what IT professionals are actually doing in practice. I invite you to fill out the January 2010 edition of the survey at http://www.surveymonkey.com/s/StateOfITUnion
. The survey should take you about 5-7 minutes to complete, and your privacy will be completely protected.
At the end of the survey you will be given the chance to be entered into a draw for one of ten copies of "The Art of Scalability: Scalable Web Architecture, Processes, and Organizations for the Modern Enterprise" by Martin Abbott and Michael Fisher published in December 2009 by Addison Wesley.
The results of this survey will be summarized in a forthcoming newsletter, most likely my March 2010 agile update. Furthermore, this is an open survey, so the source data (without identifying information to protect your privacy), a summary slide deck, and the original source questions will be posted at www.ambysoft.com/surveys/
so that others may analyze the data for their own purposes. Data from previous surveys have been used by university students and professors for their research papers, and hopefully the same will be true of the data from this survey. The results from several other surveys are already posted there, so please feel free to take advantage of this resource.
On Tuesday, Dec 1, 2009 Philippe Kruchten
, Bruce MacIsaac
, and myself participated on two virtual panels about the future of the Unified Process (we did two to support callers from around the globe) for the Global Rational User's Group (GRUG)
. During the panel sessions we discussed a bit of the history of the Unified Process, some of the misconceptions people have with it, some of the common mistakes people made implementing it (instantiating it to be documentation heavy and/or serial) due to those misconceptions, how it can be very agile if you choose to instantiate it that way, the OpenUP
, the AUP
, how UP relates to the IBM Practices
, and other topics.
The links to the recordings are:
Hope you find it interesting. As I've written in the past, the RUP can be as agile as you want to make it. Furthermore, there are a lot of really good ideas in the RUP that the agile community can and should choose to mine, although sadly I see far too many teams doing things the hard way and reinventing the process wheel on their own. I hope they're enjoying themselves, because it clearly isn't a very efficient way for them to go about process improvement.
When you’re inside, safe in the warmth of your home watching snow fall on your driveway outside, all snowflakes look the same. But, when you look at a snowflake up close, particularly when you do so under a microscope, you quickly discover that all snowflakes are in fact unique.
It’s the same with IT projects.
When you look at them from afar, particularly
from a very high level, they all look the same.
However, when you look at them up close, you quickly discover that they too
The agile scaling factors
, which are really just general scaling factors applicable to all types of IT
project regardless of paradigm, help to make this very clear.
For example, when it comes to team size some teams
are small, less than ten people perhaps, some are medium sized, and some are very
large (with hundreds of people).
comes to distribution some teams are co-located in the same room, some teams
have team members in different cubicles in the same building, some have people
working in different buildings, and some even have people working in different
Many agile teams work in
regulatory environments, in fact the July 2009 DDJ State of the IT Union survey reports that one third of agile teams must
comply to industry regulations, although clearly many agile teams do not have
this as a concern. That’s only three
scaling factors. The point is that a
small, co-located team working in a non-regulatory environment will work much
differently than a fifty-person team working in three different locations,
which in turn works differently than a two hundred person team in the same
building working in a regulatory situation.
Different teams, facing different scaling issues will work in different
ways – unique snowflakes from a process point of view.
At Agile 2009 in August Sue McKinney, VP of Development Transformation with IBM Software Group, was interviewed by DZone's Nitin Bharti about IBM's experiences adopting agile techniques. There are over 25,000 developers within IBM Software Group alone. Follow the link to the interview
to view it online (there is also a text transcript posted there. There's some great insights into the realities of scaling agile in large teams, in distributed agile development, and in particular how to transform a large organization's development staff.
Modified on by ScottAmbler
When I talk to people about scaling agile techniques, or about agile software development in general, I often put describe strategies in terms of various risks. I find that this is an effective way for people to understand the trade-offs that they're making when they choose one strategy over another. The challenge with this approach is that you need to understand these risks that you're taking on, and the risks that you're mitigating, with the techniques that you adopt. Therein lies the rub, because the purveyors of the various process religions ( oops I mean methodologies) rarely seem to coherently the discuss the risks which people take on (and there's always risk) when following their dogma (oops, I mean sage advice).
For example, consider the risks associated with the various strategies for initially specifying requirements or design. At the one extreme we have the traditional strategy of writing initial detailed speculations, more on this term in a minute, and at the other extreme we have the strategy of just banging out code. In between are Agile Modeling (AM) strategies such as requirements envisioning and architecture envisioning (to name a few AM strategies). Traditionalists will often lean towards the former approach, particularly when several agile scaling factors apply, whereas disciplined agile developers will lean towards initial envisioning. There are risks with both approaches.
Let's consider the risks involved with writing detailed speculations (there's that term again):
You're speculating, not specifying. There is clearly some value with doing some up-front requirements or architecture modeling, although the data regarding the value of modeling is fairly slim (there is a lot of dogma about it though), but that value quickly drops off in practice. However, the more you write the greater the chance that you're speculating what people want (when it comes to requirements) or how you're going to build it (when it comes to architecture/design). Traditionalists will often underestimate the risks that they're taking on when they write big requirements up front (BRUF) , or create big models up front (BMUF) in general, but in the case of BRUF the average is that a large percentage of the functionality produced is never used in practice -- this is because the detailed requirements "specifications" contained many speculations as to what people wanted, many of which proved to be poor guesses in practice.
You're effectively committing to decisions earlier than you should. A side effect of writing detailed speculations is that by putting in the work to document, validate, and then update the detailed speculations the decisions contained in the speculations become firmer and firmer. You're more likely to be willing to change the content of a two-page, high-level overview of your system requirements than you are to change the content of a 200-page requirements speculation that has been laboriously reviewed and accepted by your stakeholders. In effect the decision of what should be built gets "carved in stone" early in the process. One of the principles of lean software development is to defer decisions as late as possible, only making them when you need to, thereby maximizing your flexibility. In this case by making requirements decisions early in the process through writing detailed speculations, you reduce your ability to deliver functionality which meets the actual needs of your stakeholders, thereby increasing project risk.
You're increasing communication risk. We've known for decades that of all the means of communication that we have available to us, that sharing documentation with other people is the riskiest and least effective strategy available to us for communicating information (face-to-face communication around a shared sketching environment is the most effective). At scale, particularly when the team is large or the team geographically distributed, you will need to invest a little more time producing specifications then when the team is co-located, to reduce the inherent risks associated with those scaling factors, but that doesn't give you license to write huge tomes. Agile documentation strategies still apply at scale. Also, if you use more sophisticated tooling you'll find it easier to promote collaboration on agile teams at scale.
You're traveling heavy. Extreme Programming (XP) popularized the concept of traveling light. The basic idea is that any artifact that you create must be maintained throughout the rest of the project (why create a document if you have no intention of keeping it up to date). The implication is that the more artifacts you create the slower you work due to the increased maintenance burden.
There are also risks involved with initial envisioning:
You still need to get the details. Just because you're not documenting the details up front doesn't imply that you don't need to understand them at some point. Agile Modeling includes several strategies for exploring details throughout the agile system development life cycle (SDLC), including iteration modeling performed at the beginning of each iteration as part of your overall iteration planning activities, just in time (JIT) model storming throughout the iteration, and test-driven development (TDD) for detailed JIT executable specification.
You need access to stakeholders. One of the fundamental assumptions of agile approaches is that you'll have active stakeholder participation throughout a project. You need to be able to get information from your stakeholders in a timely manner for the previously listed AM techniques to work effectively. My experience is that this is fairly straightforward to achieve if you educate the business as to the importance of doing so and you stand up and fight for it when you need to. Unfortunately many people don't insist on access to stakeholders and put their projects at risk as a result.
You may still need some documented speculations. As noted previously you may in fact need to invest in some specifications, particularly at scale, although it's important to recognize the associated risks in doing so. For example, in regulatory compliance situations you will find that you need to invest more in documented speculations simply to ensure that you fulfill your regulatory obligations (my advice, as always, is to read the regulations and then address them in a practical manner).
The ways that you approach exploring requirements, and formulating architecture/design, are important success criteria regardless of your process religion/methodology. No strategy is risk free, and every strategy makes sense within given criteria. As an IT professional you need to understand the risks involved with the various techniques so that you can make the trade-offs best suited for your situation. One process size does not fit all.
My final advice is to take a look at the Disciplined Agile Delivery (DAD) framework as it provides a robust strategy for addressing the realities of agile software development in enterprise settings.
Modified on by ScottAmbler
For several years now I've written various articles and newsletters on the topics of estimating and funding strategies for software development projects, and in particular for agile software development projects. Whenever I talk to people about agile software development, or coach them in how to succeed at it, some of the very first questions that I'll be asked, particularly by anyone in a management role, is how to fund agile software development projects. Apparently a lot of people think that you can only apply agile strategies on small, straightforward projects where it makes sense to do a time and materials (T&M) approach. In fact you can apply agile strategies in a much greater range of situations, as the various surveys
that myself and others are showing time and again. My goal with this blog posting is to summarize the various strategies for, and issues surrounding, the funding of agile software development projects.
There are three basic strategies for funding projects, although several variations
clearly exist. These strategies are:
- "Fixed price". At the beginning of the project develop, and then commit to, an initial estimate based on your up-front requirements and architecture modeling efforts. Hopefully this estimate is given as a range, studies have shown that up-front estimating techniques such as COCOMO II or function points are accurate within +/- 30% most of the time although my July 2009 State of the IT Union survey found that on average organizations are shooting for +/- 11% (their actuals come in at +/- 19% on average, but only after doing things such as dropping scope, changing the estimate, or changing the schedule). Fixed-price funding strategies are very risky in practice because they promote poor behavior on the part of development teams to overcome the risks foisted upon them as the result of this poor business decision. It is possible to do agile on a fixed budget but I really wouldn't recommend it (nor would I recommend it for traditional projects). If you're forced to take a fixed-price approach, and many teams are because the business hopes to reduce their financial risk via this approach not realizing that it actually increases their risk, then adopt strategies that reduce the risk.
- Stage gate. Estimate and then fund the project for given periods of time. For example, fund the project for a 3-month period then evaluate it's viability, providing funding for another period of time only to the extent that it makes sense. Note that stages don't have to be based on specific time periods, they could instead be based on goals such as to intiate the project, prove the architecture with working code, or to build a portion of the system. Disciplined agile methods such as Open Unified Process have built in stage-gate decision points which enable this sort of strategy. When the estimation technique is pragmatic, the best approaches are to have either the team itself provide an estimate for the next stage or to have an expert provide a good gut feel estimate (or better yet have the expert work with the team to develop the estimate). Complex approaches such as COCOMO II or SLIM are often little more than a process facade covering up the fact that software estimating is more of an art or a science, and prove to be costly and time consuming in practice.
- Time and materials (T&M). With this approach you pay as you go, requiring your management team to actually govern the project effectively. Many organizations believe a T&M strategy to be very risky, which it is when your IT governance strategy isn't very effective. An interesting variation, particularly in a situation where a service provider is doing the development, is an approach where a low rate is paid for their time which covers their basic costs, the cost of materials is paid out directly, and delivery bonuses are paid for working software. This spreads the risk between the customer/stakeholder and the service provider. The service provider has their costs covered but won't make a profit unless they consistently deliver quality software.
The point is that there are several strategies for funding agile software development projects, just like there are several strategies for funding traditional software development projects. My experience is that fixed-price funding strategies are incredibly poor practice which increases the risk of your software development projects dramatically. I recognize how hard it can be to change this desire on the part of our business stakeholders, but have also had success changing their minds. If you choose to perservere, which is a difficult decision to make, you can help your organization's decision makers to adopt more effective strategies. Like you they want to improve the effectiveness of your IT efforts.Further reading: (In recommended order)
- Something's Gotta Give: Argues for a flexibly approach to funding, schedule, and/or scope.
- Agile on a Fixed Budget: Describes in detail how to take a fixed-price approach on agile projects.
- The Dire Consequences of Fixed-Price IT Projects: Describes in detail the questionable behavior exhibited by IT teams when forced to take a fixed-price approach.
- Is Fixed-Price Software Development Unethical?: Questions the entire concept of fixed-price IT projects, overviewing some of the overwhelming evidence against this really poor practice.
- Reducing the Risk of Fixed-Price Projects: Describes viable strategies for addressing some of the problems resulting from the decision of fixed-price projects.
- Strategies for Funding Software Development Projects: Describes several variations on the strategies described above.
- Lies, Great Lies, and Software Development Project Plans: Summarizes some results from the July 2009 State of the IT Union survey which explored issues related to project funding (among many).
Yesterday I was involved with a workshop around agile development at scale. At one point in the conversation we started talking about the relationship between cost and quality. Some of the people in the workshop were relatively new to agile and still believed the traditional theory that to build in high quality it costs more, sometimes substantially more. This does appear to be true on traditional waterfall projects, but some people were making the mistake that this was an "natural law of IT" which also must apply to agile project teams. I naturally jumped on that idea and described how agile developers have found that writing high quality code leads to lower development costs and shorter time to value, in direct contradiction to traditional theory. A few people struggled with the idea for a bit, and one was pretty adamant that in some cases the need for very high quality does in fact lead to greater cost and time. He talked about his experiences on large-scale Rational Unified Process(RUP)
projects and in particular how some URPS (usability, reliability, performance, and supportability) requirements can increase your cost. At this point Per Kroll, co-author of Agility and Discipline Made Easy: Practices from OpenUP and RUP
, jumped into the conversation and pointed out although higher quality does lead to lower cost in most cases, using Toyota's lean approach to manufacturing as an example, that the agile community didn't completely have the relationship between quality and cost completely correct. My spidey sense told me that a learning opportunity was coming my way.
Per and I had an offline discussion about this to explore what he'd been observing in practice. In most situation it appears to be the case that higher quality does in fact lead to lower costs and shorter time for delivery, something that Per and I had observed numerous times. This happens because high quality code is much easier to understand and evolve than low quality code -- the agile community has found that it is very inexpensive to write high quality code by following practices such as continuous integration
, developer regression testing [or better yet test-driven development(TDD)
], static code analysis
, following common development conventions, and agile modeling strategies
. When you "bake in" quality from the start through applying these techniques, instead of apply traditional techniques such as reviews
and end-of-lifecycle testing (which is still valid for agile projects, but should not be your primary approach to testing) which have long feedback cycles
and therefore prove costly in practice. But, as we've learned time and again, when you find yourself in more complex situations of Agility@Scale sometimes the mainstream agile strategies fall down. For example, in situations where the regulatory compliance scaling factor is applicable, particularly regulations around protecting human life (i.e. the FDA's CFR 21 Part 11), you find that some of the URPS requirements require a greater investment in quality which can increase overall development cost and time. This is particularly true when you need to start meeting 4-nines requirements (i.e. the system needs to be available 99.99% of the time) let alone 5-nines requirements or more. The cost of thorough testing and inspection can rise substantially in these sorts of situations.
In conclusion, it does seem to be true in the majority of situations, which is what the level 1 rhetoric focuses on, that higher quality leads to lower development costs. But at scale this doesn't always seem to hold true.
PS -- Sorry for the corny title, but a couple of days ago at the Rational Software Conference I had the pleasure of interviewing Jamie Hyneman and Adam Savage from the Discovery Channel's Mythbuster's show as part of the conference keynote. They're great guys, BTW, who have had a really positive impact on motivating children to be interested in science (apparently kids like to see stuff get blown up, go figure).[Read More
I recently wrote an "e-book" for Internet Evolution overviewing agile software development at scale. The goal of the Agility at Scale: Become as Agile as You Can Be
ebook is to get people thinking outside of the box a bit when it comes to agile development strategies and see that they really are ready for primetime.
The Scrum community has adopted a different set of terms than the other agile methodologies. This is done on purpose to help people realize that Agile approaches are different than traditional approaches, which can help in their adoption, but it can also hinder people's understanding because some of the terminology is not only non-standard it really doesn't make much sense. Because of this I'm often asked by people that I'm coaching to convert back and forth between terms, and recently wrote a detailed article on the subject. The following summarizes the mapping:
- Daily Scrum Meeting ==> Daily Stand-up Meeting
- Product Backlog ==> Work Item List
- Scrum Master ==> Team Lead or Team Coach
- Sprint ==> Iteration or Time Box
For more details read my article Translating Scrum Terminology
which includes explanations of a wider range of Scrum terms and discussions of why some of them really are questionable. Further reading:
Recently I visited a customer who had adopted Scrum. They were a few sprints, what Scrum calls iterations, into the project and were running into some difficulties. Although I was primarily brought in to educate senior management on disciplined agile software development, I was also asked to sit in on the team’s daily stand-up meeting so that I could hopefully provide some suggestions as to how to address the problems they were running into.
Their work area was fairly typical. They had some whiteboards which they were using for project planning and tracking, with sticky notes to indicate what work had been taken on by each team member. The current status of the task (not yet started, in progress, and completed) was indicated by putting each sticky note in a corresponding column for the status and corresponding column for the team member. This allowed everyone on the team to easily share their status and to see the status of everyone else. On the sides were sketches of the architecture as well as some business oriented models. In addition to Scrum the team had adopted several practices from Agile Modeling, in this case they had done some initial requirements envisioning
and architecture envisioning
, as well as practices from Extreme Programming (XP) for construction. In short, they had followed a fairly common strategy of combining practices from various agile methods.
This would have worked perfectly fine if they had tailored the practices to reflect the situation that they were in, but instead they adopted them "straight out of the book". First, the team was distributed, with most of the team in the location that I was visiting but some people located in two other distant cities. Therein was the source of most of their problems. The people at the other two locations weren’t getting much value out of the daily stand-up meetings, even though they would dial in, because they couldn’t see the project status information. Although people at this location were trying their best to represent these distant people in the daily stand-ups it wasn’t working well – their status information wasn’t being kept up to date and for some people it was a bit of mystery as to what they were actually working on at all.
This team also had 30 people in it, which isn’t a big deal although it can stretch the limits of the simple modeling and planning tools (in this case paper and whiteboards) that they were using. Because the team was larger they were investing a fair bit of time creating burn down charts at both the iteration/sprint and project levels. One of the unfortunate implications of using manual tools for project management is that any associated metric/status reporting in turn becomes manual as well. Considering how the agile community is so concerned with working efficiently, I find it comical that we have a tendency to overlook our own potentially unnecessary bureaucracy such as this.
The problem was that the team was applying strategies, in this case using sticky notes and whiteboards to capture the detailed iteration plan, applying similar strategies to capture key models, and were verbally relaying of status information between sub-teams. There are perfectly fine strategies for smaller co-located teams, but not so good for large or distributed teams. The solution was to recognize that they were in an Agility@Scale situation and needed to tailor their approach to reflect this fact. In this case they needed to forgo some of the manual tools and instead use electronic tooling such as Rational Team Concert (RTC) to share information across disparate locations, in particular the work assignment and corresponding status information. RTC also creates common agile reports such as burn-down charts based on the activities of the developers, providing accurate (nearly) real-time information while removing the burden of status reporting. The RTC project dashboard does more than just this, to see an actual example of one visit www.jazz.net
to see the dashboard for the RTC development team itself. You can also see their actual work item list too, a more advanced version of Scrum’s product and sprint backlogs.[Read More
I just wanted to share with you the Manifesto for Software Craftsmanship
which extends the Agile Manifesto
. The Manifesto for Software Craftsmanship states:As aspiring Software Craftsmen we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:
- Not only working software, but also well-crafted software
- Not only responding to change, but also steadily adding value
- Not only individuals and interactions, but also a community of professionals
- Not only customer collaboration, but also productive partnerships
That is, in pursuit of the items on the left we have found the items on the right to be indispensable.
I view this manifesto as an important step in the maturation of software development. More on this in a future blog posting.[Read More
And now, a brief message from our sponsor. ;-)
Just wanted to let you know that Agility@Scale will be a major theme at this year's IBM Rational Software Conference
being held in Orlando May 31 through to June 4. I'll be giving two presentations, Agility at Scale: Mitigating the Percieved Risks of Agile Adoption
and Distributed Agile Software Development: Best Practices
, facilitating a panel entitled Experience Reports from the Agile Trenches: What's Really Working, What's Not Working, and Why
, and running a half-day workshop entitled Become as Agile as You Need to Be: Scaling Agile Software Development to Meet Your Real-World Situation
. As usual, it will be a busy conference for me. ;-)
There are of course other sessions and workshops specific to agile topics. Some highlights include:
Of course there's a lot more going on at the conference when it comes to scaling agile software development than just these sessions, but I thought I'd give you a feel for what's going on. Hope to see you at the conference. There's a "March Madness" sale going on right now at the registration page
where if you purchase one pass you get the second for half price.Further reading:
Contrary to popular belief, agile development teams do in fact model and yes, they even do some up front requirements and architecture modeling. Two of the best practices of Agile Modeling are Requirements Envisioning
and Architecture Envisioning
where you spend a bit of time at the beginning of the project doing enough initial modeling to get you going in the right direction. The strategy is to take advantage of modeling, which is to communicate and think things through without taking on the risks associated with detailed specifications written early in the lifecycle
. In this blog posting I will focus on requirements envisioning, in a future posting I'll cover architecture envisioning.
The goal of initial requirements envisioning is to identify the scope of your effort. You need to do just enough modeling early in the project to come to stakeholder concurrence and answer questions such as what you're going to build, roughly how long it's going to take (give a range), and roughly how much it's likely to cost (once again, give a range). If you can get the right people together in the room, which can sometimes be a logistics challenge but not one that you couldn't choose to overcome, there are very few systems (I suspect less than 5%) that you couldn't initially scope out in a few days or a week. I also suspect that most of the remaining systems could be scoped out with less than 2 weeks of modeling, and if not then I'd take that as an indication that you're taking on too large of a project. I'm not saying that you'll be able to create big detailed specifications during this period, and quite frankly given the problems associated with "Big Requirements Up Front (BRUF)
" you really don't want to, but I am saying that you could gain a pretty good understanding of what you need to do. The details, which you'll eventually need, can be elicited throughout the lifecycle when you actually need the information. A common saying in the agile community is that requirements analysis is so important for us that we do it every single day, not just during an initial phase. I'll discuss just in time (JIT) approaches to requirements modeling in a future posting.
To envision the requirements for a business application, you might want to consider creating the following models:
- High-level use cases (or user stories). The most detail that I would capture right now would be point form notes for some of the more complex use cases, but the majority just might have a name. The details are best captured on a just-in-time (JIT) basis during construction.
- User interface flow diagram. This provides an overview of screens and reports and how they're inter-related. You just need the major screens and reports for now.
- User interface sketches. You'll likely want to sketch out a few of the critical screens and reports to give your stakeholders a good gut feeling that you understand what they need. Sketches, not detailed screen specifications, are what's needed at this point in time.
- Domain model. A high-level domain model, perhaps using UML or a data modeling notation, which shows major business entities and the relationships between them, can also be incredibly valuable. Listing responsibilities, both data attributes and behaviors, can be left until later iterations.
- Process diagrams. A high-level process diagram, plus a few diagrams overviewing some of the critical processes, are likely needed to understand the business flow.
- Use-case diagram. Instead of a high-level process diagram you might want to do a high-level use case diagram instead. This is a matter of preference, I likely wouldn't do both.
- Glossary definitions. You might want to start identify key business terms now, although I wouldn't put much effort into settling on exact definitions. I've seen too many teams run aground on "analysis paralysis" because they try to define exact terminology before moving forward. Don't fall into this trap.
For small teams simple tools such as whiteboards and paper are usually sufficient for requirements envisioning. But what happens at scale? What if you're working on a large agile team, say of 50 people, 200 people (IBM has delivered software into the marketplace with agile teams of this size), or even 500 people (IBM currently has teams of this size applying agile techniques)? What if your team is distributed? Even if you have people working on different floors of the same building, let alone working from home or working in different cities or countries, then you're distributed (see my postings about distributed agile development
). Suddenly whiteboards and paper-based tools (index cards, sticky notes, ...) aren't sufficient. You're still likely to use these sorts of tools in modeling sessions with stakeholders, but because of one or more scaling factors you need to capture your requirements models electronically.
In January Theresa Kratschmer and I gave a webcast entitled Agile Requirements: Collaborative, Contextual, and Correct
which overviewed agile approaches to requirements elicitation and management, including requirements envisioning. We also showed how Rational Requirements Composer (RRC)
can be used to electronically capture critical requirements information, enabling you to address the needs of large and/or distributed agile teams, while still remaining lightweight and flexible. I suspect that you'll find the webcast to be very illuminating and RRC something that you want to take a look at (the link leads to a trial version). Of course RRC can be used in other situations as well, but that's not what I'm focused on right now.
Teams which find themselves in regulatory environments will likely need to do more than just use RRC, as might very large teams. Regulatory compliance often requires more complex requirements documentation, which in turn requires more sophisticated tools such as DOORS or Requisite Pro, and I would consider using those tools in the types of situations that warrant it. One of the things that people often struggle to understand about agile approaches is that you need to tailor your strategy to reflect the situation at handle. One process size does not fit all, so you will end up using different tools and creating different artifacts to different extents in different situations. Repeatable results, not repeatable processes
, is the rule of the day. Further reading:
When adopting agile software development
techniques across a large number of teams within your organization it is important to provide a definition for what agile software development is, in addition to criteria
for what it means to be agile. Many people will point to the four values of the Agile Manifesto
and claim that's a good definition. Well... it might be a good definition for the visionaries and early adopters among us, but for people on the right-hand side of the technology adoption chasm (the early majority, late majority and the laggards) this isn't enough. Don't get me wrong, I'm a firm believer in the agile values but I like to cast them as philosophies instead of as a definition.
At IBM Software Group, the definition of disciplined agile software delivery which we have been sharing with our customers is:Disciplined agile software delivery is an evolutionary (iterative and incremental) approach to delivery which regularly produces high quality software in a cost effective and timely manner. It is performed in a highly collaborative and self-organizing manner, with active stakeholder participation to ensure that the team understand and addresses the changing needs of its stakeholders. Disciplined agile delivery teams provide repeatable results by adopting just the right amount of ceremony for the situation which they face.
I think that this is a pretty good definition, although I have no doubt that we'll evolve it over time.
I also suspect that the agile community will never settle on a common definition for what agile is and more than likely are smart enough not to even try. ;-)Further reading:
I've been getting a lot of questions lately about applying the acceleration metric
in practice. So, here's some answers to frequently asked questions:
1. How do I monetize acceleration?
This is fairly straightforward to do. For example, assume your acceleration is 0.007 (0.7%), there are five people on the team, your annual burdened cost per person is $150,000, and you have two week iterations. All these numbers are made up, but you know how to calculate acceleration now and IT management had darn well better know the average burdened cost (salary plus overhead) of their staff. So, per iteration the average burdened cost per person must be $150,000/26 = $5,770. Productivity improvement per iteration for this team must be $5,770 * 5 * .007 = $202. If the acceleration stayed constant at 0.7% the overall productivity improvement for the year would be (1.007)^26 (assuming the team works all 52 weeks of the year) which would be 1.198 or 19.8%. This would be a savings of $148,500 (pretty much the equivalent of one new person). Of course a 20% productivity increase over an entire year is a really aggressive improvement, regardless of some of the claims made by the agile snake oil salesman out there, although at 10-15% increase is a reasonable expectation. What I'd really want to do is calculate the acceleration for the year by comparing the velocity from the beginning of the year to the end of the year (in Western cultures I'd want to avoid comparing iterations near to the holidays). So, if the team velocity the first week of February 2008 was 20 points, now the same team's velocity the first week of February 2009 was 23 points, that's an acceleration of (23-20)/20 = 15% over a one year period, for a savings of $112,500.
2. Is acceleration really unitless?
For the sake of comparison it is. The "units" are % change in points per iteration, or % change in points per time period depending on the way that you want to look at it. Because it's a percentage I can easily monetize it, as you see above, and use it as a basis of comparison.
3. How do I convince teams to share their data?
This can be difficult. Because acceleration is easy to calculate for agile teams, and because it's easy to use to compare teams (my team has .7% acceleration whereas other teams down the hall from mine have accelerations of .3% and -.2% of teams), people are concerned that this metric will be used against them. OK, to be fair, my team might be OK with this. ;-) Seriously though, this is a valid fear that will only be addressed by an effective governance program
based on enablement, collaboration, and trust instead of the traditional command-and-control approach. Management's track record regarding how they've used measurements in the past, and how they've governed in general, have a great effect on people's willingness to trust them with new metrics such as acceleration. The implication is that you need to build up trust, something that could take years if it's possible at all.
4. Why does this work for agile teams?
Agile teams are self organizing, and an implication of that is that they will be held accountable for their estimates. Because of this accountability, and because velocity is a vital input into their planning and estimation efforts, agile teams are motivated to calculate their velocity accurately and to track it over time. Because they're eager to get their velocity right, and because acceleration is based on velocity, there's an exceptionally good chance that it's accurate.
5. What about function points or similar productivity measures?
Function points can be calculated for projects being developed via an agile approach, or other approaches for that matter, but it's a very expensive endeavor compared to calculating acceleration (which is essentially free) and likely will be seen as a bureaucratic overhead by the development team. My rule of thumb is that if you're not being explicitly paid to count function points (for example the US DoD will often pay contracting companies to create estimates based on function point counts) then I wouldn't bother with them.
6. What about calculating acceleration for iterative project teams?
Iterative project teams, perhaps following Rational Unified Process (RUP)
, can choose to calculate and track their velocity and thereby their acceleration. The key is to allow the team to be self organizing and accountable for their estimates, which in turn motivates them to get their velocity right just like agile teams (RUP can be as agile as you want to make it, don't let anyone tell you differently).
7. What about calculating acceleration for traditional project teams?
In theory this should work, in practice it is incredibly unlikely. Traditional teams don't work in iterations where working software is produced on a regular basis, they're typically not self organizing, and therefore there really isn't any motivate to calculate velocity (even if they do, there is little motivation to get it right). Without knowing the velocity you can't calculate acceleration. If you can't trust the velocity estimate, and I certainly wouldn't trust a traditional team's velocity estimate, then you can't trust your acceleration calculation. So, my fall back position to calculate productivity improvement would be to do something like function point counting (which is expensive and difficult to compare between teams due to different fudge factors used by different FP counters) and then looking at change in FPs delivered over time.
8. How can I apply this across a department?
It is fairly straightforward to roll up the acceleration of project teams into an overall acceleration measure for a portfolio of teams simply by taking a weighted average based on team size. However, this is only applicable to teams that are in a position to report an accurate acceleration (the agile and iterative teams) and of course are willing to do so.
9. What does a negative acceleration tell me?
If the acceleration is negative then productivity on the team is going down, likely an indicator of quality and/or team work problems. However, you don't want to manage by the numbers so you should talk to the team to see what's actually going on.
10. What does a zero acceleration tell me?
This is an indication that the team's productivity is not increasing, and that perhaps they should consider doing retrospectives at the end of each iteration and then acting on the results from those retrospectives. Better yet they can "dial up" their process improvement efforts by adopting something along the lines of IBM Rational Self Check
As part of the Agile/Jazz event In Milwaukee on the 29th I'll be giving a talk about the realities of agile software development. The URL for the event is http://www.iconatg.com/lp/events/jazz/jazz_milwaukee.html and I hope to see you there.
- Scott[Read More
Although it might not be obvious, and important success factor in adopting agile techniques
is to be able to determine whether a team is agile or not. The challenge that many organizations face is that many teams will claim to be agile, yet management, who often has little or no experience with agile approaches, cannot tell which claims are true and which are over zealous (I'm being polite). The following are the criteria that I suggest you look for in a disciplined agile team:1. Produce working software on a regular basis
. This is one of the 12 principles behind the Agile Manifesto
, and in my experience is a critical differentiator between the teams that are agile and those that are merely claiming it. Ideally the team should produce potentially shippable software each iteration. That doesn't mean that they'll deploy the system into production, or the marketplace, each iteration but they could if required to do so. Typically the team will deploy into a pre-production testing environment or a demo enviroment at the end of each iteration (or more often for that matter).2. Do continuous regression testing, and better yet take a Test-Driven Development (TDD) approach
. Agile developers test their work to the best of their ability, minimally doing developer regression testing via a continuous integration (CI) strategy
and better yet do developer-level TDD
. This approach enables development teams to find defects early, thereby reducing the average cost of addressing the defects, it also helps them to deliver higher quality code and to move forward safely when adding or changing functionality.3. Work closely with their stakeholders, ideally on a daily basis
. A common practice of agile teams is to have an on-site customer or product owner who prioritizes requirements and provides information on a timely manner to the team. Disciplined agile teams take it one step further and follow the practice active stakeholder participation
where the stakeholders get actively involved with modeling and sometimes even development.4. Are self-organizing within a governance framework
. Agile teams are self-organizing, which means that the people doing the work determines how the work will be done, they're not told by a manager who may not even be directly involved with the work how it will be done. In other words the team does its own planning, including scheduling and estimation. Disciplined agile teams are self governing within an effective governance framework
.5. Regularly reflect on how they work together and then act to improve on their findings
. Most agile teams hold a short meeting at the end of each iteration to reflect upon how well things are working and how they could potentially improve the way that they are working together. Sometimes this is done in a more formalized manner in the form of a retrospective
, but often it's done informally. The team then acts on one or more of their suggested improvements the next iteration. Disciplined agile teams take this one step further and measure their software process improvement (SPI) progress over time: the act of taking these measures, perhaps via a product such as Rational Self Check
, helps to keep the team on track in their SPI efforts.
I have yet to discover an ad-hoc development team which met all five criteria, and most of them rarely meet two or three.Further reading:
And now for some blatant advertising. ;-)
I just wanted to point out our agile development
page to you as it contains links to a lot of interesting agile resources produced by the folks here at Rational. There's links to white paper, recorded web casts, and Rational's Agility@Scale poster. Short story is that the page is worth checking out.
- Scott[Read More
Again and again I've seen IT organizations suffering from what I call the "Bureaucracy is Discipline" antipattern. For example, filling out forms and reviewing documents are both bureaucratic activities, neither of which require significant skill nor discipline to accomplish. However, agile practices such as developing potentially shippable software every iteration is easy to say but requires great discipline to accomplish. Respecting the decisions of your stakeholders, particularly those pertaining to requirements prioritization, is easy to talk about but proves to require great discipline in practice (particularly when you don't agree with a decision). It's easy to talk about taking a test-driven approach to development
, but in practice it requires significant skill and discipline to actually do.
A "process smell" which indicates that your organization is suffering from this antipattern is a focus on following repeatable processes instead of focusing on repeatable results. An example of repeatable processes is following the same route to work every day regardless of driving conditions. An example of repeatable results is getting to work on time every day, but being willing to change your route as required, bicycling into work instead of driving, taking public transit, and so on. Nobody really cares how you get to work each day (the process), what they really care about is that you got to work on time (the result). Sadly, we've been told for decades now that repeatable processes are critical to our success in IT, yet when you step back and think about that's really a reflection of a bureaucratic approach. On the other hand, a focus on repeatable results is a reflection of a more disciplined approach. Interestingly, the DDJ 2008 Process Framework survey
found that given the choice that people would much rather have repeatable results over repeatable processes
when it comes to IT.
Mistaking bureaucracy for discipline, or rigour if you prefer that term, is a reflection of the cultural damage that has occurred over the years in IT organizations as the result of traditional philosophies and techniques. Unfortunately, this mistaken belief is a significant inhibitor to software process improvement (SPI) efforts, in particular agile adoption efforts
, which must be addressed if you're to be successful. Overcoming this challenge will require a significant cultural shift in some organizations, and many people (particularly the bureaucrats) will find this uncomfortable.Further reading:
I'd like to leave you with this parting thought: Bureaucracy is bureaucracy and discipline is discipline, please don't confuse the two
I'm often asked by customers for case studies of successful agile adoptions or agile projects in general. This is definitely a valid request, and yes, such case studies exist. But I'm often concerned that the people making these requests don't appreciate the implications of what they're asking for. My concerns with case studies are:
- The juicy information is rarely included. The information that you really want to find out, such as what went wrong and why it went wrong, is rarely discussed. If problems, oops I mean "challenges", are discussed at all they're typically glossed over in favor of focusing in on the positives. Although many people want to write up the juicy bits this information is invariably edited out through the company's vetting process. In short, my advice is to take case studies with a grain of salt.
- Some case studies are more fiction than fact. Although this isn't a problem with IBM case studies due to the governance efforts of my good friends in IBM's legal department (we love you folks, really) it can be an issue with some case studies.
- The case study may no longer be true today. Stuff happens. Perhaps the case study was mostly true at the time it was written, but now that time has passed problems have appeared that weren't apparent earlier, thus the effort wasn't as nearly as successful as it was written up. For example, a few years ago I ran into the manager of a team that I had read about in one case study, only to find out that once the study was published the key team members left the company to become consultants in that subject area. Having lost these people, who were all very highly skilled, his system proved to be unmaintainable by the rest of his staff who weren't so highly skilled and had to be rewritten. Over time the success story turned into an abject failure.
- Waiting for case studies puts you in the position of follower. For every case study that gets written, dozens, if not hundreds of similar efforts didn't get written up. Writing case studies is hard, takes time, and the writer seldom gets much benefit from doing so. The lag time between the project completing and the case study being published can be many, many months and sometimes years. The implication is that by the time you wait for several case studies that are similar to your situation you've pretty much lost all opportunity for competitive advantage and are now merely trying to catch up to the organizations who are clearly ahead of you (the writers of the case studies).
- What has the requester given back to the community? I often hear people lament that there isn't enough case studies, or isn't something close enough to their situation. Yet, when I ask them how many case studies they've written and the answer is usually none. If you want to get you also need to give. ;-)
So, next time you think you need a case study before making a decision, recognize that you may be paying a fairly high opportunity cost for information that is questionable at best.Further reading:
I'm happy to announce Rational's Agility at Scale poster which is currently available free of charge from the Rational poster order page
Although I'm obviously biased as I was involved with its creation, the poster includes some really good information about how to scale agile software development practices effectively. It summarizes the scaling factors, such as large teams, distribution, regulatory compliance, application complexity, and so on that you should be concerned with. It also presents a full agile software development lifecycle that goes beyond the construction focused lifecycles of common "agile in the small" processes. It also summarizes the practices of lean development governance, giving you some insight into how to govern your IT environment more effectively.
I think it's a pretty good poster that's worth checking out. It's free, so it's hard to argue with the price. Most importantly, it would look pretty good hanging on your office wall.
A common misunderstanding about agile software development approaches are that they're only applicable to small, co-located teams. Yes, it's much easier to be successful with small teams, and with co-located teams, and as a result agilists being smart people prefer to work this way. After all, why take on extra risk when you don't need to do so? But, sometimes reality gets in the way and you find yourself in a situation where you need a large team, or you need to distribute your team (see previous blog postings for strategies for distributed agile development), and you would still like to be as agile as possible. The good news is that it's still possible to be agile with a large team, although you will need to go beyond some of the popular "agile in the small" strategies to succeed.
Here are some disciplined agile strategies to succeed at large-team agile:
- Question the need for a large team. Many times an organization will believe that they need a large team because their process is overly complex, because they're still organized for waterfall development, or simply because that's what they're used to. I've seen teams of 80 people doing the work of 20 as the result of over-specialization of job roles and all the bureaucracy required to organize and validate their work.
- Do some initial envisioning. In order to succeed the team must work together towards the same goals. This is true for small teams but doubly true for larger ones -- without a common vision chaos will quickly ensue. You must gain this common vision on two fronts: you need a common business vision and a common technical vision. To gain the common business vision you must do some initial, high-level requirements envisioning and to gain the common technical vision some common architecture envisioning. This isn't to say that you need to take on the risk of detailed, up-front specifications but you must at least have a high-level understanding of the scope and technical solution in order to move forward effectively. So, expect to spend the first few weeks of your project doing this initial modeling.
- Divide and conquer. You never have a team of 200 people, instead you have a collection of subteams that add up to 200 people. This is called having a team of teams.
- Align team structure with architecture. The most effective way to organize the subteams is to have each one implement one or more components, and thereby to build your overall system as a "system of systems". This reduces the coordination required because the majority of the communication will be within the subteams themselves. You'll still need to coordinate the subteams, that will never go away, but you can reduce the overhead (and the risk) by being smart about the way that you organize the people. A common mistake is to organize around job function (e.g. having architects in Toronto, developers in Raleigh, testers in Bangalore, and so on). This increases communication overhead and risk because these people need to work together closely to get something built.
- Project management coordination. Each subteam will have a team lead/coach, and these people will need to coordinate their work. There is often an overall project manager who leads this group. To coordinate the work within their subteam the team lead/coach will often have a daily meeting, in the Scrum method this is called a scrum meeting, where people share their current status and identify any problems they may be running into. To scale this effectively the team lead/coach attends a daily team coordination meeting, a scrum of scrums, where the same sort of information is shared at the overall team level.
- Product owner coordination. Similarly, each subteam has a product ownder, also referred to as an "on-site customer", who is responsible for making decisions about the requirements and for providing information to the team in a timely manner. Sometimes a single product owner will work with several subteams. The product owners will get together at the beginning of the project to do some requirements envisioning to identify the initial scope and to start portioning the requirements between the subteams. Because the requirements between the subsystems are interrelated and should be reasonably consistent, the product owners will need to meet on a regular basis to share information, to negotiate priorities, and to resolve requirements-related disputes.
- Architecture coordination. Each subteam will have an architecture owner, often a senior technical person and sometimes also in the role of the team lead/coach. These architecture owners will get together at the beginning of the project to do some initial architecture envisioning, based on the requirements envisioning efforts of the product owners. They will identify the major subsystems, and their interfaces, enabling the effective organization of the team into smaller subteams corresponding to the architecture. They will also get together regularly to evolve the architecture and to resolve any major technical issues.
- System integration team. For complex systems, which is often what large teams work on, an effective system integration effort is critical to your success. Although this may be easy at first, as the overall system evolves the need for a subteam focused solely on this quickly becomes apparent. This not only supports the development efforts of the subteams, it also supports independent investigative testing.
- Independent testing team. An independent testing team is common on mid-to-large size agile projects to enhance the testing efforts of the development subteams. This testing team will work in parallel to the developers, they get a new build on a regular basis (minimally each iteration, although more often is desirable), which they test in more advanced ways than what is typical with Test-Driven Development (TDD). For example, they often validate non-functional, quality of service (QoS) type requirements as well as technical constraints, things that often aren't captured well via user stories. They'll also do investigative testing to try to break the system by using it in ways not thought of by the product owners.
- Some specialties reappear. On larger teams it can make sense to have some people be a bit more specialized than what we normally see on small agile teams. For example, it's common to see people in the role of agile DBA, tech writer, build master, or user experience (UE) professional. More complex systems often require people in these roles, although it still behooves these poeple to not be pure specialists but instead to be generalizing specialists with a wider range of skills. Also, recognize that the reintroduction of specialists can be a slippery slope back to the bureaucracy of traditional software development.
Modified on by ScottAmbler
I just wanted to round out my discussion about agile approaches to geographically distributed development (GDD) with a few important words of advice:1. Get some experience. Worry less about enterprise adoption and instead get started with a small project, or better yet a series of increasingly more complex projects. There will be learning experiences as you build a relationship with the offshore service provider. This advice is applicable whether you’re working with your own offshore division or with an independent service provider.2. Have a long-term staffing strategy. It may be great in the short term to have work done in a lower cost country, but how are you going to transfer the necessary skills to the maintenance and support team. Outsourcing that work is also an option, but it can be a risky one as you would need to build up expertise in “your” systems if you ever decide to insource that work again.3. Be concerned about intellectual property (IP). The rules are different around the world, and you may inadvertently be financing the creation of a new international competitor if you don’t have a clear division of ownership. And yes, this may mean that some components of your systems are still built internally by your own organization.4. Show off locally before you go global. GDD makes things harder to manage, so if you’re struggling to manage local teams you’re really going to struggle managing teams at a distance. Make sure you have local success first and are good at agile development in general. Furthermore, if your agile GDD projects run into trouble, don’t end your local agile adoption just because of difficulties with distributed projects.5. Let your offshore partners lead. The offshore partner likely has more experience than you at successful distributed development, and this is particularly true when you’re dealing with an established service provider.6. Do some reading. There’s a great IBM Redbook entitled “Global Development and Delivery in Practice: Experiences of the IBM Rational India Lab” which can be downloaded free of charge from http://www.redbooks.ibm.com/abstracts/sg247424.html7. Do some viewing. We recorded a Rational Chat a few months ago entitled "Being Agile in a Global Development Environment" which is posted at https://www14.software.ibm.com/webapp/iwm/web/reg/acceptSignup.do?lang=en_US&source=dw-c-wcsdpr&S_PKG=120607&S_TACT=105AGX23&S_CMP=TALKS&cp=UTF-8 . I also gave a keynote on Agile approaches to GDD at Software Development Practices 2007 held in Boston in the Autumn of 2007. The video can be downloaded free of charge from http://www.life20.net/video/scottambler.mov .[Read More
Modified on by ScottAmbler
In my previous blog posting, http://www.ibm.com/developerworks/blogs/page/ambler?entry=strategies_for_distributed_agile_teams , I overviewed several strategies for improving your effectiveness at geographically distributed development (GDD). Those strategies were fairly generic and directly applicable to both traditional and agile development teams. In this posting I focus on strategies which are more agile in nature, although they could also be applied to more traditional approaches as well. These strategies are:
1. Get the whole team together at the beginning of the project. Your goals are to build rapport amongst the team, to get to know the people that you’re working with to facilitate communication later on, and to better understand the situation on the ground. The implication is that you will need to fly some people around, increasing your initial expenses, an investment that many organizations balk at. The reality is that you will eventually end up paying for travel anyway, either because you actually flew people around or because your communication costs are higher throughout the project. In short, don’t be penny wise and pound foolish.
2. Organize your team around the system architecture. The most effective way to organize a distributed team is around the architecture of the system that you are building, not around the job functions of the people involved. In other words, if your team is in Toronto, Rome, and Bangalore then each subteam should be responsible for one or more subsystems. It would be a mistake to organize the teams around job function, for example to have the architects and analysts in Toronto, the developers in Rome, and the testers in Bangalore because this structure would require significantly more documentation and other forms of communication to coordinate the teams, increasing both cost and risk. As I mentioned in my previous blog posting you will need to invest in some initial architecture envisioning at the beginning of a project to identify the subsystems and their public interfaces, and that to do that you’ll also need to do some initial requirements envisioning to drive this architecture effort. I suggest that you take an Agile Model Driven Development (AMDD) approach to this to enable you to gain the value from modeling without the costs and risks associated with up-front comprehensive modeling and documentation that get many traditional project teams in trouble.
3. Have “daily stand-up meetings”. A common practice on co-located agile teams is to have daily stand-up meetings where people share the status of what they did yesterday, what they intend to do today, and whether they’re running into any problems. These short meetings enable team coordination. Distributed teams can do this as well, the people in a given geographical location can hold local stand-up meetings and then representatives from each location can hold a shared meeting to coordinate the subteams. Whereas local stand-up meetings are held first thing in the morning, distributed daily stand-up meetings may need to be held at unusual times so as to include people at distant locations.
4. Have Ambassadors. Ambassadors are people who travel between sites, often technically senior people or senior business experts, to share information between the subteams. Getting the team together at the beginning of the project sets the foundation for communication, but without continual investment in maintaining effective collaboration between teams you run the risk of your subteams deviating from the overall strategy. These are typically short engagements, a week or two in length, because of the pressures it puts on the people doing the actual traveling. The implication is that you’ll have several people flying between sites at any given time on a reasonable rotation schedule. Because you’ll have some people flying around, your local team rooms should accommodate visitors by having one or more desks available for them to use when they’re visiting.
5. Have Boundary Spanners. A boundary spanner is someone who is located on site who focuses on enabling communication between subteams as well as within their subteam. On large distributed teams you’ll find that you have three flavors of boundary spanners – team leaders who take on project management responsibilities on the subteam, product owners who are responsible for representing the business within the subteam, and architecture owners responsible for technical direction on the team. These boundary spanners will work closely with their peers, having regular coordination meetings across all subteams as well as impromptu one-on-one meetings to deal with specific issues between individual subteams.
6. Ensure that the global team gets the credit it deserves. In both offshoring and nearshoring environments it’s common to see small teams in North America or Europe driving the efforts of significantly larger teams in another country. Yet, at the end of the project it always seems as if the smaller team, often because they work for the direct customer, gets the lion’s share of the credit – unless of course the project failed, then the subcontracting team often seems to get virtually all of the “credit”. This clearly isn’t fair, and it clearly doesn’t promote effective teamwork between the subteams in the future.
7. Take a lean approach to development governance. As I’ve written in the past, effective governance is based on enablement and collaboration instead of the traditional approaches of management and control. Good governance measures progress through regular delivery of working software, not through status reports or delivery of detailed specifications. Good governance is based on the idea of having a living process which changes to reflect lessons learned as your project progresses. In a previous blog posting at http://www.ibm.com/developerworks/blogs/page/ambler?entry=lean_development_governance I’ve discussed lean development governance in greater detail.
The strategies that I’ve described are clearly nothing more than common sense, something that can be said of all agile strategies. Sadly, as Mark Twain lamented, common sense isn’t very common in practice.[Read More
Modified on by ScottAmbler
A common misunderstanding about agile software development is that it’s only for co-located teams. Things are definitely easier for co-located teams, and as I found with both the Dr. Dobb’s 2007 and 2008 Agile Adoption surveys (www.ambysoft.com/surveys/) co-located agile teams appear to have a higher success rate than distributed teams, Having said that, many organizations are in fact succeeding at distributed agile development.
I’d like to share some strategies that I’m seeing work in practice, and in this blog posting summarizes generic strategies for distributed teams whether or not they’re agile. These strategies are:1. Do some up front planning. Distributed development is higher risk than co-located development, and one way to address that risk is to think things through. That doesn’t mean that you need to create a monolithic, 1000+ task Gantt chart, but it does mean that you should identify your major dependencies and milestone dates. Effective teams do this planning with the distributed developers actively involved (they are part of the team after all), they strive to consider all associated costs, and in particular they don’t overlook the low probability/high impact risks which often prove to be project killers.
2. Organize the team effectively. Once of the practices of Lean Development Governance (https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?lang=en_US&source=swg-ldg) is to organize your team structure around either your architecture or the lines of business (LOB) supported by the programme that you’re working on. Ideally each sub-team should be responsible for one or more subsystems or modules, something that can be difficult if some of your team works alone from home, to reduce the amount of information sharing and collaboration required between disparate teams. In other words, maximize the responsibilities of the “offshore” team(s) as much as possible. A very common mistake is to organize the subteams around job specialties – for example the architects are in Toronto, the developers in Mumbai, and the testers in Singapore – because to support this team structure you have to create a phenomenal amount of documentation to support communication between the teams.
3. Do some up front modeling. The implication of organizing your team around the architecture (or LOB) is that you also need to do a bit of architecture envisioning up front. Your architecture efforts should provide guidance regarding the shared infrastructure as well as critical development conventions such as coding guidelines and data naming conventions. Architecture envisioning is also a good idea for co-located agile teams too. See http://www.agilemodeling.com/essays/initialArchitectureModeling.htm for strategies to get the benefits of architecture modeling without the costs of needless documentation.
4. Recognize that communication is critical. GDD puts many barriers to communication in place, increasing overall project risk. To overcome these risks you will first need to be aware of them and act accordingly, and second, you’ll need to write more documentation than you would likely prefer. The risks associated with long-distance communication include cultural differences, time-zone differences, and the challenges with written documentation (which is the least effective way to communicate information). I make it a habit of asking open-ended questions so that I can determine whether or not the other people understand the topic under conversation. Particularly I will never ask a yes/no style of question because the simple answer of yes can mean a range of things depending on the culture. It may mean “Yes, I heard you”, “Yes, I understand what you’re saying”, or “Yes, I understand and agree with you”. When you’re dealing with people at other locations it’s good practice to ask them to summarize the conversation in writing, in particular to identify key action items and ownership of them, to ensure that everyone agrees with what was discussed. A good approach is to have the team lead on other end to do the summary so that they own it going forward.
5. Put a good technical infrastructure in place. Automate, automate, automate. In a GDD environment you need to work with collaborative multi-site tools such as ClearCase, ClearQuest, and Jazz Rational Team Concert (www.jazz.net) which enable you to share and evolve your work products (i.e. test scripts, code, documents) effectively.
In my next posting I'll describe a collection of agile-specific strategies for distributed software development teams.[Read More
The explicit phases of the Unified Process -- Inception, Elaboration, Construction, and Transition -- and their milestones are important strategies for scaling agile software development to meet the real-world needs of modern organizations. Yes, I realize that this is heresy for hard-core agilists who can expound upon the evils of serial development, yet these very same people also take a phased approach to development although are loathe to admit it. The issue is that the UP phases are like seasons of a project: although you'll do the same types of activities all throughout a project, the extent to which you do them and the way in which you do them change depending on your goals. For example, at the beginning of a development project if you want to be effective you need to do basic things like identify the scope of the project, identify a viable architecture strategy, start putting together your team, and obtain support for the project. Towards the end of a project your focus is on the activities surrounding the deployment of your system into production, including end-of-lifecycle testing efforts, training, cleaning up of documentation, piloting the system with a subset of users, and so on. In between you focus on building the system, including analysis, design, testing, and coding of it. Your project clearly progresses through different phases, or call them seasons if the term phase doesn't suit you, whether your team is agile or not.
The UP defines four phases, each of which address a different kind of risk:1. Inception. This phase focuses on addressing business risk by having you drive to scope concurrence amongst your stakeholders. Most projects have a wide range of stakeholdres, and if they don't agree to the scope of the project and recognize that others have conflicting or higher priority needs you project risks getting mired in political infighting. In the Eclipse Way this is called the "Warm Up" iteration and in other agile processes "Iteration 0".2. Elaboration. The goal of this phase is to address technical risk by proving the architecture through code. You do this by building and end-to-end skeleton of your system which implements the highest-risk requirements. Some people will say that this approach isn't agile, that your stakeholders should by the only ones to prioritize requirements. Yes, I agree with that, but I also recognize that there are a wide range of stakeholders, including operations people and enterprise architects who are interested in the technical viability of your approach. I've also noticed that the high-risk requirements are often the high-business-value ones anyway, so you usually need to do very little reorganization of your requirements stack.3. Construction. This phase focuses on implementation risk, addressing it through the creation of working software each iteration. This phase is where you put the flesh onto the skeleton.4. Transition. The goal of this phase is to address deployment risk. There is usually a lot more to deploying software than simply copying a few files onto a server, as I indicated above. Deployment is often a complex and difficult task, one which you often need good guidance to succeed at.
Each phase ends with a milestone review, which could be as simple as a short meeting, where you meet with prime stakeholders who will make a "go/no-go" decision regarding your project. They should consider whether the project still makes sense, perhaps the situation has changed, and that you're addressing the project risks appropriately. This is important for "agile in the small" but also for "agile in the large" because at scale your risks are often much greater. Your prime stakeholders should also verify that you have in fact met the criteria for exiting the phase. For example, if you don't have an end-to-end working skeleton of your system then you're not ready to enter the Construction phase. Holding these sorts of milestone reviews improves your IT governance efforts by giving senior management valuable visibility at the level that they actually need: when you have dozens or hundreds of projects underway, you can't attend all of the daily stand up meetings of each team, nor do you even want to read summary status reports.
These milestone reviews enable you to lower project risk. Last Autumn I ran a survey via Dr. Dobb's Journal (www.ddj.com) which explore how people actually define success for IT projects and how successful we really were. We found that when people define success in their own terms that Agile has a 71% success rate compared with 63% for traditional approaches. Although it's nice to that Agile appears to be lower risk than traditional approaches, a 71% success rate still implies a 29% failure rate. The point is that it behooves us to actively monitor development projects to determine if they're on track, and if not either help them to get back on track or cancel them as soon as we possibly can. Hence the importance of occasional milestone reviews where you make go/no-go decisions. If you're interested in the details behind the project, they can be found at http://www.ambysoft.com/surveys/success2007.html .
Done right, phases are critical to your project success, particularly at scale. Yes, the traditional community seems to have gone overboard with phase-based approaches, but that doesn't mean that we need to make the same mistakes. Let's keep the benefit without the cost of needless bureaucracy.[Read More
A common question that I keep running into with customers is whether you can take an agile approach to service oriented architecture (SOA). The quick answer is yes, because Agile is orthogonal to the implementation technologies used. You can take an agile approach developing COBOL applications running on mainframes, fat-client Java applications, multi-tier J2EE applications, and yes, even services. Granted, it's easier to do with some technologies than others, either because of the nature of the technology or because of the supporting tools.
The long answer is "yes, but". You don't adopt an SOA approach for the sheer joy of doing so, instead you very likely want to improve the level of reuse within your organization. To succeed at SOA-driven reuse you need an enterprise focus, something that doesn't appear to be very common on many agile teams. Therein lies the challenge. Several strategies for improving your chances with Agile SOA, and SOA in general, follows:1. Invest in some initial enterprise architecture modeling. You don't need to identify all of the details up front, that would take too long and actually put the effort at risk, but you do need to set a starting point to guide development teams. Identifying the technical architecture is critical, and identifying a few basic services which would provide immediate business value to one or more teams is critical. Involve people from several application project teams to ensure that you get a wide range of input. See http://www.agiledata.org/essays/enterpriseArchitecture.html for a streamlined approach to enterprise architecture modeling. Creating big, detailed models often proves to be a waste of time because development teams are rarely motivated to read mounds of documentation.2. Build out the initial infrastructure on a real application development project. This proves that your SOA strategy actually works and puts the technical foundation in place for future teams. During this period you'll be tempted to try to support several development teams, which is feasible but dramatically increases your risk. It's also tempting to focus simply on getting the infrastructure in place without delivering any business functionality, but this risks producing an ivory-tower architecture that nobody is interested in.3. Spread the service architects out onto application development teams. The people that formulated and then proved your SOA should be actively involved on the development teams that are working with it to ensure that the teams use it appropriately and to ensure that the architects get concrete feedback which they can use to evolve the architecture. When working on agile teams, these people will need to work in a collaborative and evolutionary approach just like other team members.4. Fund reuse separately. I've lost track of the number of organizations that I've run into that fail at reuse because their development teams never have the resources to develop reusable assets. That's simply the nature of the beast -- project teams will always be more interested in addressing their own specific requirements than they are in investing the time and effort to make something reusable. The real problem here is that you expect them to act differently. A better strategy is to have a separate reuse engineering team that has the resources to monitor existing projects to look for potentially reusable assets. When they find said assets this team does the work to harvest the asset, to reengineer it to make it reusable, and then to integrate back into the original source project. The goal is to make it as painless as possible to produce reusable assets such as services. If you expect project teams to do this work out of the goodness of their hearts then you're effectively punishing them when they do the right thing. That's not a very good governance strategy, IMHO.5. The reuse team now owns the asset. Any reusable asset, including services, will need to be maintained, evolved over time, and supported. This isn't free nor is it viable for project teams to do so.
If you're interested, I provide agile strategies for both enterprise architecture and strategic reuse in the book "Enterprise Unified Process". Although written under the assumption that you're taking a RUP-based approach to development, the reality is that the EUP can extend any evolutionary/agile software development process so that it addresses the larger-scale needs of modern IT organizations.
- Scott[Read More
During 2007 Per Kroll and myself invested a significant amount of time development a framework for lean development governance. This effort resulted in a series of three articles that were published in Rational Edge and a recently published white paper. The articles go into the various practices in detail whereas the paper provides an overview aimed at executives. I also recently did a webcast which is now available online. The URLs are at the bottom of this blog posting.
Development governance isn’t a sexy topic, but it critical to the success of any IT department. I like to compare traditional, command-and-control approaches to governance to herding cats – you do a bunch of busy work which seems like a great idea in theory, but in the end the cats will ignore your efforts and stay in the room. Yet getting cats out of a room is easy to accomplish, as long as you know what motivates cats. Simply wave some fish in front of their noses and you’ll find that you can lead them out of the room with no effort at all. Effective governance for lean development isn’t about command and control. Instead, the focus is on enabling the right behaviors and practices through collaborative and supportive techniques. It is far more effective to motivate people to do the right thing than it is to try to force them to do so.
This framework is based on the philosophical foundation provided by the 7 principles proposed in the book “Lean Software Development” by Mary and Tom Poppendieck. The 7 principles are:1. Eliminate Waste. The three biggest sources of waste in software development are the addition of extra features, churn, and crossing organizational boundaries. Crossing organizational boundaries can increase costs by 25% or more because they create buffers which slow down response time and interfere with communication. It is critical that development teams are allowed to organize themselves, and run themselves, in a manner which reflects the work that they’re trying to accomplish. 2. Build Quality In. If you routinely find problems with your verification process then your process must be defective. When it comes to governance, if you regularly find that developers are doing things that you don’t want them to do or are not doing things that they should be then your approach to governance must be at fault. The strategy is not to make governance yet another set of activities that you layer on top of your software process but instead should embed into your process to make it as easy as possible for developers to do the right thing. 3. Create Knowledge. Planning is useful, but learning is essential. 4. Defer Commitment. You do not need to start software development by defining a complete specification, but instead work iteratively. You can support the business effectively through flexible architectures that are change tolerant and by scheduling irreversible decisions to the last possible moment. This also requires the ability to closely couple end-to-end business scenarios to capabilities developed in potentially several different applications by different projects. 5. Deliver Fast. It is possible to deliver high-quality systems fast and in a timely manner. By limiting the work of a team to their capacity, by not trying to force them to do more than they are capable but instead ask them to self-organize and thereby determine what they can accomplish, you can establish a reliable and repeatable flow of work. 6. Respect People. Sustainable advantage is gained from engaged, thinking people. The implication is that you need a human resources strategy which is specific to IT, that you need to focus on enabling teams not on controlling them. 7. Optimize the Whole. If you want to govern your development efforts effectively you must look at the bigger picture, not just individual project teams. You need to understand the high-level business process which the individual systems support, processes which often cross multiple systems. You need to manage programs of interrelated systems so that you can deliver a complete product to your stakeholders. Measurements should address how well you’re delivering business value, because that is the raison d’etre of your IT department.
Based on our experiences, and guided by the 7 principles, Per Kroll and I identified 18 practices of lean development governance. We've organized these practices into 6 categories:1. The Roles & Responsibilities category: - Promote Self-Organizing Teams. The best people for planning work are the ones who are going to do it. - Align Team Structure With Architecture. The organization of your project team should reflect the desired architectural structure of the system you are building to streamline the activities of the team.
2. The Organization category: - Align HR Policies With IT Values. Hiring, retaining, and promoting technical staff requires different strategies compared to non-technical staff. - Align Stakeholder Policies With IT Values. Your stakeholders may not understand the implications of the decisions that they make, for example that requiring an “accurate” estimate at the beginning of a project can dramatically increase project risk instead of decrease it as intended.
3. The Processes category: - Adapt the Process. Because teams vary in size, distribution, purpose, criticality, need for oversight, and member skillset you must tailor the process to meet a team’s exact needs. - Continuous Improvement. You should strive to identify and act on lessons learned throughout the project, not just at the end. - Embedded Compliance. It is better to build compliance into your day-to-day process, instead of having a separate compliance process that often results in unnecessary overhead. - Iterative Development. An iterative approach to software delivery allows progressive development and disclosure of software components, with a reduction of overall failure risk, and provides an ability to make fine-grained adjustment and correction with minimal lost time for rework. - Risk-Based Milestones. You want to mitigate the risks of your project, in particular business and technical risks, early in the lifecycle. You do this by having throughout your project several milestones that teams work toward.
4. The Measures category: - Simple and Relevant Metrics. You should automate metrics collection as much as possible, minimize the number of metrics collected, and know why you’re collecting them. - Continuous Project Monitoring. Automated metrics gathering enables you to monitor projects and thereby identify potential issues so that you can collaborate closely with the project team to resolve problems early.
5. The Mission & Principles category: - Business-Driven Project Pipeline. You should invest in the projects that are well-aligned to the business direction, return definable value, and match well with the priorities of the enterprise. - Pragmatic Governance Body. Effective governance bodies focus on enabling development teams in a cost-effective and timely manner. They typically have a small core staff with a majority of members being representatives from the governed organizations. - Staged Program Delivery. Programs, which are collections of related projects, should be rolled out in increments over time. Instead of holding back a release to wait for a subproject, each individual subprojects must sign up to predetermined release date. If the subproject misses it skips to the next release, minimizing the impact to the customers of the program. - Scenario-Driven Development. By taking a scenario-driven approach, you can understand how people will actually use your system, thereby enabling you to build something that meets their actual needs. The whole cannot be defined without understanding the parts, and the parts cannot be defined in detail without understanding the whole.
6. The Polices & Standards category: - Valued Corporate Assets. Guidance, such as programming guidelines or database design conventions, and reusable assets such as frameworks and components, will be adopted if they are perceived to add value to developers. You want to make it as easy as possible for developers to comply to, and more importantly take advantage of, your corporate IT infrastructure. - Flexible Architectures. Architectures that are service-oriented, component-based, or object-oriented and implement common architectural and design patterns lend themselves to greater levels of consistency, reuse, enhanceability, and adaptability. - Integrated Lifecycle Environment. Automate as much of the “drudge work”, such as metrics gathering and system build, as possible. Your tools and processes should fit together effectively throughout the lifecycle.
The URLs for the 3 articles:Principles and Organizations: http://www.ibm.com/developerworks/rational/library/jun07/kroll/Processes and Measures: http://www.ibm.com/developerworks/rational/library/jul07/kroll_ambler/Roles and Policies: http://www.ibm.com/developerworks/rational/library/aug07/ambler_kroll/
The URL for the white paper:https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?lang=en_US&source=swg-ldg
The URL for the webcast:https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?lang=en_US&source=dw-c-wcsdpr&S_PKG=112907C[Read More
Test-driven development (TDD) is a common agile programming technique which has both specification and validation aspects. With TDD, you specify your software in detail on a just-in-time (JIT) basis via executable tests that are run in a regression manner to confirm that the system works to your current understanding of what your stakeholders require.
TDD is the combination of test-first development (TFD) and refactoring. With TFD, you write a single test (at either the requirements level with customer/acceptance tests or the design level with developer tests) and then you write just enough software to fulfill that test. Refactoring is a technique where you make a small change to your existing code to improve its design without changing its semantics.
TDD offers several benefits:1. It enables you to take small, safe steps during development, increasing programmer productivity.2. It increases quality. Agile developers are doing more testing, and doing it more often, than ever before. We're also fixing the problems that we find right on the spot.3. It helps to push validation activities early in the lifecycle, decreasing the average cost to fix defects (which rises exponentially the longer it takes you to detect them).4. Through single sourcing information, by treating tests as both specifications and as tests, we reduce the work required, increasing productivity.5. We leave behind valuable, up-to-date, detailed specifications for the people who come after us. Have you ever met a maintenance programmer who wouldn't want a full regression test suite for the code that they're working with?
But TDD isn't perfect. Although TDD is great at specifying code at a fine-grain level, tests simply don't scale to address higher level business process and architectural issues. Agile Model Driven Development (AMDD) enables you to scale TDD through initial envisioning of the requirements and architecture as well as just-in-time (JIT) modeling at the beginning and during construction iterations. To scale requirements-level TDD, you must recognize that customer tests are very good at specifying the details, but not so good at providing overall context. High-level business process models, conceptual domain models, and use cases are good at doing so, and these work products are often created as part of your initial requirements envisioning and iteration modeling activities. Similarly, to scale design-level TDD you must recognize that developer tests are very finely grained but once again do not provide overall context. High-level architecture sketches created during envisioning activities help set your initial technical direction. During each construction iteration, you'll do more detailed design modeling to think through critical issues before you implement them via TDD.
You also need to scale the validation aspects of TDD. TDD is in effect an approach to confirmatory testing where you validate the system to the level of your understanding of the requirements. The fundamental challenge with confirmatory testing, and hence TDD, is that it assumes that stakeholders actually know and can describe their requirements. Therefore you need to add investigative testing practices which explore issues that your stakeholders may not have thought of, such as usability issues, system integration issues, production performance issues, security issues, and a multitude of others.
For further reading, I suggest:1. My article "Introduction to TFD/TDD" at http://www.agiledata.org/essays/tdd.html which overviews TDD.2. My February 2008 column in Dr. Dobb's Journal entitled "Scaling TDD" at http://www.ddj.com/architect/205207998 which explores this issue in detail. 3. Andrew Glover's article "In pursuit of code quality: Adventures in behavior-driven development" at http://www.ibm.com/developerworks/java/library/j-cq09187/ which describes a new-and-improved take on TDD called BDD.[Read More
There seems to be a lot of misunderstanding around the topic of "Agile RUP", so I thought I'd put in my $0.02 on the subject. I suspect that some of the misunderstanding stems from lack of knowledge about RUP, either because the person either hasn't looked at RUP and is simply parroting the misinformation they heard from other people or because they've seen or been involved with questionable implmentations of RUP in the past. For the first problem I suggest reading "Agility and Discipline Made Easy" by Per Kroll and Bruce MacIsaac because it gives a pretty good overview of applying RUP in an Agile manner.
The second problem is a bit more challenging to overcome because of the very nature of RUP. RUP isn't a software process, it's actually a software process framework from which you instantiate software processes. Big difference. You select, and tailor where appropriate, the process elements appropriate to your situation. Unfortunately many organizations appear to have struggled with this concept. A common anti-pattern which are organizations that look at the RUP and say "there's a lot of really good stuff here" (absolutely true) and then conclude "we need to do it all" (yikes). That's the equivalent of going to a buffet and trying to eat all of the food in it, very clearly a bad dietary strategy. Just like you need to pick and choose only the food that you should eat, hopefully being mature enough to choose food that is good for you, you need to pick the appropriate process elements which are good for you. This requires significant experience and process knowledge to do effectively because software development is a complex endeavor and the best approach for one situation may be completely different for another situation. A second anti-pattern is when organizations assign their existing process engineers, who are often used to document-heavy serial processes, and ask them to tailor the RUP. It isn't surprising that they often produce a document-heavy and serial version of the RUP (at that point I would argue that it's no longer RUP).
The point is that these problems are self-inflicted, that these organizations could just as easily have chosen to instantiate the RUP in a light and effective manner, and better yet in a truly agile manner. In practice the RUP can be as agile as you want it to be, but you need to choose to work this way.
Some important observations:1. RUP socialized many of the concepts that Agile was built on. Although the concept of iterative development was around long before RUP, for many organizations RUP made the concept palatable through its mature approach (particularly when compared to some of the RAD/Spiral strategies at the time). In many organizations RUP also socialized testing throughout the entire lifecycle, delivery of working software each iteration, and collaborating closely with stakeholders throughout the project (to name but a few). These ideas seem straightfoward today, and they've been taken to even greater extremes in some cases , but back in the mid-90s this was pretty heady stuff for the vast majority of practitioners within the IT community. 2. RUP has adopted many of the "new" agile techniques. RUP is a process framework containing a wealth of IT practices, including both agile and traditional practices (and a lot in between). RUP continues to evolve, capturing industry best practices from many sources. So naturally RUP has adopted many agile concepts such as test-driven development (TDD), continuous integration, embracing change, and others. Check it out and see for yourself.3. RUP is as agile, or non-agile, as you want to make it. I've said it before and I'll say it again -- the RUP is a process framework from which you instantiate processes. You've got complete control over how agile the RUP is.4. RUP contains many of critical techniques for scaling agile. In a previous blog posting I overviewed the issues around scaling, not only is team size an issue but so is geographical distribution, regulatory compliance, application complexity, legacy systems/policies, governance, and organizational distribution.
I'm sure some people are reading this and thinking to themselves "of course this is what Scott is going to say, that's his job." Well, think what you want, but I was writing about how to take an agile approach several years before joining IBM. In fact, I believe that I'm the first to do so, writing about it in print in my Software Development column back in 2001 and more importantly in my book Agile Modeling: Effective Practices for XP and UP in 2002. And, if you go poking around the web a bit, you'll see a lot other have written about this too, including Craig Larman, Ivar Jacobson, Bob Martin, Gary Evans, Doug Rosenberg and many more.
I'd like to leave you with a sound bite: "The RUP done right is Agile. The RUP done wrong is just plain wrong."[Read More
It's customary to start a blog by describing the vision for it. Although this vision will undoubtedly evolve over time, it's always good to put a stake in the ground to get things started. Agile software development is clearly taking off and in my opinion is becoming the dominant development paradigm. Furthermore it appears that Agile approaches enjoy a higher success rate, providing better value for your IT investment, than do traditional approaches. Although organizations are succeeding at simpler projects with agile, many are struggling when applying Agile in more complex situations. They're finding that the "Agile rhetoric" doesn't always live up to its promises once you move into these complex situations. My goal with this blog is to share strategies for applying Agile techniques at scale.
When applying Agile strategies at scale you are likely to run into one or more of the following complexity factors:1. Geographical distribution. Is your team, including stakeholders, in different locations? Even being in different cubicles within the same building can erect barriers to communication, let alone being in different cities or even on different continents.2. Regulatory compliance. Regulations, including the Sarbanes-Oxley act, BASEL-II, and FDA statutes, to name a few, can increase the documentation and process burden on your projects. Complying to these regulations while still remaining as agile as possible can be a challenge.3. Entrenched policies, people, and processes. Most agile teams need to work within the scope of a larger organization, and that larger organization isn't always perfectly agile. Hopefully that will change in time, but we still need to get the job done right now. Your existing culture and organization can really hinder your ability to scale agile approaches, then a few "simple" changes can really help your efforts.4. Legacy systems. Although the politically correct term would be "proven assets" the reality is that it can be very difficult to leverage existing code and data sources due to quality problems. The code may not be well written, documented, or even have tests in place, yet that doesn't mean that your agile team should rewrite everything from scratch. Some legacy data sources are questionable at best, or the owners of those data sources difficult to work with, yet that doesn't given an agile team license to create yet another database.5. Organizational distribution. When your teams are made up of people working for different divisions, or if you have people from different companies (such as contractors, partners, or consultants), then your management complexity rises.6. Degree of governance. If you have one or more IT projects then you have an IT governance process in place. How formal it is, how explicit it is, and how effective it is will be up to you. IBM has been doing a lot of work in this topic over the past few years, and just recently Per Kroll and I have done some work around Lean Governance strategies. 7. Team size. Large teams will be organized differently than small teams, and they'll work differently too.8. System complexity. The more complex the system the greater the need for a viable architectural strategy. An interesting feature of the Rational Unified Process (RUP) is that it's Elaboration phase's primary goal is to prove the architecture via the creation of an end-to-end, working skeleton of the system. This risk-reduction technique is clearly a concept which Extreme Programming (XP) and Scrum teams can clearly benefit from.
It is definitely possible to scale Agile software development to meet the real-world complexities faced by modern organizations. Based on my experiences, I believe that over the next few years we'll discover that Agile scales better than traditional approaches. Many people have already discovered this, but as an industry I believe that there isn't yet sufficient evidence to state this as more than opinion. My goal with this blog is to provide advice for scaling Agile so as to increase your chances of success.
So, it looks like I have my work cut out for me. My strategy will be to address common questions which I get when working with customers and with internal IBM development teams. I have the privilege to work with a variety of software development teams worldwide, helping them to become more agile. They're all struggling with the same basic issues although don't recognize it because they're too focused on their own situation. So hopefully I'll be able to spread the word about what's actually working in practice.
I hope that you stay tuned.
- Scott[Read More