One of the scaling factors
called out in the Agile
Scaling Model (ASM)
is “geographic distribution". As with the other scaling factors the level of geographic distribution is a range, with co-located teams at one extreme and far-located at the other. When your team is co-located the developers and the primary stakeholders are all situated in the same work room. If you have some team members in cubicles or in separate offices then you're slightly distributed, if you're working on different floors in the same building you're a bit more distributed, if you're working in different buildings within the same geographic area (perhaps your team is spread across different office buildings in the same city or some people work from home some days) then your team is more distributed, if people are working in different cities in the same country you're more distributed, and finally if people are working in different cities around the globe you're even more distributed (I call this far located).
As your team becomes more distributed your project risk increases for several reasons:
- Communication challenges. The most effective means of communication between two people is face-to-face around a shared sketching space such as a whiteboard, and that requires you to be in the same room together. As you become more distributed you begin to rely on less effective communication strategies.
- Temporal challenges. When people are in different time zones it becomes harder to find common working times, increasing the communication challenges. One potential benefit, however, is the opportunity to do "follow-the-sun" development where a team does some work during their workday, hands off the work to another team in a significantly different time zone, who picks up the work and continues with it. This strategy of course requires a high degree of sophistication and discipline on the part of everyone involved, but offers the potential to reduce overall calendar time.
- Cultural challenges. As the team becomes more distributed the cultural challenges between sites typically increases. Different cultures have different work ethics, treat intellectual property differently, have different ideas about commitment, have different holidays, different approaches to things, and so on.
As you would imagine, because the project risk increases the more distributed your team is, the lower the average success rates of agile projects decrease as they become more distributed. The 2008 IT Project Success Survey
found that co-located agile teams has an average success rate of 79%, that near located teams (members were in same geographic area) had a success rate of 73%, and that far-located agile teams had a success rate of 55%. The success rate decreases similarly for project teams following other paradigms.
The practices that you adopt, and the way that you tailor the agile practices which you follow, will vary based on the level of geographic distribution of your team. For example, a co-located team will likely do initial architecture envisioning
on a whiteboard and keep it at a fairly high-level. A far-located team will hopefully choose to fly in key team members at the beginning of the project, at least the architecture owners
on the various sub-teams, to do the architecture envisioning together. They will likely go into greater detail because they will want to identify, to the best of their ability, the interfaces of the various subsystems or components which they'll be building. They'll also likely use a modeling tool such as Rational Rhapsody
to capture this information (note that it is very possible to use CASE tools in an agile manner, and that some agile teams in fact do so as the 2008 Modeling and Documentation Survey
Geographically distributed agile teams, at least the effective ones, will also use tools which reflect the realities of agile geographically distributed development (GDD). The Jazz platform
tools, particularly Rational Team Concert (RTC)
, are developed with agile GDD in mind. In fact, the RTC team itself is an agile GDD and they use RTC to develop RTC. Although index cards are a great way for co-located agile teams to capture high-level requirements such as user stories, you need an electronic strategy for a GDD team. RTC also supports communication between team members, distributed debugging, and many other features which distributed teams will find useful.
Interestingly, the Agility at Scale 2009 survey
found that it was quite common for agile teams to be geographically distributed in some manner:
- 45% of respondents indicated that some of their agile teams were co-located
- 60% of respondents indicated that some of their agile teams had team members spread out through the same building
- 30% of respondents indicated that some of their agile teams were working from home
- 21% of respondents indicated that some of their agile teams had people working in different offices in the same city
- 47% of respondents indicated that some of their agile teams had team members that were far located
The bottom line is that some organizations, including IBM, have been very successful applying agile techniques on geographically distributed teams. In fact, agile GDD is far more common than mainstream agile discussion seem to let on. Recommended Resources:
One of the scaling factors
called out in the Agile Scaling Model (ASM)
is “regulatory compliance”. This name is a bit of a misnomer because this scaling factor really addresses two issues: complying to regulations imposed upon you from external sources and choosing to adhere to internal regulations willingly adopted by your organization. It is relatively common for agile teams to find themselves in such situations. For example, in the 2009 Agile Practices Survey
one third of respondents said that they were applying agile on projects where one or more industry regulations applied.
First let’s consider external regulatory compliance. In these situations you may face the need to undergo an audit by an external regulatory body with consequences for non-compliance ranging from anywhere to a warning to a fine or even to legal action. Sometimes even a warning may be a grave thing. A few years ago I was working with a pharmaceutical company which had discovered that a warning from the FDA for non-compliance with their CFR 21 Part 11 regulation, when reported in major newspapers, resulted on average in a half-billion dollar loss to their market capitalization as the result of a dip in their stock price. There are financial regulations such as Sarbanes-Oxley and Basel II, informational regulations such as HIPAA which focuses on health information privacy, technical regulations such as ISO 27002 for security practices, and even life-critical regulations such as some of the FDA regulations.
External regulations are typically managed by a government organization or industry watchdog will range in complexity and can have a myriad of effects on project teams. For example, you may need to be able to prove that you had a documented process and that you followed it appropriately; you may need to produce extra artifacts, or more detailed artifacts, than you normally would; you may need to add extra features to your solution, such as tracking financial information, that you wouldn’t have normally implemented; you may need to produce specific reports to be submitted to the regulatory body; or you may even need to submit your team to audits, sometimes scheduled and sometimes not, to ensure regulatory compliance. Interestingly, even though many of those requirements go against the agile grain, the 2009 Agility at Scale Survey
found that organizations were successfully applying agile techniques while still conforming to external regulations. So yes, it is possible to scale your agile strategy to address regulatory compliance.
Second, let’s consider compliance to internally adopted, or sometimes even developed, “regulations” which you will be potentially evaluated/appraised against. Perfect examples of these are process improvement frameworks such as CMMI and ISO 900x. Similar to external regulations, the 2009 Agility at Scale Survey
found that some agile teams are succeeding in situations where they have chosen to adopt such frameworks. It’s important to note that frameworks such as CMMI aren’t primarily about ensuring the compliance of development teams to a standard process, regardless of what CMMI detractors may claim, but instead about process improvement. Process improvement at the IT department (or beyond) is an enterprise discipline issue from the point of view of ASM, implying that frameworks such as CMMI affect more than one scaling factor.
When you find yourself in a regulatory situation, whether those regulations are imposed or willingly adopted, the best advice that I can give is to read the regulations and develop a strategy to conform to them in the most agile manner possible. If you let bureaucrats interpret the regulations you’ll likely end up with a bureaucratic strategy, but if you instead choose to take a pragmatic approach you will very likely end up with a very practical strategy. Part of that strategy is to treat the regulatory representative(s) within your organization as important stakeholders whom you interact with regularly throughout the project.
People who are new to agile are often confused about how agile teams address architecture, but luckily we're seeing more discussion around agile architecture
now in the community so this problem is slowly being addressed from what I can tell. But, what I'm not seeing enough discussion about, at least not yet, is how is enterprise architecture addressed in the overall agile ecosystem. So I thought I'd share some thoughts on the subject, based on both my experiences over the years (see the recommended resources at the bottom of this posting) as well as on an enterprise architecture survey
which I ran in January/February 2010.
My belief is that effective enterprise architecture, particularly in an agile environment, is:
- Business driven. Minimally your EA effort should be driven by your business, not by your IT department. Better yet it should be business owned, although this can be a challenge in many organizations because business executives usually aren't well versed in EA and view it as an IT function. Yes, IT is clearly an important part of EA but it's not the entirety of EA nor is it the most critical part. In many organizations the IT department initiates EA programs, typically because the business doesn't know to do so, but they should quickly find a way to educate the business in the need to own your organization's EA efforts.
- Evolutionary. Your enterprise architecture should evolve over time, being developed iteratively and introduced incrementally over time. An evolutionary approach enables you act on the concrete feedback that you receive when you try to actually implement it, thereby enabling you to steer its development successfully.
- Collaborative. The EA survey clearly pointed to "people issues" being critical determinants of success, and of failure, of EA programs. My experience is that the best enterprise architects, just like the best application architects, work closely with the intended audience of their work, both on the business side of things as well as on the IT side. They will "roll up their sleeves" and become active members of development teams, often in the role of Architecture Owner on agile teams or Architect on more traditional teams. Their mission is to ensure that the development teams that they work with leverage the EA, to mentor developers in architecture skills, and to identify what works and what doesn't in practice so that they can evolve the EA accordingly. Enterprise architects, architects in general, who don't participate actively on development teams (holding architecture reviews isn't active participation) run the risk of being thought of as "ivory tower" and thus easy to ignore.
- Focused on producing valuable artifacts. The most valuable artifacts are useful to the intended audience, are light weight, and ideally are executable. Many EA programs run aground when the enterprise architects focus on artifacts that they've always wanted but that development teams really aren't very excited about -- yes, it might be interesting to have a comprehensive comparison of cloud technologies versus mainframe technologies, but a collection of reusable services would be fare more interesting to them. A detailed enterprise data model indicating suggested data attributes would be intellectually interesting to develop, but a list of legacy data sources with a high-level description of their contents would be immediately valuable to many development teams. A detailed model depicting desired web services would be useful, but an actual collection of working services that I can reuse now would be even better.
- An explicit part of development. In Disciplined Agile Delivery (DAD) architectural activities are an explicit part of the overall delivery process. Part of the architectural advice is that delivery teams should work closely with their organization's enterprise architects so that they can leverage the common infrastructure, and sometimes to help build it out, effectively. Disciplined agile teams realize that they can benefit greatly by doing so.
The Agile Scaling Model (ASM)
calls out addressing enterprise disciplines, such as enterprise architecture, as one of eight scaling factors which may apply to a given project. The interesting thing about this scaling factor is that it's the only one where things get potentially easier for development teams when we move from the simple approach, having a project focus, to the more complex approach, where we have an enterprise focus. By having a common infrastructure to build to, common guidelines to follow, and valuable artifacts to reuse project teams can benefit greatly. So, I guess my advice is to seriously consider adding enterprise disciplines to your agile strategy.Recommended Resources:
My new paper Scaling Agile: An Executive Guide
is now available. As the title suggests the paper overviews how to scale agile strategies to meet your organization's unique needs. The executive summary:
Agile software development is a highly collaborative, quality-focused approach to software and systems delivery, which emphasizes potentially shippable working solutions produced at regular intervals for review and course correction. Built upon the shoulders of iterative development techniques, and standing in stark contrast to traditional serial or sequential software engineering methods, agile software delivery techniques hold such promise that IBM has begun to adopt agile processes throughout its Software Group, an organization with over 25,000 developers. But how can practices originally designed for small teams (10-12) be “scaled up” for significantly larger operations? The answer is what IBM calls “agility@scale.”
There are two primary aspects of scaling agile techniques that you need to consider. First is scaling agile techniques at the project level to address the unique challenges individual project teams face. This is the focus of the Agile Scaling Model (ASM).
Second is scaling your agile strategy across your entire IT department, as appropriate. It is fairly straightforward to apply agile on a handful of projects, but it can be very difficult to evolve your organizational culture and structure to fully adopt the agile way of working.
The Agile Scaling Model (ASM) defines a roadmap for effective adoption and tailoring of agile strategies to meet the unique challenges faced by a software and systems delivery team. Teams must first adopt a disciplined delivery lifecycle
that scales mainstream agile construction techniques to address the full delivery process, from project initiation to deployment into production. Then teams must determine which agile scaling factors
– team size, geographical distribution, regulatory compliance, domain complexity, organizational distribution, technical complexity, organizational complexity, or enterprise discipline, if any — are applicable to a project team and then tailor their adopted strategies accordingly to address their specific range of complexities.
When scaling agile strategies across your entire IT organization you must effectively address five strategic categories — the Five Ps of IT
: People, principles, practices, process, and products (i.e., technology and tooling). Depending on your organizational environment the level of focus on each area will vary. What we are finding within many organizations, including IBM, is that the primary gating factor for scaling agile across your entire organization is your organization’s ability to absorb change.
When you are first adopting agile techniques in your organization a common strategy is to run one or more pilot projects. When organizing these projects you typically do as much as you can to make them successful, such as finding:
- Projects where the stakeholders are willing to actively work with you.
- IT people who are flexible, willing to try new things, and willing to collaborate with one another.
- IT people who are generalizing specialists, or at least willing to become so.
- Finding a project which is of medium complexity (therefore it's "real" in the sense that it's significant to your organization) but not one where it can make or break your organization (therefore it's safe to experiment with).
In North America we refer to this as "cherry picking" because you're picking the cherry/best situation that you can find.
- Being agile may not have been the primary determinant of success. You set up an environment where you have a good relationship with your stakeholders, where you have good people who want to work together, and the project is challenging but not impossible. Oh, and by the way you adopted a few agile techniques as well. Sounds to me that situation you could have adopted a few not-so-agile techniques instead and still succeed. Although my various project success surveys, see my IT surveys page for details, have shown time and again that agile project teams are more successful than traditional project teams I haven't been able to tease out (yet) whether this success is attributable to agile or just attributable to improved project initiation efforts.
- When adopting agile/lean widely across your organization, you can't cherry pick any more. For the past few years I've been working with IT organizations that are in the process of adopting agile/lean strategies across their entire organization, not just across a few pilot projects. What these organizations are finding is that they need to find ways to adopt agile where the business isn't as willing to work with IT, where some of the people aren't so flexible or collaborative, where some of the people are narrowly specialized and not as willing to expand their skills, or where the project exhibits scaling factors which motivates you to tailor your agile approach. It's harder to succeed with agile in these situations because they're not as "cherry" as what you've experienced previously. Luckily, if you've been successful previously then you now have some agile experienced people, you have successes to reference, and you've likely overcome some problems even in the cherry situations which you have learned from. So, your cherry successes will hopefully improve your ability to succeed even in "non cherry" situations.
- You need to work smarter, not harder. If the source of your success was actually from improved project initiation practices and not from agile, then recognize that and act accordingly. Realistically part of your success was from that and part was from agile, and the organizations that adopt a measured improvement approach potentially have the data to determine which practices lead to success and which didn't. Without the metrics you're effectively flying blind when it comes to deciding how to improve. There is clearly a mandate for smarter work practices within IT, within your organization as a whole for that matter.
If you want to gain more insight into some of the issues that you'll face when adopting agile across your organization, I suspect that you'll find my recent paper Scaling Agile: An Executive Guide
to be interesting. I've got a more detailed paper in the works, so stay tuned to this blog.
Modificado por ScottAmbler
An imporant step in scaling your agile strategy is to adopt a Disciplined Agile Delivery (DAD)
approach instead of one which is just focused on agile construction. One aspect of adopting a DAD approach it to mature your focus from just producing software to instead providing a solution which meets the needs of its stakeholders within the appropriate economic, cultural, and technical constraints. The fundamental observation is that as IT professionals we do far more than just develop software. Yes, this is clearly important, but in addressing the needs of our stakeholders we will often:
Provide new or upgraded hardware
Change the business/operational processes which stakeholders follow
Change the organizational structure in which our stakeholders work
Update supporting documentation
And yes, develop high-quality software
Although delivery of high-quality, working software is important it is even more important that we deliver high-quality working solutions to our stakeholders. Minimally IT professionals should have the skills and desire to produce good software, but what they really need are the skills and desire to provide good solutions. We need strong technical skills, but we also need strong "soft skills" such as user interface design and process design to name just two.
The shift to a solution-oriented focus from a software-oriented focus requires your agile teams to address some of the software-oriented prejudices which crept into the Agile Manifesto
. The people who wrote the manifesto (which I fully endorse) were for the most part software developers, consultants, and in many cases both. It is little wonder that this group would allow a bias towards software development creep into the language of their manifesto.
activities are evolutionary (iterative and incremental) and highly collaborative in nature. Initially requirements are explored at a high level via requirements envisioning
at the beginning of the project and the details are explored on a just-in-time (JIT) basis via iteration modeling
and model storming
activities. The way that you perform these agile practices, and the extent to which you do so, depends on the situation in which a project team finds itself. The Agile Scaling Model (ASM)
is a contextual framework for effective adoption and tailoring of agile practices to meet the unique challenges faced by a system delivery team of any size. To see how this works, let's apply the concepts of the ASM to see how we would scale our agile approach to requirements.
First, let's consider how a small, co-located team would work. The first two categories of the ASM are core agile development and disciplined agile delivery
, the focus of both are small co-located teams in a fairly straightforward situation. In these situations simple techniques such as user stories
written on index cards and sketches on whiteboards
work very well, so the best advice that I can give is to stick with them. Some teams will take a test-driven development
(TDD) approach where they capture their requirements and design in the form of executable specifications
, although this sort of strategy isn't as common as it should be (yet!), likely because of the greater skill and discipline that it requires. Traditionalists often balk at this approach, believing that they need to document the requirements in some manner. But, for a small co-located team working in a collaborative manner, requirements documentation proves to be little more than busy work, often doing nothing more than justifying the existence of a business analyst who hasn't made the jump to agile yet. Don't get me wrong, there are good reasons to write some requirements documentation, and we'll see this in a minute, but you should always question any request for written specifications and try to find more effective ways to address the actual goal(s) motivating the request. Never forget that written documentation
is the least effective communication
option available to you.
Although inclusive tools
such as whiteboards and paper work well for requirements, for development activities you will need electronic tools. You will either put together an environment from point-specific tools or adopt something more sophisticated such as IBM Rational Team Concert (RTC)
which is already fully integrated and instrumented. RTC is a commercial tool, but luckily you can download a 10-license environment free of charge, which is just perfect for a small team. Larger teams, of course, will need to purchase licenses. One of the things that a disciplined agile delivery approach adds to core agile development is it addresses the full delivery life cycle, which is important because it explicitly includes pre-construction activities such as requirements envisioning. The first step in scaling agile techniques is to adopt a full delivery life cycle which covers the full range of activities required to initiate a project, produce the solution, and then release to solution to your end users.
More interesting is the third category of the ASM, Agility@Scale, and how its eight agile scaling factors
affect the way that you tailor your process and tooling strategy. Let's explore how each one could potentially affect your agile requirements strategy:
- Geographical distribution. The majority of agile teams are distributed in some manner -- some people are working in cubicles or private offices, on different floors, in different buildings, or even in different countries -- and when this happens your communication and coordination risks goes up. To counter this risk you will need to perform a bit more requirements envisioning up front to help ensure that everyone is working to the same vision, although this doesn't imply that you need to write detailed requirements speculations which would dramatically increase the risk to your project. Remember, agilists do just barely enough modeling and are prepared to iteratively elicit the details when they need to do so. The more distributed the team is the more likely they will need to adopt software-based requirements modeling tools such as IBM Rational Requirements Composer (RRC) which supports streamlined, agile requirements elicitation throughout the delivery life cycle. Index cards and whiteboards are great, but they're difficult to see if you're outside the room where they're posted. I've written a fair bit about distributed agile development in this blog.
- Team size. Some organizations, including IBM, are successfully applying agile techniques with teams of hundreds of people. A team of one hundred people will naturally work much differently than a team of ten people, or of one thousand people. Large teams are organized into collections of smaller teams, and the requirements for the overall project must be divvied up somehow between those teams. The implications are that as the team size grows you will need to invest a bit more time in initial requirements envisioning, and in initial architecture envisioning for that matter; you will need to use more sophisticated tools; and may need to use more sophisticated modeling techniques such as use cases and functional user interface prototypes. See large agile teams for more advice.
- Compliance requirement. When regulatory issues – such as Sarbanes Oxley, ISO 9000, or FDA CFR 21 – are applicable you are likely going to be required to capture requirements specifications in some manner and to enact traceability between those requirements. However, I highly recommend that you read the actual regulations yourself and don't let bureaucrats interpret them for you (doesn't it always seem that their interpretation always results in an onerous, documentation heavy solution?) because I have yet to run into a regulation which required you to work in an ineffective manner. Managing your requirements as work items in RTC can often more than meet your regulatory requirements for documentation and traceability, although you may want to consider a tool such as IBM Rational RequisitePro for complex regulatory situations.
- Domain complexity. The manner in which you elicit requirements for a data entry application or an informational web site will likely be much simpler than for a bio-chemical process monitoring or air traffic control system. More complex domains will require greater emphasis on exploration and experimentation, including but not limited to prototyping, modeling, and simulation. Although user stories may be effective as a primary requirements artifact in simple domains, in more complex domains you are likely to find that you need to drive your requirements effort with more sophisticated modeling techniques.
- Organization distribution. Sometimes a project team includes members from different divisions, different partner companies, or from external services firms. In these cases, particularly where the work is strictly organized between the various organizations (perhaps for security concerns), you may need a more sophisticated approach to managing the requirements. RTC enables you to organize the requirements between teams, and then to automatically track progress in real time via the RTC project dashboard.
- Technical complexity. The technical complexity of a solution can vary widely, from a single platform silo application to a multi-platform application working with legacy systems and data to a full-blown systems engineering effort. Complex technical domains, just like complex business domains, require more complex strategies for requirements elicitation and management. The requirements for your legacy systems are likely to have been captured using tools and techniques appropriate for that platform, for example the requirements for your COBOL application may have been captured using data flow diagrams and data models, whereas the requirements for your Java legacy application where captured using UML diagrams. The subteam working on the COBOL system might be using IBM Rational Application Developer (RAD) and RTC for Z whereas the Java subteam may use Eclipse with RTC. Because systems engineering projects can stretch on for years, particularly when the hardware is being developed in parallel to the software, sophisticated tooling such as IBM Rational DOORS is often used in these situations. For more information about systems engineering, see the IBM Rational Harmony process.
- Organizational complexity. Your approach to requirements elicitation and management will be affected by a host of organizational complexities, including your corporate culture. When the culture is flexible and collaborative you can be very agile in your approach to requirements, but as it becomes more rigid you become more constrained in what is considered acceptable and thus take on greater project risk. For example, many organizations still struggle with their approach to funding projects, often demanding that the project team provides an "accurate" estimate up front to which they will be held to. This in turn motivates risky behavior on the part of the development, including a "big requirements up front (BRUF)" approach where a detailed requirements speculation is developed early in the project. This is just one example of how questionable corporate culture can impact the way in which an agile team works.
- Enterprise discipline. Some organizations have enterprise-level disciplines, such as enterprise architecture, enterprise business modeling, strategic reuse, and portfolio management in place. These disciplines can easily be agile and from what I can tell the more successful efforts appear to lean more towards the agile end of the spectrum rather than the traditional end. Having an enterprise business modeling effort underway will affect your project-level requirements strategy -- you'll be able to leverage existing models, have access to people who understand the domain at an enterprise level, and will likely need to map your project efforts back to your enterprise models. The enterprise modelers will likely be using tools such as IBM Rational System Architect or IBM Websphere Business Modeler.
It is important to note that the way that you tailor the agile practices that you follow, and the tools that you use, will reflect the situation that you find yourself in. In other words, you need to right size your process and the Agile Scaling Model (ASM) provides the context to help you do so. As you saw above, in simpler situations you will use the simpler tools and techniques which are commonly promoted within the core agile development community. But, when things become a bit more complex and one or more of the scaling factors applies you need to modify your approach -- just don't forget that you should strive to be as agile as you can be given the situation that you find yourself in.
Yesterday I was involved with a workshop around agile development at scale. At one point in the conversation we started talking about the relationship between cost and quality. Some of the people in the workshop were relatively new to agile and still believed the traditional theory that to build in high quality it costs more, sometimes substantially more. This does appear to be true on traditional waterfall projects, but some people were making the mistake that this was an "natural law of IT" which also must apply to agile project teams. I naturally jumped on that idea and described how agile developers have found that writing high quality code leads to lower development costs and shorter time to value, in direct contradiction to traditional theory. A few people struggled with the idea for a bit, and one was pretty adamant that in some cases the need for very high quality does in fact lead to greater cost and time. He talked about his experiences on large-scale Rational Unified Process(RUP)
projects and in particular how some URPS (usability, reliability, performance, and supportability) requirements can increase your cost. At this point Per Kroll, co-author of Agility and Discipline Made Easy: Practices from OpenUP and RUP
, jumped into the conversation and pointed out although higher quality does lead to lower cost in most cases, using Toyota's lean approach to manufacturing as an example, that the agile community didn't completely have the relationship between quality and cost completely correct. My spidey sense told me that a learning opportunity was coming my way.
Per and I had an offline discussion about this to explore what he'd been observing in practice. In most situation it appears to be the case that higher quality does in fact lead to lower costs and shorter time for delivery, something that Per and I had observed numerous times. This happens because high quality code is much easier to understand and evolve than low quality code -- the agile community has found that it is very inexpensive to write high quality code by following practices such as continuous integration
, developer regression testing [or better yet test-driven development(TDD)
], static code analysis
, following common development conventions, and agile modeling strategies
. When you "bake in" quality from the start through applying these techniques, instead of apply traditional techniques such as reviews
and end-of-lifecycle testing (which is still valid for agile projects, but should not be your primary approach to testing) which have long feedback cycles
and therefore prove costly in practice. But, as we've learned time and again, when you find yourself in more complex situations of Agility@Scale sometimes the mainstream agile strategies fall down. For example, in situations where the regulatory compliance scaling factor is applicable, particularly regulations around protecting human life (i.e. the FDA's CFR 21 Part 11), you find that some of the URPS requirements require a greater investment in quality which can increase overall development cost and time. This is particularly true when you need to start meeting 4-nines requirements (i.e. the system needs to be available 99.99% of the time) let alone 5-nines requirements or more. The cost of thorough testing and inspection can rise substantially in these sorts of situations.
In conclusion, it does seem to be true in the majority of situations, which is what the level 1 rhetoric focuses on, that higher quality leads to lower development costs. But at scale this doesn't always seem to hold true.
PS -- Sorry for the corny title, but a couple of days ago at the Rational Software Conference I had the pleasure of interviewing Jamie Hyneman and Adam Savage from the Discovery Channel's Mythbuster's show as part of the conference keynote. They're great guys, BTW, who have had a really positive impact on motivating children to be interested in science (apparently kids like to see stuff get blown up, go figure).[Read More
I recently wrote an "e-book" for Internet Evolution overviewing agile software development at scale. The goal of the Agility at Scale: Become as Agile as You Can Be
ebook is to get people thinking outside of the box a bit when it comes to agile development strategies and see that they really are ready for primetime.
Recently I visited a customer who had adopted Scrum. They were a few sprints, what Scrum calls iterations, into the project and were running into some difficulties. Although I was primarily brought in to educate senior management on disciplined agile software development, I was also asked to sit in on the team’s daily stand-up meeting so that I could hopefully provide some suggestions as to how to address the problems they were running into.
Their work area was fairly typical. They had some whiteboards which they were using for project planning and tracking, with sticky notes to indicate what work had been taken on by each team member. The current status of the task (not yet started, in progress, and completed) was indicated by putting each sticky note in a corresponding column for the status and corresponding column for the team member. This allowed everyone on the team to easily share their status and to see the status of everyone else. On the sides were sketches of the architecture as well as some business oriented models. In addition to Scrum the team had adopted several practices from Agile Modeling, in this case they had done some initial requirements envisioning
and architecture envisioning
, as well as practices from Extreme Programming (XP) for construction. In short, they had followed a fairly common strategy of combining practices from various agile methods.
This would have worked perfectly fine if they had tailored the practices to reflect the situation that they were in, but instead they adopted them "straight out of the book". First, the team was distributed, with most of the team in the location that I was visiting but some people located in two other distant cities. Therein was the source of most of their problems. The people at the other two locations weren’t getting much value out of the daily stand-up meetings, even though they would dial in, because they couldn’t see the project status information. Although people at this location were trying their best to represent these distant people in the daily stand-ups it wasn’t working well – their status information wasn’t being kept up to date and for some people it was a bit of mystery as to what they were actually working on at all.
This team also had 30 people in it, which isn’t a big deal although it can stretch the limits of the simple modeling and planning tools (in this case paper and whiteboards) that they were using. Because the team was larger they were investing a fair bit of time creating burn down charts at both the iteration/sprint and project levels. One of the unfortunate implications of using manual tools for project management is that any associated metric/status reporting in turn becomes manual as well. Considering how the agile community is so concerned with working efficiently, I find it comical that we have a tendency to overlook our own potentially unnecessary bureaucracy such as this.
The problem was that the team was applying strategies, in this case using sticky notes and whiteboards to capture the detailed iteration plan, applying similar strategies to capture key models, and were verbally relaying of status information between sub-teams. There are perfectly fine strategies for smaller co-located teams, but not so good for large or distributed teams. The solution was to recognize that they were in an Agility@Scale situation and needed to tailor their approach to reflect this fact. In this case they needed to forgo some of the manual tools and instead use electronic tooling such as Rational Team Concert (RTC) to share information across disparate locations, in particular the work assignment and corresponding status information. RTC also creates common agile reports such as burn-down charts based on the activities of the developers, providing accurate (nearly) real-time information while removing the burden of status reporting. The RTC project dashboard does more than just this, to see an actual example of one visit www.jazz.net
to see the dashboard for the RTC development team itself. You can also see their actual work item list too, a more advanced version of Scrum’s product and sprint backlogs.[Read More
In the early days of agile, the applications where agile development was applied were smaller in scope and relatively straightforward. Today, the picture has changed significantly and organizations want to apply agile development to a broader set of projects. Agile hence needs to adapt to deal with the many business, organization, and technical complexities today’s software development organizations are facing. This is what Agility@Scale is all about – explicitly addressing the complexities which disciplined agile delivery teams face in the real world.These agile scaling factors which we've found to be important are:
- Team size. Mainstream agile processes work very well for smaller teams of ten to fifteen people, but what if the team is much larger? What if it’s fifty people? One hundred people? One thousand people? Paper-based, face-to-face strategies start to fall apart as the team size grows.
- Geographical distribution. What happens when the team is distributed, perhaps on floors within the same building, different locations within the same city, or even in different countries? Suddenly effective collaboration becomes more challenging and disconnects are more likely to occur.
- Compliance requirement. What if regulatory issues – such as Sarbanes Oxley, ISO 9000, or FDA CFR 21 – are applicable? These issues bring requirements of their own that may be imposed from outside your organization in addition to the customer-driven product requirements.
- Enterprise discipline. Most organizations want to leverage common infrastructure platforms to lower cost, reduce time to market, and to improve consistency. To accomplish this they need effective enterprise architecture, enterprise business modeling, strategic reuse, and portfolio management disciplines. These disciplines must work in concert with, and better yet enhance, your disciplined agile delivery processes.
- Organizational complexity. Your existing organization structure and culture may reflect traditional values, increasing the complexity of adopting and scaling agile strategies within your organization. To make matters worse different subgroups within your organization may have different visions as to how they should work. Individually the strategies can be quite effective, but as a whole they simply don’t work together effectively.
- Organization distribution. Sometimes a project team includes members from different divisions, different partner companies, or from external services firms. This lack of organizational cohesion can greatly increase the risk to your project.
- Technical complexity. Some applications are more complex than others. It’s fairly straightforward to achieve high-levels of quality if you’re building a new system from scratch, but not so easy if you’re working with existing legacy systems and legacy data sources which are less than perfect. It’s straightforward to build a system using a single platform, not so easy if you’re building a system running on several platforms or built using several disparate technologies. Sometimes the nature of the problem that your team is trying to address is very complex in its own right.
Each factor has a range of complexities, and each team will have a different combination and therefore will need a process, team structure, and tooling environment tailored to meet their unique situation. Further reading:
Contrary to popular belief, agile development teams do in fact model and yes, they even do some up front requirements and architecture modeling. Two of the best practices of Agile Modeling are Requirements Envisioning
and Architecture Envisioning
where you spend a bit of time at the beginning of the project doing enough initial modeling to get you going in the right direction. The strategy is to take advantage of modeling, which is to communicate and think things through without taking on the risks associated with detailed specifications written early in the lifecycle
. In this blog posting I will focus on requirements envisioning, in a future posting I'll cover architecture envisioning.
The goal of initial requirements envisioning is to identify the scope of your effort. You need to do just enough modeling early in the project to come to stakeholder concurrence and answer questions such as what you're going to build, roughly how long it's going to take (give a range), and roughly how much it's likely to cost (once again, give a range). If you can get the right people together in the room, which can sometimes be a logistics challenge but not one that you couldn't choose to overcome, there are very few systems (I suspect less than 5%) that you couldn't initially scope out in a few days or a week. I also suspect that most of the remaining systems could be scoped out with less than 2 weeks of modeling, and if not then I'd take that as an indication that you're taking on too large of a project. I'm not saying that you'll be able to create big detailed specifications during this period, and quite frankly given the problems associated with "Big Requirements Up Front (BRUF)
" you really don't want to, but I am saying that you could gain a pretty good understanding of what you need to do. The details, which you'll eventually need, can be elicited throughout the lifecycle when you actually need the information. A common saying in the agile community is that requirements analysis is so important for us that we do it every single day, not just during an initial phase. I'll discuss just in time (JIT) approaches to requirements modeling in a future posting.
To envision the requirements for a business application, you might want to consider creating the following models:
- High-level use cases (or user stories). The most detail that I would capture right now would be point form notes for some of the more complex use cases, but the majority just might have a name. The details are best captured on a just-in-time (JIT) basis during construction.
- User interface flow diagram. This provides an overview of screens and reports and how they're inter-related. You just need the major screens and reports for now.
- User interface sketches. You'll likely want to sketch out a few of the critical screens and reports to give your stakeholders a good gut feeling that you understand what they need. Sketches, not detailed screen specifications, are what's needed at this point in time.
- Domain model. A high-level domain model, perhaps using UML or a data modeling notation, which shows major business entities and the relationships between them, can also be incredibly valuable. Listing responsibilities, both data attributes and behaviors, can be left until later iterations.
- Process diagrams. A high-level process diagram, plus a few diagrams overviewing some of the critical processes, are likely needed to understand the business flow.
- Use-case diagram. Instead of a high-level process diagram you might want to do a high-level use case diagram instead. This is a matter of preference, I likely wouldn't do both.
- Glossary definitions. You might want to start identify key business terms now, although I wouldn't put much effort into settling on exact definitions. I've seen too many teams run aground on "analysis paralysis" because they try to define exact terminology before moving forward. Don't fall into this trap.
For small teams simple tools such as whiteboards and paper are usually sufficient for requirements envisioning. But what happens at scale? What if you're working on a large agile team, say of 50 people, 200 people (IBM has delivered software into the marketplace with agile teams of this size), or even 500 people (IBM currently has teams of this size applying agile techniques)? What if your team is distributed? Even if you have people working on different floors of the same building, let alone working from home or working in different cities or countries, then you're distributed (see my postings about distributed agile development
). Suddenly whiteboards and paper-based tools (index cards, sticky notes, ...) aren't sufficient. You're still likely to use these sorts of tools in modeling sessions with stakeholders, but because of one or more scaling factors you need to capture your requirements models electronically.
In January Theresa Kratschmer and I gave a webcast entitled Agile Requirements: Collaborative, Contextual, and Correct
which overviewed agile approaches to requirements elicitation and management, including requirements envisioning. We also showed how Rational Requirements Composer (RRC)
can be used to electronically capture critical requirements information, enabling you to address the needs of large and/or distributed agile teams, while still remaining lightweight and flexible. I suspect that you'll find the webcast to be very illuminating and RRC something that you want to take a look at (the link leads to a trial version). Of course RRC can be used in other situations as well, but that's not what I'm focused on right now.
Teams which find themselves in regulatory environments will likely need to do more than just use RRC, as might very large teams. Regulatory compliance often requires more complex requirements documentation, which in turn requires more sophisticated tools such as DOORS or Requisite Pro, and I would consider using those tools in the types of situations that warrant it. One of the things that people often struggle to understand about agile approaches is that you need to tailor your strategy to reflect the situation at handle. One process size does not fit all, so you will end up using different tools and creating different artifacts to different extents in different situations. Repeatable results, not repeatable processes
, is the rule of the day. Further reading:
A common misunderstanding about agile software development approaches are that they're only applicable to small, co-located teams. Yes, it's much easier to be successful with small teams, and with co-located teams, and as a result agilists being smart people prefer to work this way. After all, why take on extra risk when you don't need to do so? But, sometimes reality gets in the way and you find yourself in a situation where you need a large team, or you need to distribute your team (see previous blog postings for strategies for distributed agile development), and you would still like to be as agile as possible. The good news is that it's still possible to be agile with a large team, although you will need to go beyond some of the popular "agile in the small" strategies to succeed.
Here are some disciplined agile strategies to succeed at large-team agile:
- Question the need for a large team. Many times an organization will believe that they need a large team because their process is overly complex, because they're still organized for waterfall development, or simply because that's what they're used to. I've seen teams of 80 people doing the work of 20 as the result of over-specialization of job roles and all the bureaucracy required to organize and validate their work.
- Do some initial envisioning. In order to succeed the team must work together towards the same goals. This is true for small teams but doubly true for larger ones -- without a common vision chaos will quickly ensue. You must gain this common vision on two fronts: you need a common business vision and a common technical vision. To gain the common business vision you must do some initial, high-level requirements envisioning and to gain the common technical vision some common architecture envisioning. This isn't to say that you need to take on the risk of detailed, up-front specifications but you must at least have a high-level understanding of the scope and technical solution in order to move forward effectively. So, expect to spend the first few weeks of your project doing this initial modeling.
- Divide and conquer. You never have a team of 200 people, instead you have a collection of subteams that add up to 200 people. This is called having a team of teams.
- Align team structure with architecture. The most effective way to organize the subteams is to have each one implement one or more components, and thereby to build your overall system as a "system of systems". This reduces the coordination required because the majority of the communication will be within the subteams themselves. You'll still need to coordinate the subteams, that will never go away, but you can reduce the overhead (and the risk) by being smart about the way that you organize the people. A common mistake is to organize around job function (e.g. having architects in Toronto, developers in Raleigh, testers in Bangalore, and so on). This increases communication overhead and risk because these people need to work together closely to get something built.
- Project management coordination. Each subteam will have a team lead/coach, and these people will need to coordinate their work. There is often an overall project manager who leads this group. To coordinate the work within their subteam the team lead/coach will often have a daily meeting, in the Scrum method this is called a scrum meeting, where people share their current status and identify any problems they may be running into. To scale this effectively the team lead/coach attends a daily team coordination meeting, a scrum of scrums, where the same sort of information is shared at the overall team level.
- Product owner coordination. Similarly, each subteam has a product ownder, also referred to as an "on-site customer", who is responsible for making decisions about the requirements and for providing information to the team in a timely manner. Sometimes a single product owner will work with several subteams. The product owners will get together at the beginning of the project to do some requirements envisioning to identify the initial scope and to start portioning the requirements between the subteams. Because the requirements between the subsystems are interrelated and should be reasonably consistent, the product owners will need to meet on a regular basis to share information, to negotiate priorities, and to resolve requirements-related disputes.
- Architecture coordination. Each subteam will have an architecture owner, often a senior technical person and sometimes also in the role of the team lead/coach. These architecture owners will get together at the beginning of the project to do some initial architecture envisioning, based on the requirements envisioning efforts of the product owners. They will identify the major subsystems, and their interfaces, enabling the effective organization of the team into smaller subteams corresponding to the architecture. They will also get together regularly to evolve the architecture and to resolve any major technical issues.
- System integration team. For complex systems, which is often what large teams work on, an effective system integration effort is critical to your success. Although this may be easy at first, as the overall system evolves the need for a subteam focused solely on this quickly becomes apparent. This not only supports the development efforts of the subteams, it also supports independent investigative testing.
- Independent testing team. An independent testing team is common on mid-to-large size agile projects to enhance the testing efforts of the development subteams. This testing team will work in parallel to the developers, they get a new build on a regular basis (minimally each iteration, although more often is desirable), which they test in more advanced ways than what is typical with Test-Driven Development (TDD). For example, they often validate non-functional, quality of service (QoS) type requirements as well as technical constraints, things that often aren't captured well via user stories. They'll also do investigative testing to try to break the system by using it in ways not thought of by the product owners.
- Some specialties reappear. On larger teams it can make sense to have some people be a bit more specialized than what we normally see on small agile teams. For example, it's common to see people in the role of agile DBA, tech writer, build master, or user experience (UE) professional. More complex systems often require people in these roles, although it still behooves these poeple to not be pure specialists but instead to be generalizing specialists with a wider range of skills. Also, recognize that the reintroduction of specialists can be a slippery slope back to the bureaucracy of traditional software development.
Modificado por ScottAmbler
I just wanted to round out my discussion about agile approaches to geographically distributed development (GDD) with a few important words of advice:1. Get some experience. Worry less about enterprise adoption and instead get started with a small project, or better yet a series of increasingly more complex projects. There will be learning experiences as you build a relationship with the offshore service provider. This advice is applicable whether you’re working with your own offshore division or with an independent service provider.2. Have a long-term staffing strategy. It may be great in the short term to have work done in a lower cost country, but how are you going to transfer the necessary skills to the maintenance and support team. Outsourcing that work is also an option, but it can be a risky one as you would need to build up expertise in “your” systems if you ever decide to insource that work again.3. Be concerned about intellectual property (IP). The rules are different around the world, and you may inadvertently be financing the creation of a new international competitor if you don’t have a clear division of ownership. And yes, this may mean that some components of your systems are still built internally by your own organization.4. Show off locally before you go global. GDD makes things harder to manage, so if you’re struggling to manage local teams you’re really going to struggle managing teams at a distance. Make sure you have local success first and are good at agile development in general. Furthermore, if your agile GDD projects run into trouble, don’t end your local agile adoption just because of difficulties with distributed projects.5. Let your offshore partners lead. The offshore partner likely has more experience than you at successful distributed development, and this is particularly true when you’re dealing with an established service provider.6. Do some reading. There’s a great IBM Redbook entitled “Global Development and Delivery in Practice: Experiences of the IBM Rational India Lab” which can be downloaded free of charge from http://www.redbooks.ibm.com/abstracts/sg247424.html7. Do some viewing. We recorded a Rational Chat a few months ago entitled "Being Agile in a Global Development Environment" which is posted at https://www14.software.ibm.com/webapp/iwm/web/reg/acceptSignup.do?lang=en_US&source=dw-c-wcsdpr&S_PKG=120607&S_TACT=105AGX23&S_CMP=TALKS&cp=UTF-8 . I also gave a keynote on Agile approaches to GDD at Software Development Practices 2007 held in Boston in the Autumn of 2007. The video can be downloaded free of charge from http://www.life20.net/video/scottambler.mov .[Read More
Modificado por ScottAmbler
In my previous blog posting, http://www.ibm.com/developerworks/blogs/page/ambler?entry=strategies_for_distributed_agile_teams , I overviewed several strategies for improving your effectiveness at geographically distributed development (GDD). Those strategies were fairly generic and directly applicable to both traditional and agile development teams. In this posting I focus on strategies which are more agile in nature, although they could also be applied to more traditional approaches as well. These strategies are:
1. Get the whole team together at the beginning of the project. Your goals are to build rapport amongst the team, to get to know the people that you’re working with to facilitate communication later on, and to better understand the situation on the ground. The implication is that you will need to fly some people around, increasing your initial expenses, an investment that many organizations balk at. The reality is that you will eventually end up paying for travel anyway, either because you actually flew people around or because your communication costs are higher throughout the project. In short, don’t be penny wise and pound foolish.
2. Organize your team around the system architecture. The most effective way to organize a distributed team is around the architecture of the system that you are building, not around the job functions of the people involved. In other words, if your team is in Toronto, Rome, and Bangalore then each subteam should be responsible for one or more subsystems. It would be a mistake to organize the teams around job function, for example to have the architects and analysts in Toronto, the developers in Rome, and the testers in Bangalore because this structure would require significantly more documentation and other forms of communication to coordinate the teams, increasing both cost and risk. As I mentioned in my previous blog posting you will need to invest in some initial architecture envisioning at the beginning of a project to identify the subsystems and their public interfaces, and that to do that you’ll also need to do some initial requirements envisioning to drive this architecture effort. I suggest that you take an Agile Model Driven Development (AMDD) approach to this to enable you to gain the value from modeling without the costs and risks associated with up-front comprehensive modeling and documentation that get many traditional project teams in trouble.
3. Have “daily stand-up meetings”. A common practice on co-located agile teams is to have daily stand-up meetings where people share the status of what they did yesterday, what they intend to do today, and whether they’re running into any problems. These short meetings enable team coordination. Distributed teams can do this as well, the people in a given geographical location can hold local stand-up meetings and then representatives from each location can hold a shared meeting to coordinate the subteams. Whereas local stand-up meetings are held first thing in the morning, distributed daily stand-up meetings may need to be held at unusual times so as to include people at distant locations.
4. Have Ambassadors. Ambassadors are people who travel between sites, often technically senior people or senior business experts, to share information between the subteams. Getting the team together at the beginning of the project sets the foundation for communication, but without continual investment in maintaining effective collaboration between teams you run the risk of your subteams deviating from the overall strategy. These are typically short engagements, a week or two in length, because of the pressures it puts on the people doing the actual traveling. The implication is that you’ll have several people flying between sites at any given time on a reasonable rotation schedule. Because you’ll have some people flying around, your local team rooms should accommodate visitors by having one or more desks available for them to use when they’re visiting.
5. Have Boundary Spanners. A boundary spanner is someone who is located on site who focuses on enabling communication between subteams as well as within their subteam. On large distributed teams you’ll find that you have three flavors of boundary spanners – team leaders who take on project management responsibilities on the subteam, product owners who are responsible for representing the business within the subteam, and architecture owners responsible for technical direction on the team. These boundary spanners will work closely with their peers, having regular coordination meetings across all subteams as well as impromptu one-on-one meetings to deal with specific issues between individual subteams.
6. Ensure that the global team gets the credit it deserves. In both offshoring and nearshoring environments it’s common to see small teams in North America or Europe driving the efforts of significantly larger teams in another country. Yet, at the end of the project it always seems as if the smaller team, often because they work for the direct customer, gets the lion’s share of the credit – unless of course the project failed, then the subcontracting team often seems to get virtually all of the “credit”. This clearly isn’t fair, and it clearly doesn’t promote effective teamwork between the subteams in the future.
7. Take a lean approach to development governance. As I’ve written in the past, effective governance is based on enablement and collaboration instead of the traditional approaches of management and control. Good governance measures progress through regular delivery of working software, not through status reports or delivery of detailed specifications. Good governance is based on the idea of having a living process which changes to reflect lessons learned as your project progresses. In a previous blog posting at http://www.ibm.com/developerworks/blogs/page/ambler?entry=lean_development_governance I’ve discussed lean development governance in greater detail.
The strategies that I’ve described are clearly nothing more than common sense, something that can be said of all agile strategies. Sadly, as Mark Twain lamented, common sense isn’t very common in practice.[Read More