Agility@Scale: Strategies for Scaling Agile Software Development
with Tags: agility-at-scale X
ScottAmbler 120000HESD Tags:  distributed gdd agility-at-scale scaling-agile geographic-distribution sdcf 1 Comment 9,744 Views
I just wanted to round out my discussion about agile approaches to geographically distributed development (GDD) with a few important words of advice:1. Get some experience. Worry less about enterprise adoption and instead get started with a small project, or better yet a series of increasingly more complex projects. There will be learning experiences as you build a relationship with the offshore service provider. This advice is applicable whether you’re working with your own offshore division or with an independent service provider.2. Have a long-term staffing strategy. It may be great in the short term to have work done in a lower cost country, but how are you going to transfer the necessary skills to the maintenance and support team. Outsourcing that work is also an option, but it can be a risky one as you would need to build up expertise in “your” systems if you ever decide to insource that work again.3. Be concerned about intellectual property (IP). The rules are different around the world, and you may inadvertently be financing the creation of a new international competitor if you don’t have a clear division of ownership. And yes, this may mean that some components of your systems are still built internally by your own organization.4. Show off locally before you go global. GDD makes things harder to manage, so if you’re struggling to manage local teams you’re really going to struggle managing teams at a distance. Make sure you have local success first and are good at agile development in general. Furthermore, if your agile GDD projects run into trouble, don’t end your local agile adoption just because of difficulties with distributed projects.5. Let your offshore partners lead. The offshore partner likely has more experience than you at successful distributed development, and this is particularly true when you’re dealing with an established service provider.6. Do some reading. There’s a great IBM Redbook entitled “Global Development and Delivery in Practice: Experiences of the IBM Rational India Lab” which can be downloaded free of charge from http://www.redbooks.ibm.com/abstracts/sg247424.html7. Do some viewing. We recorded a Rational Chat a few months ago entitled "Being Agile in a Global Development Environment" which is posted at https://www14.software.ibm.com/webapp/iwm/web/reg/acceptSignup.do?lang=en_US&source=dw-c-wcsdpr&S_PKG=120607&S_TACT=105AGX23&S_CMP=TALKS&cp=UTF-8 . I also gave a keynote on Agile approaches to GDD at Software Development Practices 2007 held in Boston in the Autumn of 2007. The video can be downloaded free of charge from http://www.life20.net/video/scottambler.mov .[Read More]Modified by ScottAmbler
ScottAmbler 120000HESD Tags:  scaling-agile sdcf teams distributed governance amdd gdd agility-at-scale 10,598 Views
In my previous blog posting, http://www.ibm.com/developerworks/blogs/page/ambler?entry=strategies_for_distributed_agile_teams , I overviewed several strategies for improving your effectiveness at geographically distributed development (GDD). Those strategies were fairly generic and directly applicable to both traditional and agile development teams. In this posting I focus on strategies which are more agile in nature, although they could also be applied to more traditional approaches as well. These strategies are:Modified by ScottAmbler
1. Get the whole team together at the beginning of the project. Your goals are to build rapport amongst the team, to get to know the people that you’re working with to facilitate communication later on, and to better understand the situation on the ground. The implication is that you will need to fly some people around, increasing your initial expenses, an investment that many organizations balk at. The reality is that you will eventually end up paying for travel anyway, either because you actually flew people around or because your communication costs are higher throughout the project. In short, don’t be penny wise and pound foolish.
2. Organize your team around the system architecture. The most effective way to organize a distributed team is around the architecture of the system that you are building, not around the job functions of the people involved. In other words, if your team is in Toronto, Rome, and Bangalore then each subteam should be responsible for one or more subsystems. It would be a mistake to organize the teams around job function, for example to have the architects and analysts in Toronto, the developers in Rome, and the testers in Bangalore because this structure would require significantly more documentation and other forms of communication to coordinate the teams, increasing both cost and risk. As I mentioned in my previous blog posting you will need to invest in some initial architecture envisioning at the beginning of a project to identify the subsystems and their public interfaces, and that to do that you’ll also need to do some initial requirements envisioning to drive this architecture effort. I suggest that you take an Agile Model Driven Development (AMDD) approach to this to enable you to gain the value from modeling without the costs and risks associated with up-front comprehensive modeling and documentation that get many traditional project teams in trouble.
3. Have “daily stand-up meetings”. A common practice on co-located agile teams is to have daily stand-up meetings where people share the status of what they did yesterday, what they intend to do today, and whether they’re running into any problems. These short meetings enable team coordination. Distributed teams can do this as well, the people in a given geographical location can hold local stand-up meetings and then representatives from each location can hold a shared meeting to coordinate the subteams. Whereas local stand-up meetings are held first thing in the morning, distributed daily stand-up meetings may need to be held at unusual times so as to include people at distant locations.
4. Have Ambassadors. Ambassadors are people who travel between sites, often technically senior people or senior business experts, to share information between the subteams. Getting the team together at the beginning of the project sets the foundation for communication, but without continual investment in maintaining effective collaboration between teams you run the risk of your subteams deviating from the overall strategy. These are typically short engagements, a week or two in length, because of the pressures it puts on the people doing the actual traveling. The implication is that you’ll have several people flying between sites at any given time on a reasonable rotation schedule. Because you’ll have some people flying around, your local team rooms should accommodate visitors by having one or more desks available for them to use when they’re visiting.
5. Have Boundary Spanners. A boundary spanner is someone who is located on site who focuses on enabling communication between subteams as well as within their subteam. On large distributed teams you’ll find that you have three flavors of boundary spanners – team leaders who take on project management responsibilities on the subteam, product owners who are responsible for representing the business within the subteam, and architecture owners responsible for technical direction on the team. These boundary spanners will work closely with their peers, having regular coordination meetings across all subteams as well as impromptu one-on-one meetings to deal with specific issues between individual subteams.
6. Ensure that the global team gets the credit it deserves. In both offshoring and nearshoring environments it’s common to see small teams in North America or Europe driving the efforts of significantly larger teams in another country. Yet, at the end of the project it always seems as if the smaller team, often because they work for the direct customer, gets the lion’s share of the credit – unless of course the project failed, then the subcontracting team often seems to get virtually all of the “credit”. This clearly isn’t fair, and it clearly doesn’t promote effective teamwork between the subteams in the future.
7. Take a lean approach to development governance. As I’ve written in the past, effective governance is based on enablement and collaboration instead of the traditional approaches of management and control. Good governance measures progress through regular delivery of working software, not through status reports or delivery of detailed specifications. Good governance is based on the idea of having a living process which changes to reflect lessons learned as your project progresses. In a previous blog posting at http://www.ibm.com/developerworks/blogs/page/ambler?entry=lean_development_governance I’ve discussed lean development governance in greater detail.
The strategies that I’ve described are clearly nothing more than common sense, something that can be said of all agile strategies. Sadly, as Mark Twain lamented, common sense isn’t very common in practice.[Read More]
ScottAmbler 120000HESD Tags:  gdd scaling-agile agility-at-scale distributed teams 1 Comment 12,156 Views
A common misunderstanding about agile software development is that it’s only for co-located teams. Things are definitely easier for co-located teams, and as I found with both the Dr. Dobb’s 2007 and 2008 Agile Adoption surveys (www.ambysoft.com/surveys/) co-located agile teams appear to have a higher success rate than distributed teams, Having said that, many organizations are in fact succeeding at distributed agile development.Modified by ScottAmbler
I’d like to share some strategies that I’m seeing work in practice, and in this blog posting summarizes generic strategies for distributed teams whether or not they’re agile. These strategies are:1. Do some up front planning. Distributed development is higher risk than co-located development, and one way to address that risk is to think things through. That doesn’t mean that you need to create a monolithic, 1000+ task Gantt chart, but it does mean that you should identify your major dependencies and milestone dates. Effective teams do this planning with the distributed developers actively involved (they are part of the team after all), they strive to consider all associated costs, and in particular they don’t overlook the low probability/high impact risks which often prove to be project killers.
2. Organize the team effectively. Once of the practices of Lean Development Governance (https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?lang=en_US&source=swg-ldg) is to organize your team structure around either your architecture or the lines of business (LOB) supported by the programme that you’re working on. Ideally each sub-team should be responsible for one or more subsystems or modules, something that can be difficult if some of your team works alone from home, to reduce the amount of information sharing and collaboration required between disparate teams. In other words, maximize the responsibilities of the “offshore” team(s) as much as possible. A very common mistake is to organize the subteams around job specialties – for example the architects are in Toronto, the developers in Mumbai, and the testers in Singapore – because to support this team structure you have to create a phenomenal amount of documentation to support communication between the teams.
3. Do some up front modeling. The implication of organizing your team around the architecture (or LOB) is that you also need to do a bit of architecture envisioning up front. Your architecture efforts should provide guidance regarding the shared infrastructure as well as critical development conventions such as coding guidelines and data naming conventions. Architecture envisioning is also a good idea for co-located agile teams too. See http://www.agilemodeling.com/essays/initialArchitectureModeling.htm for strategies to get the benefits of architecture modeling without the costs of needless documentation.
4. Recognize that communication is critical. GDD puts many barriers to communication in place, increasing overall project risk. To overcome these risks you will first need to be aware of them and act accordingly, and second, you’ll need to write more documentation than you would likely prefer. The risks associated with long-distance communication include cultural differences, time-zone differences, and the challenges with written documentation (which is the least effective way to communicate information). I make it a habit of asking open-ended questions so that I can determine whether or not the other people understand the topic under conversation. Particularly I will never ask a yes/no style of question because the simple answer of yes can mean a range of things depending on the culture. It may mean “Yes, I heard you”, “Yes, I understand what you’re saying”, or “Yes, I understand and agree with you”. When you’re dealing with people at other locations it’s good practice to ask them to summarize the conversation in writing, in particular to identify key action items and ownership of them, to ensure that everyone agrees with what was discussed. A good approach is to have the team lead on other end to do the summary so that they own it going forward.
5. Put a good technical infrastructure in place. Automate, automate, automate. In a GDD environment you need to work with collaborative multi-site tools such as ClearCase, ClearQuest, and Jazz Rational Team Concert (www.jazz.net) which enable you to share and evolve your work products (i.e. test scripts, code, documents) effectively.
In my next posting I'll describe a collection of agile-specific strategies for distributed software development teams.[Read More]
ScottAmbler 120000HESD Tags:  agileexec disciplined-agile-deliver... agile agility-at-scale risk rup phases governance 6 Comments 10,693 Views
The explicit phases of the Unified Process -- Inception, Elaboration, Construction, and Transition -- and their milestones are important strategies for scaling agile software development to meet the real-world needs of modern organizations. Yes, I realize that this is heresy for hard-core agilists who can expound upon the evils of serial development, yet these very same people also take a phased approach to development although are loathe to admit it. The issue is that the UP phases are like seasons of a project: although you'll do the same types of activities all throughout a project, the extent to which you do them and the way in which you do them change depending on your goals. For example, at the beginning of a development project if you want to be effective you need to do basic things like identify the scope of the project, identify a viable architecture strategy, start putting together your team, and obtain support for the project. Towards the end of a project your focus is on the activities surrounding the deployment of your system into production, including end-of-lifecycle testing efforts, training, cleaning up of documentation, piloting the system with a subset of users, and so on. In between you focus on building the system, including analysis, design, testing, and coding of it. Your project clearly progresses through different phases, or call them seasons if the term phase doesn't suit you, whether your team is agile or not.
The UP defines four phases, each of which address a different kind of risk:1. Inception. This phase focuses on addressing business risk by having you drive to scope concurrence amongst your stakeholders. Most projects have a wide range of stakeholdres, and if they don't agree to the scope of the project and recognize that others have conflicting or higher priority needs you project risks getting mired in political infighting. In the Eclipse Way this is called the "Warm Up" iteration and in other agile processes "Iteration 0".2. Elaboration. The goal of this phase is to address technical risk by proving the architecture through code. You do this by building and end-to-end skeleton of your system which implements the highest-risk requirements. Some people will say that this approach isn't agile, that your stakeholders should by the only ones to prioritize requirements. Yes, I agree with that, but I also recognize that there are a wide range of stakeholders, including operations people and enterprise architects who are interested in the technical viability of your approach. I've also noticed that the high-risk requirements are often the high-business-value ones anyway, so you usually need to do very little reorganization of your requirements stack.3. Construction. This phase focuses on implementation risk, addressing it through the creation of working software each iteration. This phase is where you put the flesh onto the skeleton.4. Transition. The goal of this phase is to address deployment risk. There is usually a lot more to deploying software than simply copying a few files onto a server, as I indicated above. Deployment is often a complex and difficult task, one which you often need good guidance to succeed at.
Each phase ends with a milestone review, which could be as simple as a short meeting, where you meet with prime stakeholders who will make a "go/no-go" decision regarding your project. They should consider whether the project still makes sense, perhaps the situation has changed, and that you're addressing the project risks appropriately. This is important for "agile in the small" but also for "agile in the large" because at scale your risks are often much greater. Your prime stakeholders should also verify that you have in fact met the criteria for exiting the phase. For example, if you don't have an end-to-end working skeleton of your system then you're not ready to enter the Construction phase. Holding these sorts of milestone reviews improves your IT governance efforts by giving senior management valuable visibility at the level that they actually need: when you have dozens or hundreds of projects underway, you can't attend all of the daily stand up meetings of each team, nor do you even want to read summary status reports.
These milestone reviews enable you to lower project risk. Last Autumn I ran a survey via Dr. Dobb's Journal (www.ddj.com) which explore how people actually define success for IT projects and how successful we really were. We found that when people define success in their own terms that Agile has a 71% success rate compared with 63% for traditional approaches. Although it's nice to that Agile appears to be lower risk than traditional approaches, a 71% success rate still implies a 29% failure rate. The point is that it behooves us to actively monitor development projects to determine if they're on track, and if not either help them to get back on track or cancel them as soon as we possibly can. Hence the importance of occasional milestone reviews where you make go/no-go decisions. If you're interested in the details behind the project, they can be found at http://www.ambysoft.com/surveys/success2007.html .
Done right, phases are critical to your project success, particularly at scale. Yes, the traditional community seems to have gone overboard with phase-based approaches, but that doesn't mean that we need to make the same mistakes. Let's keep the benefit without the cost of needless bureaucracy.[Read More]
ScottAmbler 120000HESD Tags:  agility-at-scale agileadopt architecture reuse disciplined-agile-deliver... soa 2 Comments 10,016 Views
A common question that I keep running into with customers is whether you can take an agile approach to service oriented architecture (SOA). The quick answer is yes, because Agile is orthogonal to the implementation technologies used. You can take an agile approach developing COBOL applications running on mainframes, fat-client Java applications, multi-tier J2EE applications, and yes, even services. Granted, it's easier to do with some technologies than others, either because of the nature of the technology or because of the supporting tools.
The long answer is "yes, but". You don't adopt an SOA approach for the sheer joy of doing so, instead you very likely want to improve the level of reuse within your organization. To succeed at SOA-driven reuse you need an enterprise focus, something that doesn't appear to be very common on many agile teams. Therein lies the challenge. Several strategies for improving your chances with Agile SOA, and SOA in general, follows:1. Invest in some initial enterprise architecture modeling. You don't need to identify all of the details up front, that would take too long and actually put the effort at risk, but you do need to set a starting point to guide development teams. Identifying the technical architecture is critical, and identifying a few basic services which would provide immediate business value to one or more teams is critical. Involve people from several application project teams to ensure that you get a wide range of input. See http://www.agiledata.org/essays/enterpriseArchitecture.html for a streamlined approach to enterprise architecture modeling. Creating big, detailed models often proves to be a waste of time because development teams are rarely motivated to read mounds of documentation.2. Build out the initial infrastructure on a real application development project. This proves that your SOA strategy actually works and puts the technical foundation in place for future teams. During this period you'll be tempted to try to support several development teams, which is feasible but dramatically increases your risk. It's also tempting to focus simply on getting the infrastructure in place without delivering any business functionality, but this risks producing an ivory-tower architecture that nobody is interested in.3. Spread the service architects out onto application development teams. The people that formulated and then proved your SOA should be actively involved on the development teams that are working with it to ensure that the teams use it appropriately and to ensure that the architects get concrete feedback which they can use to evolve the architecture. When working on agile teams, these people will need to work in a collaborative and evolutionary approach just like other team members.4. Fund reuse separately. I've lost track of the number of organizations that I've run into that fail at reuse because their development teams never have the resources to develop reusable assets. That's simply the nature of the beast -- project teams will always be more interested in addressing their own specific requirements than they are in investing the time and effort to make something reusable. The real problem here is that you expect them to act differently. A better strategy is to have a separate reuse engineering team that has the resources to monitor existing projects to look for potentially reusable assets. When they find said assets this team does the work to harvest the asset, to reengineer it to make it reusable, and then to integrate back into the original source project. The goal is to make it as painless as possible to produce reusable assets such as services. If you expect project teams to do this work out of the goodness of their hearts then you're effectively punishing them when they do the right thing. That's not a very good governance strategy, IMHO.5. The reuse team now owns the asset. Any reusable asset, including services, will need to be maintained, evolved over time, and supported. This isn't free nor is it viable for project teams to do so.
If you're interested, I provide agile strategies for both enterprise architecture and strategic reuse in the book "Enterprise Unified Process". Although written under the assumption that you're taking a RUP-based approach to development, the reality is that the EUP can extend any evolutionary/agile software development process so that it addresses the larger-scale needs of modern IT organizations.
- Scott[Read More]
ScottAmbler 120000HESD Tags:  tdd requirements design specification agileadopt agility-at-scale amdd testing 3 Comments 8,656 Views
Test-driven development (TDD) is a common agile programming technique which has both specification and validation aspects. With TDD, you specify your software in detail on a just-in-time (JIT) basis via executable tests that are run in a regression manner to confirm that the system works to your current understanding of what your stakeholders require.
TDD is the combination of test-first development (TFD) and refactoring. With TFD, you write a single test (at either the requirements level with customer/acceptance tests or the design level with developer tests) and then you write just enough software to fulfill that test. Refactoring is a technique where you make a small change to your existing code to improve its design without changing its semantics.
TDD offers several benefits:1. It enables you to take small, safe steps during development, increasing programmer productivity.2. It increases quality. Agile developers are doing more testing, and doing it more often, than ever before. We're also fixing the problems that we find right on the spot.3. It helps to push validation activities early in the lifecycle, decreasing the average cost to fix defects (which rises exponentially the longer it takes you to detect them).4. Through single sourcing information, by treating tests as both specifications and as tests, we reduce the work required, increasing productivity.5. We leave behind valuable, up-to-date, detailed specifications for the people who come after us. Have you ever met a maintenance programmer who wouldn't want a full regression test suite for the code that they're working with?
But TDD isn't perfect. Although TDD is great at specifying code at a fine-grain level, tests simply don't scale to address higher level business process and architectural issues. Agile Model Driven Development (AMDD) enables you to scale TDD through initial envisioning of the requirements and architecture as well as just-in-time (JIT) modeling at the beginning and during construction iterations. To scale requirements-level TDD, you must recognize that customer tests are very good at specifying the details, but not so good at providing overall context. High-level business process models, conceptual domain models, and use cases are good at doing so, and these work products are often created as part of your initial requirements envisioning and iteration modeling activities. Similarly, to scale design-level TDD you must recognize that developer tests are very finely grained but once again do not provide overall context. High-level architecture sketches created during envisioning activities help set your initial technical direction. During each construction iteration, you'll do more detailed design modeling to think through critical issues before you implement them via TDD.
You also need to scale the validation aspects of TDD. TDD is in effect an approach to confirmatory testing where you validate the system to the level of your understanding of the requirements. The fundamental challenge with confirmatory testing, and hence TDD, is that it assumes that stakeholders actually know and can describe their requirements. Therefore you need to add investigative testing practices which explore issues that your stakeholders may not have thought of, such as usability issues, system integration issues, production performance issues, security issues, and a multitude of others.
For further reading, I suggest:1. My article "Introduction to TFD/TDD" at http://www.agiledata.org/essays/tdd.html which overviews TDD.2. My February 2008 column in Dr. Dobb's Journal entitled "Scaling TDD" at http://www.ddj.com/architect/205207998 which explores this issue in detail. 3. Andrew Glover's article "In pursuit of code quality: Adventures in behavior-driven development" at http://www.ibm.com/developerworks/java/library/j-cq09187/ which describes a new-and-improved take on TDD called BDD.[Read More]
It's customary to start a blog by describing the vision for it. Although this vision will undoubtedly evolve over time, it's always good to put a stake in the ground to get things started. Agile software development is clearly taking off and in my opinion is becoming the dominant development paradigm. Furthermore it appears that Agile approaches enjoy a higher success rate, providing better value for your IT investment, than do traditional approaches. Although organizations are succeeding at simpler projects with agile, many are struggling when applying Agile in more complex situations. They're finding that the "Agile rhetoric" doesn't always live up to its promises once you move into these complex situations. My goal with this blog is to share strategies for applying Agile techniques at scale.
When applying Agile strategies at scale you are likely to run into one or more of the following complexity factors:1. Geographical distribution. Is your team, including stakeholders, in different locations? Even being in different cubicles within the same building can erect barriers to communication, let alone being in different cities or even on different continents.2. Regulatory compliance. Regulations, including the Sarbanes-Oxley act, BASEL-II, and FDA statutes, to name a few, can increase the documentation and process burden on your projects. Complying to these regulations while still remaining as agile as possible can be a challenge.3. Entrenched policies, people, and processes. Most agile teams need to work within the scope of a larger organization, and that larger organization isn't always perfectly agile. Hopefully that will change in time, but we still need to get the job done right now. Your existing culture and organization can really hinder your ability to scale agile approaches, then a few "simple" changes can really help your efforts.4. Legacy systems. Although the politically correct term would be "proven assets" the reality is that it can be very difficult to leverage existing code and data sources due to quality problems. The code may not be well written, documented, or even have tests in place, yet that doesn't mean that your agile team should rewrite everything from scratch. Some legacy data sources are questionable at best, or the owners of those data sources difficult to work with, yet that doesn't given an agile team license to create yet another database.5. Organizational distribution. When your teams are made up of people working for different divisions, or if you have people from different companies (such as contractors, partners, or consultants), then your management complexity rises.6. Degree of governance. If you have one or more IT projects then you have an IT governance process in place. How formal it is, how explicit it is, and how effective it is will be up to you. IBM has been doing a lot of work in this topic over the past few years, and just recently Per Kroll and I have done some work around Lean Governance strategies. 7. Team size. Large teams will be organized differently than small teams, and they'll work differently too.8. System complexity. The more complex the system the greater the need for a viable architectural strategy. An interesting feature of the Rational Unified Process (RUP) is that it's Elaboration phase's primary goal is to prove the architecture via the creation of an end-to-end, working skeleton of the system. This risk-reduction technique is clearly a concept which Extreme Programming (XP) and Scrum teams can clearly benefit from.
It is definitely possible to scale Agile software development to meet the real-world complexities faced by modern organizations. Based on my experiences, I believe that over the next few years we'll discover that Agile scales better than traditional approaches. Many people have already discovered this, but as an industry I believe that there isn't yet sufficient evidence to state this as more than opinion. My goal with this blog is to provide advice for scaling Agile so as to increase your chances of success.
So, it looks like I have my work cut out for me. My strategy will be to address common questions which I get when working with customers and with internal IBM development teams. I have the privilege to work with a variety of software development teams worldwide, helping them to become more agile. They're all struggling with the same basic issues although don't recognize it because they're too focused on their own situation. So hopefully I'll be able to spread the word about what's actually working in practice.
I hope that you stay tuned.
- Scott[Read More]