People often ask me if I think that Agility is appropriate for large projects or whether it applies only to small projects and teams. I have come to realize that this is a poorly worded question. You might just as well ask if cooking works for feeding 1000 people as well as it does for a family of four. It certainly does work, but you use vastly different methods, recipes, and utensils. Instead of mixing a salad dressing in a small jar, you might have to prepare it in a 20-gallon container. This article examines the role of Agility in large projects and considers better questions we might ask about its scalability.

Why the question is wrong

Ron Jeffries, a well-known Extreme Programming (XP) practitioner, claims that XP practices work just fine at any scale. He says that when people object to Agile practices on large projects they actually do not understand Agility; therefore, they are not able to assess its suitability for large projects.¹ I agree with him. I will not going to go into a discussion of the definition of Agility, which I've already done in my May 2007 article in The Rational Edge.² I will assume you've read that article as well as the Agile Manifesto and its principles.³

The Agile Manifesto makes no specific references to project size. In fact, the manifesto is a declaration of values -- values that the manifesto signatories choose to apply to software development. If you work in an organization that has such values, or the project team you work on has those values, then you may decide that you can use Agile practices. If for some reason you don't have or cannot adopt these values, then you should not worry about Agility. Contrary to some popular belief, this does not label you as a bad person!

Since values are attributes of an organization, it should now be apparent why the question "Does Agility scale" is a poor one. Values define how the organization carries out business, not the size of the organization or project. Clearly large organizations typically have more processes and governance in place, but the core values are independent of these.

I want to try a small experiment. I did a search of several companies, large and small. I looked at their mission statements and information about their philosophies of how they conduct business. Read the following and see if you can identify the company or at least the size of the company and its software projects.

1. Company A's philosophy lists several things they believe in. The following ones are:
   1. Focus on the user and all else will follow.
      
      
   2. Fast is better than slow.
      
      
   3. You don't need to be at your desk to need an answer.
      
      
   4. You can be serious without a suit.
      
      
   5. Great just isn't good enough.
2. Company B's principles state that they are "dedicated to building software quickly and building it right. The most successful software projects are a combination of strong management, superior technical skills, and the use of well-defined development methodologies or processes."
   
   
3. Company C has three values they highlight:
   1. Dedication to every client's success.
      
      
   2. Innovation that matters, for our company and for the world.
      
      
   3. Trust and personal responsibility in all relationships.
4. Company D values integrity, honesty, openness, personal excellence, constructive self-criticism, continual self-improvement, and mutual respect. They are committed to their customers and partners and have a passion for technology. They take on big challenges, and pride themselves on seeing them through. They hold themselves accountable to their customers, shareholders, partners, and employees by honoring their commitments, providing results, and striving for the highest quality.
   
   
5. Company E has one statement describing their mission: "Serve the client through the conception and execution of innovative, intelligent, usable digital media solutions."
   
   
6. Company F believes:
   1. The best projects start with clear and tangible goals.
      
      
   2. The best sites are standards compliant.
      
      
   3. The best software is developed iteratively and with strong customer involvement.
      
      
   4. The best code starts with tests.

Needless to say, these mission statements and commitments are all very noble. Now, were you able to determine anything about the company or its size? I won't name the companies here, but I'll simply characterize them as follows:

1. A very large international company many of us interact with everyday.
   
   
2. A small company that specializes in coaching XP teams and writing software for their clients.
   
   
3. One of the largest technology companies in the world.
   
   
4. One of the largest software companies in the world.
   
   
5. A moderate-sized company that builds Web applications and other systems for their clients.
   
   
6. A very small consulting firm started by one of my former students.

When I looked for these companies and their missions and values, I found nothing that hinted about their size or their "agileness."

What Agile practices scale well for larger projects?

What, then, regarding Agility on large projects is the right question to ask? I believe it is "What practices support Agility in large organizations and projects?"

It's likely that companies of greatly different size use different processes and practices. A project involving hundreds of people must necessarily have different controls and constraints compared to a project involving one or two people. Communication channels grow exponentially as the number of project members increases. In order to capture important communication, larger teams must use more rigor and formality in the way they work. All of the companies I characterized above conduct projects where Agile methods are appropriate. Perhaps not all projects warrant Agility, but many do. Such projects have regular interaction with the customer and must deal with rapid change to requirements. The development team members must collaborate very closely with each other and make decisions about the system quickly with little delay due to process overhead.

Let's look at some Agile practices and see how well they scale up for larger projects. Let's agree that a larger project has twenty or more people working on it, although this is an arbitrary threshold. You can determine if my reasoning below needs adjustment for your notion of a large project.

Ron Jeffries says that most XP practices scale to larger projects. Most people interested in Agile know something about XP, which makes this a good place to begin. Ron's XProgramming.com Web site identifies thirteen practices.⁴ What follows is my assessment of whether or not these practices scale well. For each practice I indicate whether I think it scales and present a very brief rationale for my opinion.

Whole Team (no). This practice requires all team members to sit together. There must be at least one customer representative team member. Most large teams are distributed, so while it's possible to have a customer representative at each location, this may not be economically viable. Additionally, finding a single room where twenty or more people can work may be very difficult. Often a large team will have smaller sub-teams form as appropriate during the lifetime of the project. Each of these teams may be able to use the Whole Team practice.

Planning Game (yes). The planning game lets a team simply plan and track work by iteration. Regardless of the tools that a team might use to capture requirements or the length of an iteration, the planning game makes it easy to produce realistic iteration plans.

Small Releases (maybe). "Small" is a subjective term, so a "small" release depends upon your definition. However, most software-intensive systems allow a team to select a portion of the final system to implement completely for one of the small releases. This follows the principle of iterative and incremental software development and has worked well on various-sized projects that used the IBM® Rational® Unified Process (RUP). Some systems, especially those that involve the integration of custom hardware with software, may not be amenable to using Small Releases.

Customer Tests (yes). Who better knows what the system must do than the customer? The customer representative can write the test cases or help write the test cases on any project. In fact, on a less Agile project that is contract-driven, the customer often specifies the acceptance tests as part of the contract or as an addendum to it.

Simple Design (maybe). A design should be as simple as possible, but no simpler. That's the ideal. This practice, as described by Jeffries, is really incremental design. The amount of design needed on a project should be determined by the type of project and the project requirements.

Pair Programming (yes). Pair programming has been shown to produce better quality code without undue overhead. As long as two team members are available to work together the practice can work. If the team is completely distributed, the practice will not work,⁵ but this has nothing to do with project size.

Test-Driven Development (yes). Developers who have a serious commitment to quality should be writing unit tests for their work. If they write unit tests, they can write them before implementing the code. The choice of when to write tests is independent of project size.

Refactoring (yes). Good programmers continually work to create more robust, more maintainable and efficient code. Again, project size has little to do with this.

Continuous Integration (maybe). Small projects can use simple tools to implement a continuous integration practice. Larger projects that have multiple locations and multiple code repositories may not be able to build the complete project continuously. They certainly will need more sophisticated tools in order to achieve an effective continuous integration process. Economic considerations might be the overriding factor as to whether this practice can be adopted.

Collective Code Ownership (doubtful). While members of a sub-team might understand all of the code that their sub-team creates, the chances that they understand the details of other teams' work well enough to make changes is slim. Typically code ownership will be partitioned among the developers.

Coding Standard (yes). A project of any size without a coding standard is asking for trouble.

Metaphor (no). Devoted XP followers will take issue with me on this. Some have been trying for years to get people to realize the value of metaphor. I'm still not convinced. Clearly, when we develop an architecture we use many metaphors, but it certainly does not take the place of an architecture. As projects get larger, trying to have the right set of metaphors would, in my opinion, be less than cost effective.

Sustainable Pace (yes). This is more about the organization and project culture than organization and project size.

I agree with Jeffries' assessment that most XP practices scale well to larger projects. What about other Agile processes?

A note on Scrum

Another popular one is Scrum.⁶ Scrum is a project management process that can be applied to many types of work.

Fundamentally, Scrum is an incremental, iterative process for building products. It works best when building new products, but can be used in other situations. Scrum's primary characteristics are the daily cycle and the 30-day cycle. Figure 1 is a diagram, borrowed from the Wikipedia Web page on Scrum⁷, shows this:

Figure 1: Scrum cycles.

Scrum takes some very basic ideas and organizes them in a way that works well for software development teams. The specific practices are:

1. Iterative development. This is the de rigueur practice for most software development processes, Agile or not. In Scrum the iterations, called sprints, are defined to be 30 days long.
   
   
2. Daily cycle anchored by a short stand up meeting where three questions are answered by each team member:
   1. What did I accomplish since the last meeting?
      
      
   2. What obstacles are keeping me from achieving my goals?
      
      
   3. What do I plan to do by the next meeting?

One person has the role of Scrum Master and is responsible for removing the obstacles.

• Two backlogs: the product backlog of all the requirements for the product and the sprint backlog containing the requirements planned for the current sprint. The sprint backlog does not change during the sprint.
  
  
• Retrospectives at the end of each sprint to improve the process.

None of the Scrum concepts are particularly new. Scrum organizes them in a way that is low overhead and relies on self-organizing teams.

Does Scrum scale to larger projects? Ken Schwaber, one of the Scrum inventors, says that it does. He indicates that the scaling occurs by creating "scrums of scrums."⁸ Scrum practitioners say that this technique works for large teams of over 500 people. I am not sure that these teams are all working on a single project.

There are many other Agile methodologies that we could discuss regarding their applicability to large projects. You can perform the exercise on your own. Here, I want suggest a way to use your time more profitably.

Reframing the problem

I think we fall into a trap when we try to determine how we can apply the "hot" technologies, tools, and processes to our projects by focusing on the technology or process rather than the project. When I worked on the team that designed and developed the Rational Unified Process, or RUP, I visited customers and found that those who were successful focused on the project and the work that needed to be done for the project's success.

One of the misconceptions about RUP has been that it is a process rather than a process framework. The successful RUP adoption started with their project and customized RUP to fit their project and organization. I think the same approach works when we try to adapt any methodology to a project and organization, including the application of Agile techniques.

Identify values

Any attempt to introduce change into an organization will fail unless the change is compatible with the organization's core values. Consider, for example, trying to introduce a software reuse practice into your organization. The benefits of software reuse are well known. But if the organization rewards developers by the amount of code they write, this value may be hard to instill. Developers know how they are evaluated. Programmers will likely go out of their way to make sure they don't reuse a module because it would result in fewer lines of code to write, which might suggest poor performance.

If we assume that in your adoption of Agile practices you are not going to change the organization's values, then you must select practices that are compatible with the actual values at work if you want your Agile adoption to succeed.

Identify the context

Projects and organizations exist in a context. The context is the environment in which the project or organization exists. Some typical features of the context are:

• Team location. Is the team co-located (i.e., in a single area)? Can team members meet regularly, formally, or informally? If they are not in a single area, how geographically distributed are they? Are they separated by buildings on the same campus or by several time zones?
  
  
• Existing practices and tools. If you incrementally introduce Agile practices they must complement the existing processes and tools. Change must be integrated as seamlessly as possible. Any change initially causes a drop in productivity while the team integrates it into their way of working. If the change is not compatible with existing practices, this drop can be too great and the new practice may never yield the desired results.
  
  
• People. Since people are ultimately who will use new processes and practices, their personalities and work habits must be considered. If your organization has been successful with a group of independent-minded contributors, introducing a process that requires more collaboration and communication may fail with such a group.
  
  
• Ceremony. Many organizations operate with a lot of ceremony, or formality. Government projects typically run in such contexts. Well-defined milestones require formal reviews and a tremendous amount of exactly formatted and prepared documentation supporting the reviews. Practices that stress less ceremony in order to enhance other features of the project will have a hard time gaining traction in such organizations.

Other factors can be added to determining the context depending upon the project and organization.

Define success

After you have identified the operating values and context for your organization and project, you still have the most important question to answer: What is the ultimate goal of the new practice or process? In other words, What defines success?

I submit that "being more Agile" is not a goal. That's too easy. Many managers and executives today have been bombarded by Agile media articles. They hear about Agile from developers who are looking for better ways to use their time writing code rather than working on supporting other facets of software development. However, trying to be Agile is not the end in itself. There must be one or more tangible business values identified as the goal of any change.

Once you have done your due diligence, you can then ask the right question: "What practices can we incorporate into our existing context that match our values, that our people will respond to positively, and will achieve our business goals?" If your values and context indicate an Agile position, then you should certainly look into the Agile methods. As I've illustrated, the project size probably does not matter.

A few references to consider

I recommend that you become familiar with the fundamentals of Agility and several Agile methodologies in order to help you decide what practices best suite your needs. The following books and Web sites, in my opinion, will give you a good base from which to reason well about the issues.

Agile Software Development 2nd edition.: The Cooperative Game, Alistair Cockburn, Pearson Education, 2007. Alistair Cockburn takes a broad view of the software development discipline. He has defined a set of processes -- his Crystal family -- and classifies process applicability by several dimensions, including size. This book provides a good basis for understanding Agility.

Agile Software Development in the Large: Diving into the Deep, Jutta Eckstein, Dorset House, 2004. Jutta Eckstein has focused much of her consulting and research on how Agile methods can be applied to software development in the large. This book gives her view of the issues of applying Agile methods to larger projects, how to do it, and how to do it well. If you are looking to apply Agile methods in a large project or organization, this is a must read.

Martin Fowler's Web pages, http://www.martinfowler.com/. I love Martin Fowler's writings. He and Craig Larman are, in my opinion, two of the best thinkers about software development. They write things I want to read. Martin's articles on Agile and XP should be on your hot list.

Speaking of Craig Larman, his Web site at http://www.craiglarman.com is quite valuable. And his book, Agile and Iterative Development: A Manager's Guide, Addison-Wesley, 2003, offers a wonderful comparison of four methodologies. After reading this, you will understand how to compare and contrast others.

Conclusion

Every professional manager and development professional wants to do better. Today Agile methods seem to offer several benefits for small teams and projects, leading us to think how we might realize such benefits in a larger context. The way to approach this is to first focus on the desired results and then determine what practices might be best rather than deciding upon Agility before understanding the desired results.

Whatever you do, base your practices on iterative development if possible. There are very few situations where a waterfall approach works. All modern methods are based upon iterative, incremental development.⁹ Using this as a base, and add the Agile practices that are compatible with your organization and project culture.

Back to top

Notes

1. See http://forum.agilesoftwaredevelopment.org/viewthread.php?tid=139.
   
   
2. See http://www.ibm.com/developerworks/rational/library/mar07/pollice/index.html.
   
   
3. See http://www.agilemanifesto.org/ and http://www.agilemanifesto.org/principles.html.
   
   
4. See http://www.xprogramming.com/xpmag/whatisxp.htm.
   
   
5. Several experimental systems have been built to support distributed pair programming. The results are inconclusive thus far.
   
   
6. See http://www.controlchaos.com for information about Scrum.
   
   
7. See http://en.wikipedia.org/wiki/Scrum_(development)
   
   
8. A description of this concept may be found at http://www.mountaingoatsoftware.com/scrum-team.
   
   
9. One trap many people fall into is equating Agile as the opposite of waterfall. This is not the case. Iterative development is the opposite of waterfall. You do not have to be Agile in order to practice iterative development.

Resources

Learn

• Learn about other applications in the IBM Rational Software Delivery Platform, including collaboration tools for parallel development and geographically dispersed teams, plus specialized software for architecture management, asset management, change and release management, integrated requirements management, process and portfolio management, and quality management.
  
  
• Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
  
  
• Explore Rational computer-based, Web-based, and instructor-led online courses. Hone your skills and learn more about Rational tools with these courses, which range from introductory to advanced. The courses on this catalog are available for purchase through computer-based training or Web-based training. Additionally, some "Getting Started" courses are available free of charge.
  
  
• Subscribe to the Rational Edge newsletter for articles on the concepts behind effective software development.
  
  
• Subscribe to the IBM developerWorks newsletter, a weekly update on the best of developerWorks tutorials, articles, downloads, community activities, webcasts and events.
  
  
• Browse the technology bookstore for books on these and other technical topics.
  
  

Get products and technologies

• Download trial versions of IBM Rational software.
  
  
• Download these IBM product evaluation versions and get your hands on application development tools and middleware products from DB2®, Lotus®, Tivoli®, and WebSphere®.
  
  

Discuss

• Participate in the discussion forum.
  
  
• Check out developerWorks blogs and get involved in the developerWorks community.
  
  

About the author

Gary Pollice is a professor of practice at Worcester Polytechnic Institute, in Worcester, MA. He teaches software engineering, design, testing, and other computer science courses, and also directs student projects. Before entering the academic world, he spent more than thirty-five years developing various kinds of software, from business applications to compilers and tools. His last industry job was with IBM Rational Software, where he was known as "the RUP Curmudgeon" and was also a member of the original Rational Suite team. He is the primary author of Software Development for Small Teams: A RUP-Centric Approach, published by Addison-Wesley in 2004. He holds a B.A. in mathematics and an M.S. in computer science.

Comments

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Notes
• Resources
• About the author
• Comments

Next steps from IBM

• Try: Rational Method Composer
• Briefing: Jazz and Rational Team Concert
• Buy: Rational Method Composer

My developerWorks community

Dig deeper into Rational on developerWorks

Cloud computing for developers

Virtual workshop: Learn how to build a cloud infrastructure and develop/deploy cloud-based applications. (6-8 Apr 2010)

Special offers