Agile software development: A tour of its origins and authors
In the movie, The Princess Bride, the Dread Pirate Roberts is climbing up a rope, which Vizzini cuts. When Roberts doesn't fall, Vizzini says "He didn't fall? INCONCEIVABLE!" Inigo Montoya replies, "You keep using that word. I do not think it means what you think it means."
When I discuss the concept of Agility with people, I feel like Montoya in the above scene. I wonder if we're talking about the same thing. That doesn't mean that I'm right and they're wrong. It simply means that confusion surrounds the current usage of the word Agility.1 People in the software industry have a habit of making words mean whatever they want them to mean, especially new technical terminology. To some extent this is understandable: Our technologies change so rapidly that the number of new terms and acronyms that appear almost daily make it difficult to keep up, and we tend to be a little sloppy with our terms and hope that we're more right than wrong.
Even though Agility has been around for a while, many academics and practitioners misuse the term.
This month I'd like to present a survey of the Agile landscape. In order to begin, however, we need to establish a definition for the term. This is not easy. In fact, I'm not sure that I'm up to the task, but I'll try. Let's start with the first principles and then look at how these are often distorted by people who have developed a personal definition of Agility. After we (hopefully) agree on a definition, I'll review some of the books and references that I have found useful for navigating the Agile landscape.
At the beginning...
Before Feb. 2001, the word "agile" meant either "marked by ready ability to move with quick easy grace" or "having a quick resourceful and adaptable character."2 In 2001 this word began to mean something more specific for software developers. A group of seventeen people who refer to themselves as organizational anarchists,3 but were primarily software consultants and thought leaders in software development, gathered at Snowbird, Utah and defined Agile software development. Agile was a common ground upon which they could agree, although each attendee had his own idea about how to build high-quality software.
The common ground agreement resulting from the Snowbird meeting is The Agile Manifesto.4 I've discussed the manifesto before, but it's worth repeating the four values defined there because they give us the basis for understanding the intended meaning of Agility (with a capital 'A'). These values are phrased as preferences for one aspect of software development over another:
- Individuals and interactions over process and tools.
- Working software over comprehensive documentation.
- Customer collaboration over contract negotiation.
- Responding to change over following a plan.
That's it. It seems quite simple and straightforward. Yet, it has led to more misunderstanding and different interpretations than any word I can think of. Why? I can think of three reasons.
First, people understand the word "agile" as it is commonly used. When we speak of Agile development, listeners hear "agile" and, as I mentioned in the introduction, apply their own semantic filter to believe we are speaking of something fast-moving, something rapidly shifting gears and changing often. Certainly, many of our software projects do change rapidly and move quickly, but not all of them. Of course, the same problem would occur if the Snowbird attendees chose any other already-defined word to describe their approach to software development. At the time, many considered using lightweight to differentiate it from the perceived heavyweight processes they felt were foisted upon development organizations by large consulting companies.
Second, even when people know that there is a different meaning behind the word Agile, they think of their own personal definition. They may have read some articles or books about Agile development and tried to implement some practices that would make their projects more Agile (according to their definition). Unfortunately, people tend to distort the intended meaning of Agile, and that even includes some of the experts or people who have been involved in the Agile movement for some time. All you have to do is attend one of the Agile conferences and you'll see what I mean. Many have come to the conclusion that Agility is like art: "I know it when I see it," and "It's a very personal definition." Last year at the Agile 2006 conference, someone said they had implemented "full-blown" Agile. When I questioned them about what that meant, their definition was that they were doing unit testing and continuous integration. These are practices that can be applied to support the four values, but they are not, in themselves, Agile.
The final reason comes from the statement of the values. A lot of thought went into how the values were stated, but many people look a them and question -- rightly so -- the dichotomies embodied in them. For instance, is "comprehensive documentation" at the opposite pole from "working software"? The values seem to imply that but these two aspects are opposed to each other, but in fact there's no logical justification for that. So what we have are four pairs of values, each placed in opposition to the other four by the Agile Manifesto creators. My purpose here is not to argue the validity of the choice, but simply to say that you cannot consider being Agile unless you agree to the pairing and choose to value the first value over the second in each pair. If you decide that comprehensive documentation is more appropriately compared with individuals and interactions, then, strictly speaking, you do not have the vocabulary to discuss Agility in a comparative manner. This is the third reason for the confusion when we talk about Agility. Many of us do not agree with the initial pairings.
A scientific approach
One of the best ways to approach any subject, new or well-established, is to begin with the definitions, and then, through rigorous analysis and questioning, see where that leads. Let's do this with Agility.
If we consider the Agile Manifesto as the source document, we can determine that any organization or project that wishes to be Agile must value the characteristics named first in each of the four values (e.g., "customer collaboration") more than the characteristics named second (e.g., "contract negotiation"). Alistair Cockburn once told me that Agility is one position in a universe of sixteen possible positions. What he means is that if you consider each pair of values, you have the choice of valuing the first value over the second, or not. This is a binary decision. Since there are four of them, there are sixteen possible configurations. I think this is the simplest, clearest way to think about Agility. This avoids the problem of trying to distinguish between degrees of Agility. Using Cockburn's description, we can decide if an organization or project is Agile or not. If the organization possesses the values, and applies those values to the way it operates, then it is Agile. If it doesn't then it is not Agile.
We can approach Agility from a negative viewpoint as well, with apologies to Jeff Foxworthy.5 If you value the use of process and tools as much as or more than interaction of individuals, you are not Agile. If you value comprehensive documentation as much as or more than working software, you are not Agile. If you value contract negotiations as much or more than customer collaboration, you are not Agile. If you value following a plan as much or more than responding to change, then you are not Agile.
The key in the above wording is to notice that you don't have to value non-Agile characteristics more than the Agile characteristics. You just have to value them at least as much as the Agile characteristics. If you are working on a systems engineering project, I would think that you will value sticking to your plan as much as responding to change. You are dealing with hardware and software and need to work hard to bring them together according to a schedule. While it might be easy to change the software, changing the hardware is often difficult, if not impossible.
So, let me say it here -- and I'll take the hits from the Agile community -- not being Agile is okay. Agility is not something to strive for on its own. It is just something you may choose to strive for because it makes sense for your project and organization. When you speak with many in the Agile community you find that they understand this and agree with it. But, like the evangelist who has seen the light and wants to share it, they come on so strong that their zeal for their beliefs overwhelms you.
It's one thing to say you possess certain values and another to put those values into practice. The authors of the manifesto added a set of principles to follow if you want to be Agile.6 If you follow the principles in the way you do work, you are exhibiting Agile behavior.
Let's now agree upon the definition we will use for Agile and Agility for the rest of this paper. Agility is defined as holding the values defined in the Agile Manifesto and following the set of accompanying principles -- no more, no less.
What went wrong?
The definition I just stated is simple. Why then is there so much confusion and conflicting views on whether or not an organization or project is Agile? Part of the problem is that the principles are stated in a way that leaves a lot of room interpretation and implementation of those practices. Let's look at a couple of these.
One principle says "Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done." How do you implement this practice? First of all, you should ask what is a motivated individual? Some people are motivated simply by financial gain. Others are motivated by being part of a great team. One team's motivated individuals may be disruptive on another team. When you have a talent pool that has been pre-selected by your organization, you may not be able to have individuals who fit your definition of motivation on your project team. Can you still be Agile?
Another principle says "Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely." Clearly, if we set the bar low and do little work, that's a pace that we can maintain indefinitely. This is also clearly not the intention behind the principle.
We could take most of the principles individually and distort them somehow to go against the spirit of Agility. When taken as a complete body of work, however, they do tend to support each other. There is still, however, a lot of room to implement the principles. This has led to lucrative markets for Agile coaches, consultants, methodologists, and others who want to help an organization become Agile and adopt the consultant's flavor of Agility. It has also caused many organizations to adopt new practices and claim they have achieved Agility according to the methodologist's interpretation.
Having an interpretation of Agility is easy. Arriving at a valid interpretation may be more difficult. There is no single oracle we can ask to get the definitive answer about whether your project or organization is Agile. The question you really have to ask is, does it matter? If you are effectively producing software that meets all of your stakeholders' needs, you're in good shape. You should always be looking to improve, but do you really have to fit a label? This has been a problem in trying to follow CMM and CMMI, RUP, and other methodologies. People and organizations get caught up in being "certified" at a certain level rather than concentrating on the ultimate goal -- which is to deliver software. Methodologies and practices are means, they are not the end.
OK, if it's that simple...
One question that was posed at the Agile2006 conference, in Minneapolis, was "If Agility is so simple, how come there are so many books telling us how to do it?" This was said somewhat tongue-in-cheek, but there is a serious point to the question. Sitting in my office at school, I have one shelf that is almost completely devoted to books in the Agile category. There are more than thirty of them. I have about another dozen titles in my den at home. That's a lot of words. Why are there so many books on Agility?
The simple answer is that books are a great way for consultants to sell themselves. It's also a way for academics and practitioners to state their views about a topic. Books influence people and they have an effect upon the state-of-the-art and practice. Whenever new concepts are discovered, books, journal papers, and articles naturally follow. For those concepts that have a large effect, conferences emerge. The technology sector is an especially fertile ground for ideas and the books that follow, because, as I noted at the beginning of this article, it's hard for those of us in the IT industry -- juggling projects, meeting deadlines, staying within budget, all the while trying to satisfy our clients -- to keep up with the latest ways to accomplish our goals without good, self-guided help.
A second answer is that many authors who claim a new approach to a hot topic seem to be successful at publishing their books. As a result, each book purports to present something new and interesting about the authors' interpretations of the values and principles. Many of them present one way that leads to the implementation of the principles and values. Like many self-help books, they claim that you will become Agile if you follow their program. If you've ever tried different diets, relaxation programs, reading improvement programs, and so on, you know that what works for one person doesn't work for another. The same is true with many of the books on Agility. One method may work for project A, but fail miserably for project B. Each project and organization needs to find its own way to Agility, if being Agile is appropriate.
Agility today is everywhere. It is the ether that surrounds software development. If a practice is worth the time and effort to learn and apply, it must be Agile. If a tool is worth learning and using on our projects, it must support Agility. This is a marketing issue more than a matter of actual content. Agile is like the new brand of sneaker that we want to buy to help make our development teams jump higher, run faster.
I was in a bookstore the other day and picked up a book on the Ruby programming language. On the back cover it began: "Ruby is an agile object-oriented programming language. ..." They had the good sense not to capitalize the 'a' in agile, although I'm not sure the average reader would realize that or care. Does Ruby really embrace the values set forth in the Agile Manifesto? That seems like a ridiculous question, and logically it is. But we're talking marketing now and logic may not be the most important thing. Use the word "Agile" to catch your customer's attention, which lets you get your foot in the door. (I hope that someday we'll develop smarter customers who will ask the right questions and put Agility in perspective.)
I am not against Agility. I also am not what some call an Agilist. I would like to think that I'm a pragmatist and that I will use those things that help me and discard those that don't. Sometimes, Agility works. Sometimes I need something different. I think that's a pretty good place to be.
The top Agile books and methodologies
I'd like to use the rest of this article to provide a brief description of those books in the Agile space that I think are important. I'll tell you why I think each one is important and what it has to offer you. I'll present the list starting at a high-level view -- one that deals mainly with Agility in general, its values and principles. Then I'll go further down to specific methodologies and practices.
Books on Agile values and principles
Agile Software Development: the Cooperative Game, 2ed., Alistair Cockburn, Addison-Wesley Professional, 2006, ISBN 0321482751.
Alistair Cockburn writes well. This book provides one of the best descriptions of Agility from the point of view of one of the original Agile promoters. The writing is clear and quite balanced. Cockburn describes Agility and puts it in perspective to other places in the spectrum of values. He adeptly points out the sweet spot for Agile methods, why they work, and benefits you gain from them.
Cockburn's treatment is not a technical one. He doesn't get into coding issues and a lot of nitty-gritty details. He gives you enough material to be literate about Agility. Cockburn is known for his focus on the interpersonal issues in software development and spends a good amount of time discussing the people aspects of Agility. If someone knows nothing about Agile software development, this is the book I would recommend to them.
Agile & Iterative Software Development A Manager's Guide, Craig Larman, Addison-Wesley, 2004, ISBN 0131111558.
Craig Larman is a master teacher of software development, especially object-oriented, practices. He is well versed in different methodologies and knows how and when to apply them. In this book, Larman looks at iterative methods, Scrum, XP, RUP, and Evo. Scrum and XP are squarely in the Agile sector while RUP and Evo are more in the traditional (plan-driven) iterative camp. Larman compares and contrasts the different methodologies to help the readers evaluate their benefits and decide the type of process that is best for specific types of projects and organizations.
The comparison of the methodologies comes in the last half of the book. The first six chapters are the gold of this volume. In these chapters, Larman looks critically at software development, agility, iterative development, and presents the reader with evidence for using iterative and agile methods. The evidence comes from research, experience, and other sources. Larman has clearly done his homework.
Balancing Agility and Discipline A Guide for the Perplexed, Barry Boehm and Richard Turner, Addison-Wesley, 2004, ISBN 0321186125.
This book will appeal to those managers who come from large organizations and projects or have a solid background in software engineering (not software development).7 Boehm and Turner are academics who have experience with large projects, many of them in the defense industry. They approach the topic by trying to identify the places where each type of methodology can best be used -- the sweet spots. Boehm is best known for his development of the COCOMO model of project estimation and has written many seminal papers in software engineering, including the one that introduces iterative development.8
The main problem I have with the book is that I have little confidence that either Boehm or Turner have ever worked on the type of project that I usually spend time on. That is, one that is less than 100K lines of code. They focus their research on the huge projects upon which traditional software engineering has been effective. But I think it's nevertheless important to read this book, since it does approach Agility from a different perspective than most of the other books in this list.
Books on Agile methodologies
Agile Software Development with Scrum, Ken Schwaber and Mike Beedle, Prentice Hall, 2001, ISBN 0130676349.
Scrum has received a lot of attention in the last couple of years. It is a very simple method for project management and has loose connection to software development. For the most part, Scrum consists of a few practices that are not new, but are implemented rigorously. Scrum proponents claim great success applying it to software development projects of all sizes. There are no practices in the methodology that I would consider technical practices. They are all about project management.
This book describes the Scrum methodology in the words of the inventors of Scrum, Schwaber and Beedle. One of the nice things about Scrum is that its practices can be applied in conjunction with most other practices. It's worth reading this book just so you understand something about how Agile projects might approach project management.
Extreme Programming Explained: Embrace Change, 2ed., Kent Beck with Cynthia Andres, Addison-Wesley Professional, 2004, ISBN 0321278658.
This book, in its first edition, probably had more impact on launching the Agile movement than any other. In fact, many practitioners have come to equate Extreme Programming (XP) with Agile. For the software developer, XP is the methodology that most addresses the issues they're interested in. Beck, aided by Andres in the second edition, describes the fundamental practices that he has honed to the most used -- or believed to be the most used -- Agile methodology among software developers. I mention that it is believed to be the most used because there are few organizations that actually can apply all of the practices to their situation, but they still claim to be using XP.
The second edition of this book bulks up over the first edition, and not in very useful ways in my opinion. The latter edition adds more material about strategies and modifications to the basic XP methodology. This is unfortunate. If you can get a copy of the first edition and are new to Agile methods, I would recommend reading that to get a flavor of XP.
Extreme Programming Installed, Ron Jeffries, Ann Anderson, and Chet Hendrickson, Addison-Wesley Professional, 2000, ISBN 0201708426.
This is the second volume in the flood of XP-related books that Addison-Wesley published. It is worth reading because it describes the XP practices and how they were used by members of the original XP project team.9 The book is very readable and you will get a feel of what working on an XP project is like. This book was the one that convinced me that 100% application of XP practices was not the type of project I would choose to work on. It also helped convince me that there were some really good practices that I needed to learn. Even though XP has evolved, this is a great book to have an inexperienced team read before journeying on their maiden XP voyage.
Test Driven Development: A Practical Guide, David Astels, Prentice Hall Ptr, 2003, ISBN 0131016490.
I believe that test-driven development (TDD) is one of the most important practices to come out of the Agile movement. It puts an emphasis on quality and the responsibility for quality on the developer. It requires a quality focus throughout the development cycle, regardless of the methodology you apply it to. This book is a great introduction to TDD. It was the one that helped convince me of its power beyond simple testing.
Pragmatic Unit Testing in Java with JUnit, Andrew Hunt and David Thomas, The Pragmatic Programmers, LLC, 2003, ISBN 0974514012.
This complements the previous book with very practical advice on how to really implement the practice of TDD. The Pragmatic Programmers publishes a wonderful series of books that speak directly to software developers on topics of the latest technology. This book is one of their earlier publications and is a must-read for any Java programmer who wants to be good at writing unit tests. It places unit testing in the context of TDD and give the reader all of the tools needs to write, manage, and automate great unit tests.
User Stories Applied, Mike Cohn, Addison-Wesley Professional, 2004, ISBN 0321205685.
User stories seem to be the requirements specification vehicle of choice for many Agile projects; although other approaches, like use cases, are acceptable. The user story is a small piece of functionality that cam be described by the customer on an index card. This fits the very small iteration approach of XP and other Agile methods. Like writing use cases, writing user stories is a skill that must be learned and practiced. Mike Cohn provides the best introduction to writing user stories that I have seen. His book is to user stories what Alistair Cockburn's book is to use cases.10 If you work with use cases and haven't read Cockburn's book, Cohn gives you a complete course on how to write and apply user stories to your project. He uses a lot of examples and his experience on real projects clearly comes through. If you are an analyst who might even be mildly interested in user stories, you should read this book.
Planning Extreme Programming, Kent Beck and Martin Fowler, Addison-Wesley Professional, 2000, ISBN 0210710919.
Another key practice for XP projects is the planning game. This is a fairly simple set of activities that helps the customer and team decide what goes into each iteration, how to estimate the effort, and how to track the results so that you can become better at estimating. Beck and Fowler do a great job of describing the practice in a way that will interest developers, managers, and all other members of an XP team.
The following two books don't really fit the above categories, but I have found both of them useful and refer to them regularly. I use the second as the textbook for my software engineering course that I teach twice a year.
Agile Software Development Principles, Patterns, and Practices, Robert C. Martin, Prentice Hall, 2002, ISBN 0135974445.
This is a developer's book by a developer's developer. Bob Martin is a master developer who has a deep understanding of object-oriented and Agile principles. In this book Uncle Bob brings together the two and gives a developer a tour de force of object-oriented design principles and an insight on how to use them on an Agile project. This should be in every developer's library.
Extreme Software Engineering: A Hands-On Approach, Daniel H. Steinberg and Daniel W. Palmer, Prentice Hall, 2003, ISBN 013047812.
This is a small book that gives a fairly balanced look at Agile projects, with a preference for XP. It is not dogmatic in its approach and tries to show how Agility is not a haphazard approach to software development nor an excuse for doing things any old way. I find that the book is readable by my students and leaves me plenty of time to emphasize those aspects of software development that I think are important. This is a good weekend read.
Agility is everywhere. Ignore it and you will be lost in the current technical dialogue. Learn about it and you will be able to make intelligent decisions about whether it is right for you. You will also be able to understand many of the other practices and methodologies that are emerging. If you are a manager of technology at any level, this is your responsibility as well as a survival necessity.
I urge you to start looking at books I've listed above, as well as other books by authors such Mary Poppendieck (Lean development), Scott Ambler (databases), Jim Highsmith (management practices), and other writers who have either contributed directly to the Agile movement or have developed principles and practices that "play well" with Agile teams and projects. I hope that, like me, you will find some that give you ideas to try with your people, projects, and organization. No doubt, you will find a number of books to dismiss -- not because they are wrong, but because they are not right for your needs. Become an informed consumer, and you will increase your value to your organization.
1 I will use the capitalized form of the word to distinguish it from the common use of the word. This is the convention used by the people in the Agile community.
2 From the Merriam-Webster OnLine dictionary. http://www.merriam-webster.com/.
3 See the history of the Agile Manifesto. http://www.agilemanifesto.org/history.html.
5 Jeff Foxworthy is a comedian who is best known for making jokes that are in the form of: "If you ... then you are a redneck" (a member of the Southern U.S. rural laboring class who are stereotyped as simpletons).
7 See my Dec. 2005 column for what I consider to be the differences. http://www.ibm.com/developerworks/rational/library/dec05/pollice/index.html.
8 Barry Boehm, "A Spiral Model of Software Development and Enhancement", ACM SIGSOFT Software Engineering Notes, August 1986.
9 The project that launched the thousand books :-) was the Chrysler Comprehensive Compensation System, started in 1995. This is considered to be the project where all of the XP practices were applied, recorded, and refined into the methodology we now call XP.
10Writing Effective Use Cases, Alistair Cockburn, Addison-Wesley Professional, 2000, ISBN 0201702258. If you work with use cases and haven't read this, you're missing out.