"XP distilled" revisited, Part 3
Management on an XP project bears little resemblance to typical project management. This is a good thing.
Software development for business began in earnest in the 1950s. By then, Scientific Management, as articulated by Frederick W. Taylor, was the standard for all management in the United States. People in companies who made software to sell, or who used it to facilitate sales of other things, assumed Scientific Management applied to software development as well. They were wrong.
Taylor refined and articulated the principles of Scientific Management over a period from 1880 to 1911, when he published The Principles of Scientific Management. He believed problems in most companies at the time (like conflict between management and workers, low productivity, and low profits) grew from inefficiency. Taylor set out to fix the problem. Scientific Management was the ultimate result, with its emphasis on finding the "one best way" (that is, the most efficient set of steps) to do each job.
Taylor did his work in primarily manufacturing companies in the late 1800s and early 1900s. There were a few big competitors in each industry, markets were stable, and requirements were obvious. Does this describe the world of software today? Not even close. Our environment is full of competition (some gorillas perhaps, but lots of upstarts too). Markets are far from stable. Requirements change at least quarterly, if not monthly, weekly, or daily. Our context is different from Taylor's. And our problem is different.
Scientific Management might be appropriate for auto manufacturing assembly lines or steel refining, but not for software development. Scientific Management tells us to optimize our approach to everything. Find the "one best way" to do a job, which is the most efficient way, and then do it over and over. This assumes a particular kind of problem: a linear one. These kinds of problems are predictable. They're understandable. You can find a solvable equation to describe them. Creating software isn't like that. You can't always follow the same set of well-defined steps and get the right software out the other side. Requirements change in the middle, and we want them to so that we produce the software the customer wants at the end of the process, not the system he thought he wanted at the beginning. Our problem is different than Taylor's. Trying to apply a solution for one type of problem in a particular context to another type of problem in another context is a dangerous game. Yet this is precisely what we've done with software. We've tried to fit software development into an old way of thinking that no longer applies.
Scientific Management is management by command and control. The goal is predictability and stability. If we control the environment and the process, so the reasoning goes, outcomes will be more predictable. If outcomes are more predictable, we can plan better and manage better. Unfortunately, the software world isn't like this. Software emerges, whether we want to recognize that or not. We can't predict emergent reality very well. Stability is a fable in a world like this. We are confronting the stark reality that stability in any complex system (and in my upcoming book I argue that software development is one of those) means death. Stable projects die. Change is the norm, and change is good.
What we need is a way to move projects and IT organizations to what Chris Langton calls "the edge of chaos," where we have sort of a "stable instability." Change is constant there, but so is growth, innovation, and creativity. Complex systems experience maximum positive change at this edge. I believe we need to create a software development environment that allows this to happen. Such an environment would be characterized by a small set of simple rules, applied consistently, by imperfectly intelligent human agents, acting locally. In short, we need to create what John Holland of the Sante Fe Institute first called a Complex Adaptive System (CAS). To my way of thinking, this is the primary reason XP is worth talking about. I believe XP can help us create just such a CAS for software development.
Programmers are the imperfectly intelligent agents (I certainly qualify). The XP practices, and the fact that we do them all the time, are the simple rules applied consistently. The programmer team is the local environment for the CAS. Management in an environment like this means nudging the team in the right direction rather than enforcing rigid and elaborate rules. It means providing areas for growth and letting the team solve problems in ways you might not have thought of, rather than doing things the same way every time. It means allowing room for creativity, while keeping the team focused on the goal of delivering value to the customer in the form of working software. It is not about control; it is about gently directed growth.
Any software development approach that doesn't recognize the reality of emergence by acting like a CAS is, in my opinion, doomed to fail. Any approach that supposes management by command and control, rather than catalyzed emergence will get locked into a suboptimal outcome. XP can help you avoid both pitfalls. Of any approach I've ever seen, I believe XP has the best chance of success.