XP requires a profound shift in the way business people think. Over the past 30 years, software development methodologies have conditioned non-technical people in business to think and behave in certain ways. Unfortunately, traditional methodologies are wrong because they doesn't consistently produce the results these people want: software that meets their needs at the end of a project, software that has a long useful life, and software that gets delivered often.

Talk about a tall order. The sad truth is that traditional software development has consistently not delivered. One of the primary reasons is probably one of the least expected: customers haven't done their job. Before I explain, you need to understand what "the customer" for an XP project looks like.

The impossible dream?

XP tells a nice tale. The customer (or group of people speaking in agreement) tells the programmers what software to write. The customer identifies a feature and provides a reason why it is necessary. Ideally, the customer quantifies the feature -- what does it mean for the bottom line? While quantifying a feature isn't always possible, it's always a worthwhile goal. If it isn't possible, don't let that stop you from making great software, but don't assume it's always impossible. Try and see.

Business need should drive software, but more often than not, it's the technical people who lead the project. They build what they want to build, usually because customers go AWOL once the project starts, or the programmers give up talking to business people at some point. Why does this happen? The history of software development has conditioned everybody involved to think and behave in certain ways. Far too frequently, the result is disappointment, embarrassment, and frustration.

Back to top

Stuck on Taylorism

Frederick W. Taylor (see the sidebar from the October installment) was a machine shop foreman in the 1880s, and ended up studying factory production for over thirty years. He noticed one overriding characteristic: inefficiency. He set out to kill it by finding "the one best way" to do any job, and a system for telling managers how to find that way for every segment of their operations. By the 1950s, Taylorism was the primary management philosophy. Software development for business made its debut in that decade, and people running businesses made what looked like a good assumption. They assumed the same management principles -- the same way of thinking about work -- that applied to other kinds of production also applied to software.

Think about an automobile assembly line for a minute. Parts and raw material come in one end and finished cars come out the other. The last thing you want is for everybody on the line to get together and decide to build bicycles instead. There isn't any room for creativity on an assembly line. What you really want is predictability -- you want the line to run exactly the same every time. In this world, the problem and the solution are always the same, and you can effectively optimize the process. Of course, once you've got things optimized and running smoothly, change becomes the enemy. Even if the environment is dynamic underneath, people go to great pains to make it look predictable.

This model of "efficiency" couldn't be a worse fit for software. With software, the problem is always different. It may be similar to ones you've seen before, but it's never the same. That means the solution can't be the same. You can't just follow a manual to get the right answer. And if the problem's different and the solution's different, you can forget optimization. It just doesn't exist. Change is constant.

In a way, you could say software development teams (for internal or external users) are locked into assembly-line thinking. Taylorism is etched deeply in corporate culture. The people who act as customers for software projects are usually people in charge of groups of users, or are users themselves. If they're in charge of users, they're managers. If they're users themselves, they exist in an environment controlled by managers. In either case, assembly-line thinking is the norm.

Business people involved with software development have come to believe that their role is mostly a front-end one: they specify an exhaustive, unchanging set of requirements for the programmers early in the development cycle, then sit back and wait for the results. Of course, they should have control over the process. After all, they're paying for it, they're providing the requirements for it, and they're the ones who must be satisfied with the finished product. But the business rarely drives software. Technical people are quick to tell business people that software is "soft," but then these same technical people fight change, because it's painful and expensive. Programmers don't want to release software until it's perfect, and business people don't want the new software to be too disruptive for the users, so the two cooperate to release infrequently -- sometimes years after the project starts.

What you wind up with at the end of this cycle is software that isn't what the business needs -- it's what the business thought it needed when the project started. Most customers are not interested in buying such "vintage" software, but it's what they get. Technical people don't like to deliver software like this, but they believe they have no choice. Everybody feels frustrated, tired, angry, and short-changed.

Somewhere along the line, each person involved with creating software compromised. Maybe it was to keep a job or to minimize conflict. Maybe it was because fatalism crept up and attacked in the dark. Maybe it was laziness. Whatever it was, each person made a bad bargain. Each person decided that having things as they are was better than trying to make things different. Unfortunately, the road to better goes right through different. We need to get back on that road. One of the best places to start is with customer behavior.

Back to top

Learning to drive

Software development has been remarkably consistent over the past thirty years. There have been refinements along the way, of course, but the basic themes are the same:

• Users don't really know what they want, so developers have to tell them in order to produce anything.
  
  
• Change hurts, so we should suppress it almost at all costs.
  
  
• The way to suppress change is to write down everything at the beginning, get somebody to sign off on it, then stick to the plan.
  
  
• The only way to define all the requirements at the beginning is to have business people commit to a set of requirements, try the finished product at the end to confirm that everything's consistent with those requirements, and stay out of the way in the middle.

The result is that most software projects produce (if they produce anything at all) the software that somebody thought he wanted at the beginning, rather than the software users end up wanting at the end. We end up with exactly what we don't want, but there doesn't seem to be a way out of the rut. Folks in charge can't just dictate change. That has never worked in the past and never will. Why are we stuck here and what do we do about it?

First and foremost, people who are equipped to be XP customers need to start acting the part. Who can be a customer? Usually it's somebody with enough domain and end-user knowledge to be able to make the tough decisions about the priority of features. This could be a super-user who knows what users really need and want. It could be a marketing person who knows what the market is asking for and what competitors are doing. Rarely can it be a manager who gets the job by default, although that's probably better than no customer at all. Somebody has to take the job and commit to doing it.

Second, whoever takes the customer job has to see himself as part of the team. He has to be willing to spend as much time with the rest of the team as necessary to produce the right software. The customer can't specify requirements up front and then disappear, counting on plausible deniability to cover for himself if the software ends up a bomb. And the customer can't acquiesce when other people on the team tell him to go away and stop making trouble. He's not making trouble; he's giving the team direction.

Third, customers have to suspend disbelief and try to behave as if having the team explore its way to the final product will yield better results than specifying everything up front. This is hard for programmers and for customers. Programmers want to make great software without being hassled every step of the way. They think specifying everything up front and not deviating from the spec will ensure this. Customers are afraid, too. They want to throw in every possible requirement, including ones they don't really need, to give themselves more negotiating room when they have to give things up. Neither of these perspectives helps the team make great software. Customers need to say what the software really needs, and what is most important. Then they have to trust the programmers (yes, a hard thing to do) to work diligently toward that. Programmers, of course, have to earn more trust by doing what they said they would do. This is risky for everybody, but so is the typical way of doing things. With XP, though, the risk is obvious, instead of being hidden by mounds of paper and promises.

Fourth, customers have to change the way they think about software. How do you know what software you really need? There are only two approaches I know of: guess and hope you're right, or deliver the software in small bits and make adjustments as you get a better understanding of what the customer really needs. Spending lots of time having people talk about the software they need without actually using any software is just another version of the first approach. People are people. Having them use computers doesn't change that. The only way really to understand the requirements is to have a working example that both team members can use to find common ground: "See this here? That's not what I meant at all. That's really in a bad place because it gets in the way. It would be better over there. But this, this is perfect." Feedback is king, so the team should be releasing early and often. Waiting until the software is "perfect" only delays the bad news that you guessed wrong.

If this is all customers have to do, what's the problem? It's just not that easy. In fact, it's so difficult that most people have no desire to do it. Even if they do, they can't. Most organizations aren't set up to allow this process to happen. There is huge resistance to using XP to make software, because some powerful people have legitimate concerns about whether it's sensible -- and because some powerful people are stubborn.

Back to top

The organization might be the problem

An XP customer can't do the job if the organization he's part of doesn't make it possible. This means two things:

• An XP team must exist for the customer to be included.
• The customer's boss has to let the customer be part of that team.

Teams have to be using XP before an XP customer can do the job. If a team is pretending to use XP, or restricts the customer's ability to fill that role, it won't work. Unfortunately, most organizations are set up to oppose XP, either actively or by default. Unless at least a small part of an organization is ready to try XP, it won't fly. But you can start small. One customer, a few developers, and a smallish app is all you need. If that team produces great results, there is a chance other teams will want some of that action.

What if there's an XP team shopping for a customer, but the prospective customer's boss won't allow that person to spend any time with the team? Then the project is doomed at the start. If the team needs the customer for a planning session or to play with the latest app, but the customer is always busy filling out report A-324-XYZ for the upcoming bi-weekly status meeting, the team won't be able to deliver what they promised. You can't explore effectively without somebody who is an expert at similar terrain -- that's the customer. People in the organization who control that customer's time have to give up some of their claims.

Most organizations are set up to make this nearly impossible. So are we stuck forever? Maybe, but I don't think we have to be. Changing the situation simply takes courage.

Back to top

The courage to change

To be blunt, I believe there are three reasons corporate IT is stuck in a rut:

1. Habit
2. Cowardice
3. Lack of vision

Habits are hard to break. You don't even think about doing them. They're a natural part of your life, even if at some conscious level you know you shouldn't behave that way. This is why people smoke, or overeat, or scream at their kids. Once you have a habit, breaking it requires conscious effort. The way most corporations make software comes directly from the habits of the people who compose that organization.

A good number of people in corporations are gutless. When something really matters, and they really think the organization is headed in the wrong direction, they either cave in or they flee. Now, I'll be the first to agree that if life in an organization gets bad enough, and you can't improve it despite your best efforts, it's probably time to go. But running away at the first sign of resistance is not a solution. You're running from the inevitable. You'll go somewhere else where similar problems will crop up again, and you'll leave that new place too. Is it any wonder organizations don't change? Everybody runs away and leaves them broken.

What we need is a new generation of corporate leaders, both technical and non-technical, who will stand up and speak the truth -- even when it flies in the face of tradition and habit. The strategy of appeasement and retreat simply doesn't work. The non-technical leaders must act as customers for XP teams and participate full time if necessary. The technical leaders must be willing to give up the power to control the spec and explore their way to great software.

We need more brave people, but bravery without vision isn't worth much. We need people who have a vision of what could be and the courage to try to get there. In true XP fashion, the vision will change as we move toward it, but we have to move.

Software development hurts and produces junk because you and I allow it to. It's our fault. Blaming "those people" or "the organization" does nothing except perpetuate the cycle. We need to cry foul, require change, and start leading the charge ourselves. Doing that is difficult. In the end, it comes down to one thing: do you want to skate through life and collect your paycheck, or do you want to change the world?

Back to top

Next month

For the past few months I've told you what I'm going to write about next month. Now I want you to tell me what you'd like me to write about. Post to the discussion forum for this column, and give me your top five topics. If one particular topic reaches critical mass (a purely subjective judgment by me), I'll write about it in the next article. If nobody posts anything, or if votes are all over the place, I'll pick something I want to write about. This is your chance to get some coaching on demand. It's almost as good as being there.

Resources

• Participate in the discussion forum.
  
  
• Read all of the articles in the Demystifying Extreme Programming series.
  
  
  
  
• Read the original "XP distilled" (developerWorks, March 2001).
  
  
  
  
• Ron Jeffries' article "Business Analysis in Extreme Programming" talks about how business analysts fit into an XP team. They can really help customers and programmers do a better job.
  
  
  
  
• If you want to learn more about XP, be sure to pick up a copy of the following books:
  • Extreme Programming Explored by William Wake
    
    
  • Extreme Programming Applied: Playing to Win by Ken Auer and Roy Miller
    
    
  • Planning Extreme Programming by Kent Beck and Martin Fowler
    
    
  • Extreme Programming Explained: Embrace Change by Kent Beck
  
  
  
• Find hundreds of other Java technology-related resources at the developerWorks Java technology zone.
  
  

About the author

Rate this article

Comments

Back to top

Table of contents

• The impossible dream?
• Stuck on Taylorism
• Learning to drive
• The organization might be the problem
• The courage to change
• Next month
• Resources
• About the author
• Comments