I've been following the evolving story
about how the F.B.I. may have to scrap a $170 million computer system overhaul because users are rejecting it.
Compared to something like biology or civil engineering, software engineering is a young field - roughly 50 years old - yet in that time we have discovered a number of best practices that tend to result in the right system, on time and within budget. So why are stories about spectacular software development project failures so common? George Santayana
said it best and most famously: "Those who do not study history are doomed to repeat it."
Well-known lessons and best practices are often either not known or ignored. The result is that every year or two there is some spectacular software failure, and everyone in the media holds their hands up and says "Oh dear me, whatever is the matter with the software industry?" But when you examine what exactly went wrong on said disaster project, it's the same exact stuff you would have found on a disaster project from the 1960s!
I want to look at some specific problems mentioned in the NY Times article, but before doing so I want to preface my analysis with some humility. Even though the problems listed below are well-known and tactics to deal with the problems are well-known, I have no doubt that the people working on the F.B.I. project (both from the F.B.I. side and the service-provider side) were working to the best of their ability, and have probably worked many hours and made many sacrifices in an attempt to successfully deliver that system. To quote Grady
, "Software development has been, is, and will remain hard". It's much easier to play armchair quarterback, as I am about to do, than it is to successfully deliver software projects. I completely empathize with those involved, because I've been a part of failed projects, and it's not fun. Still, I think it's important to look at these well-known problems, and talk about tactics to deal with them.
Some quotes from the NY Times article
One idea under strong consideration is for the bureau to use off-the-shelf software instead of the expensive customized features it has unsuccessfully sought to develop.
It is a natural tendency to want to create software from scratch. There are two main factors: 1) exactness of capbilities and 2) ability to make modifications. However, quality software is hard to create, and many well-known problems have been solved by component vendors to a high-enough degree of quality to make the costs and risks of custom software highly untenable. In specialized domains, you will still need to write some
custom software. But for any part of your system that is not of unique value to your users, you should look to off-the-shelf components first. For instance, you do not want to spend many person months creating and debugging the world's 5,000th object-relational (O-R) persistence layer - you want to find an off-the-shelf persistence technology that works for you (like Hibernate), learn how to use it effectively, then drive on.
For more information on component-based development, Clemens Szyperski's excellent book: "Component Software: Beyond Object-Oriented Programming, 2nd Edition
" is strongly
recommended. Also one of RUP
's best practices is to use component architectures - so there is a lot good information in RUP on this topic.
"I did not get what I envisioned" from the project, the senior official acknowledged. But he said the F.B.I. today had a better understanding of its computer needs and limitations as a result of the effort." The lesson we have learned from this $170 million is invaluable," he said.
WHOA NELLY! Whenever you see something along the lines of "I didn't get what I envisioned" or even worse "I got what I asked for, but now that I see what that is, I realize I really wanted something different" you should immediately ask, "Did you do prototyping?" and "Were the actual system users involved in reviewing the prototypes?" It is very difficult for a system's future users to specify the system they truly need. As Suzanne and James Robinson point out in their recent excellent book, "Requirements-led Project Management
", this is generally because users don't think in abstract, essential terms, but rather their minds are "muddied" by prejudices about implicit technical constraints imposed by technology they understand - usually the current system. It is the analysts' job to find the essence of the problem at hand, rather than jumping to a solution without truly understanding the problem.
Once you have a fairly good idea of the true problem to be solved, the next step is to create some form of the actual system that can be seen and preferably touched by the actual users. This is simply because a visible working system is a much better point-of-truth than a bunch of words in a written specification. The key thing is that the first version of the system that the users touch should not be the final system
. This is for a very simple reason. The first attempt at the system will be wrong!
So instead of spending $170 million to determine "well, that's not exactly what we want", it is much better to create light-weight prototypes (using use cases, executable code, HTML, or pen and paper) to show the user what the envisioned solution to their problem will look and feel like. They will immediately spot deficiencies.
The difference here is that you've only spent a few thousand dollars on the "wrong" prototype rather than many millions of dollars on the wrong system. From the user's feedback you can quickly modify the prototype. It will still not be perfect, but it will be less wrong. And by the time you're getting to real code intended for production, the system will start to resemble the users' underlying intentions, rather than their original misconstrued interpretation of those intentions. If some ill-informed executive tells you that you either don't have time or money to do prototyping, write up a risk assessment using some historical data, and ask them to formally take ownership of the risk. It is likely that he or she will discover time and money to allocate to prototyping.
"As recently as last May, the F.B.I. was still claiming that V.C.F. would be completed by the end of 2004, and that it would at last give the F.B.I. the 'cutting-edge technology' it needs," the senator said.
Along similar lines to the previous item on prototyping, another well-known best practice is to incrementally deliver working versions of the system in relatively short, time-boxed durations, rather than trying to deliver the final system in one "big-bang delivery". The rationale for this is two-fold.
The first item is similar to prototyping - when user's get a working version of the system early, they're going to immediately spot problems with it - I'm talking problems with how the system "feels" - not bugs that can be caught in test. Hopefully you got 90% of these problems out during prototyping, but the first delivery of the system (unless you did a very sophisticated prototype) will be a new level of detail and immersion for the user, and they will inevitably dislike things about how the system works. It is much simpler to make adjustments mid-course if you have only implemented 10% of the total number of planned features, than if you have delivered the entire system (especially if the change requires you to tweak major system plumbing).
The second, and probably more crucial benefit of incremental delivery, is that it forces you to integrate the total system before you have written massive amounts of code. Integration is very hard, and in the big-bang approach it is frequently put off until near the end of the entire project. This can lead to a situation where you're "80% done for 200% of the allotted project schedule". It is very hard to speculate if this was the cause for the quote above, but if I were a betting man, I would bet that when system development was being advertised as on target, true integration hadn't happened yet. The RUP best practice "Develop iteratively" describes techniques around this area very well.
Well, enough for this post.
In closing, immerse yourself in software literature, and try to avoid the mistakes of your predecessors. To quote Benjamin Franklin: "Experience is the best teacher, but a fool will learn from no other."
In a later post, I will list some books I consider essential reading on software development. If you are into the literature, you will not be surprised by the list.
PS - I mention RUP several times as a place that defines and describes these best practices. I realize that RUP does not have a monopoly on these ideas - it is simply the process with which I am most familiar.[Read More