This is my first developerWorks blog posting. It is something I wrote a while ago, but I had some very positive feedback on this whilst at Innovate 2011, so I thought it worthwhile sharing again.
From my experiences of working with many software project teams over the years, I have regularly been frustrated with the instinctive response of the project to revert to a waterfall mindset at the first sign of project difficulty. The moment anything "scares" them, like a risk, technical complexity, a large project team, long delivery times-scales, or even a new technique or technology, they go running for cover and hold on tightly to the one thing that gives them a feeling of security, their "waterfall comfort blanket".
So, where did this phenomenon come from, why is this such an instinctive reaction, and does it help them?
Waterfall, or the linear software development lifecycle, has been present in the industry for a long time despite software leaders in the past trying many times to discredit the approach, it still remains commonplace for software delivery projects, even if they say they are doing otherwise. The use of iterative and agile practices, which are widely recognised as superior to a waterfall approach, are rarely adopted well. There are a number of reasons for this, so here I will explore a few key examples.
Unfortunately, the world of academia, the first place so many of our software colleagues learn about how to organise and manage a software project, is still very slow to catch up with iterative and agile practices. So many graduates are arriving in the workplace having been primarily educated using the waterfall model as their primary lifecycle, with only side reference made to these more modern approaches, if they are lucky.
This may be because it is easier to learn and to teach waterfall as opposed to iterative practices. Waterfall can be learnt as a step by step set of instructions, which is how most things are taught. Iterative is much harder to explain if you’re used to explaining or follow a step by step process. However, all hope should not be lost, as some academic subjects do naturally use iterative principles; art for example, setting a context and prototyping with pencil sketches, prior to investing cost and effort with canvas, paints and hours of detail.
I have often heard it said that an iterative or agile approach isn't feasible because of the constraints imposed upon a project by delivery contracts. It's true that this is a common issue for software delivery organisations, but these constraints have typically come about from a breakdown of trust between the delivery team and its customer, typically resulting in a scenario such as:
- The customer has little confidence in the delivery team and demands that they provide a fixed-price for the delivery of the solution.
- In reaction to this fixed-price constraint, the delivery team requests that the customer defines everything they need in detail before a price will be given.
- The customer, knowing that this is their one chance to specify what they want or risk severe change control later on, provides the details of everything they might ever want.
- The delivery team ties the customer to this specification, unless they pay extra to change it, resulting in a solution that provides functions that are either never used or no-longer need.
There are many project governance frameworks being used in industry, with most of them seemingly advocating a waterfall approach where the early phases (or service gates) are go/no-go checkpoints when it’s possible to stop a project if problems are discovered. Of course, stopping a doomed project early is good as it saves money. However, the early phases of a waterfall project are usually more paper-based exercises, and not until the latter phases are show-stopping problems discovered, by which point it’s very hard to stop a project because of the investment that’s already been made. This means the ability to stop a project before any code is written is rarely more than theory.
However, many of these governance frameworks are only interpreted as being waterfall-aligned, when in reality they are trying to achieve the same objectives as a more iterative or agile approach. This mistake typically occurs because those people asked to "police" the governance rules don't understand software development and interpret the objectives as a list of documents that need to be completed and signed-off. This mistake typically forces even well-meaning projects down a high-bureaucracy waterfall approach.
A common misconception when an organisation tries to maximise efficiency and minimise waste is that turning software delivery into the equivalent of a manufacturing environment will improve the situation. e.g. They see it as analogous to a car plant: Start with a load of bits, assemble components, assemble bodywork, paint body, assemble car, test car, finished. However, the mistake is that they see this process as a series of functional steps, a waterfall, forgetting that iterations are time markers. So if we revisit the production line at snapshots in time, one car is having its components assembled, another is having its bodywork painted, another is being tested, etc. At the end of an iteration, a car rolls off the production line and the start is fed with a load of bits. i.e. the production line doesn’t build all the engines at once, paint all the bodywork, test all the cars at the same time...
For software delivery too, the deliverable at the end of each iteration needs to be incrementally different. We are not typically building what has been built before. It's unique and has to be to justify the investment for the stakeholders. So the activity of a software project has to been seen as more akin to research and development. This takes time, experimentation and repeated iterations to achieve progress over time. A waterfall process assumes everything is known up-front.
Another unfortunate side-effect of the "industrialisation" mindset is the creation of capability silos. This is where people with similar skill-sets are grouped together to perform specfic roles for project teams. However, this leads to a number of disadvantages for software delivery. The people in each silo become highly specialised in their skills and experiences, unable to perform the generalising specialist role that is capable of stepping to help others whilst knowing how to support the overall team to their best ability. It can also lead to the physical separation of roles within the project team, resulting in formal document handovers, a non-collaborative work environment, and the tendency to "throw work over the wall" to the next role in the process chain. A perfect breeding ground for a waterfall mindset.
It's not just the organisation that sows waterfall seeds into the fertile minds of software practitioners. Many tools vendors and their training courses, when providing a context to their solutions, often do so by referring to the value added during the "testing phase" for example. They may do so to highlight to the value of their tools to a certain aspect of delivery, but this can result in a mindset that considers the tools to be the solution for a poorly performing project team. Test tools are a classic example, focussing the organisation or project teams on the reactive use of testing tools to identify defects just prior to deployment, as opposed to encouraging project teams to use iterative and agile practices to discover mistakes early and have them fixed at a time when it costs the least.
A focus on power tools supporting only one activity or role, as opposed to collaborative tools encouraging a whole team approach, can certainly cause problems.
Fear of mistakes
And finally, what's the biggest reason why the waterfall comfort blanket is so common-place? I believe it is fear. Fear of making mistakes, of not having thought of everything before trying to share that knowledge with others, of having to change their minds.
Too often team members are measured by the "completion" of documentation, encouraged to "gold-plate" their work product or knowledge prior to sharing it or having it reviewed. They may achieve beautiful examples of the perfect document or code, but they passed the point of diminishing returns for their efforts far earlier on, a point at which most of the consumers of that information would have had either enough to help them or the ability to provide valuable feedback to the author. This is a principle that applies to every member of team, but when measured by "completion", waterfall remains in charge.
So, in summary, whilst it is frustrating to see so many people still clinging tightly to their waterfall comfort blanket, it has to be expected. Many of the messages these people are given, together with the measures of "success" placed upon them, encourage a waterfall mentality.
As an industry, we have a long way to go before we see a thorough decline in waterfall activity. However, if we continue to tackle all of the above challenges, it can be achieved.