This is my first developerWorks blog posting. It is something I wrote a while ago, but I had some very positive feedback on this whilst at Innovate 2011, so I thought it worthwhile sharing again.
From my experiences of working with many software project teams over the years, I have regularly been frustrated with the instinctive response of the project to revert to a waterfall mindset at the first sign of project difficulty. The moment anything "scares" them, like a risk, technical complexity, a large project team, long delivery times-scales, or even a new technique or technology, they go running for cover and hold on tightly to the one thing that gives them a feeling of security, their "waterfall comfort blanket".
So, where did this phenomenon come from, why is this such an instinctive reaction, and does it help them?
Waterfall, or the linear software development lifecycle, has been present in the industry for a long time despite software leaders in the past trying many times to discredit the approach, it still remains commonplace for software delivery projects, even if they say they are doing otherwise. The use of iterative and agile practices, which are widely recognised as superior to a waterfall approach, are rarely adopted well. There are a number of reasons for this, so here I will explore a few key examples.
Unfortunately, the world of academia, the first place so many of our software colleagues learn about how to organise and manage a software project, is still very slow to catch up with iterative and agile practices. So many graduates are arriving in the workplace having been primarily educated using the waterfall model as their primary lifecycle, with only side reference made to these more modern approaches, if they are lucky.
This may be because it is easier to learn and to teach waterfall as opposed to iterative practices. Waterfall can be learnt as a step by step set of instructions, which is how most things are taught. Iterative is much harder to explain if you’re used to explaining or follow a step by step process. However, all hope should not be lost, as some academic subjects do naturally use iterative principles; art for example, setting a context and prototyping with pencil sketches, prior to investing cost and effort with canvas, paints and hours of detail.
I have often heard it said that an iterative or agile approach isn't feasible because of the constraints imposed upon a project by delivery contracts. It's true that this is a common issue for software delivery organisations, but these constraints have typically come about from a breakdown of trust between the delivery team and its customer, typically resulting in a scenario such as:
The customer has little confidence in the delivery team and demands that they provide a fixed-price for the delivery of the solution.
In reaction to this fixed-price constraint, the delivery team requests that the customer defines everything they need in detail before a price will be given.
The customer, knowing that this is their one chance to specify what they want or risk severe change control later on, provides the details of everything they might ever want.
The delivery team ties the customer to this specification, unless they pay extra to change it, resulting in a solution that provides functions that are either never used or no-longer need.
This classic waterfall scenario is typically driven by an old-school procurement mindset that wants to know exactly what they are going to get for their money, even if the business can't say what it is that they will actually need.
There are many project governance frameworks being used in industry, with most of them seemingly advocating a waterfall approach where the early phases (or service gates) are go/no-go checkpoints when it’s possible to stop a project if problems are discovered. Of course, stopping a doomed project early is good as it saves money. However, the early phases of a waterfall project are usually more paper-based exercises, and not until the latter phases are show-stopping problems discovered, by which point it’s very hard to stop a project because of the investment that’s already been made. This means the ability to stop a project before any code is written is rarely more than theory.
However, many of these governance frameworks are only interpreted as being waterfall-aligned, when in reality they are trying to achieve the same objectives as a more iterative or agile approach. This mistake typically occurs because those people asked to "police" the governance rules don't understand software development and interpret the objectives as a list of documents that need to be completed and signed-off. This mistake typically forces even well-meaning projects down a high-bureaucracy waterfall approach.
A common misconception when an organisation tries to maximise efficiency and minimise waste is that turning software delivery into the equivalent of a manufacturing environment will improve the situation. e.g. They see it as analogous to a car plant: Start with a load of bits, assemble components, assemble bodywork, paint body, assemble car, test car, finished. However, the mistake is that they see this process as a series of functional steps, a waterfall, forgetting that iterations are time markers. So if we revisit the production line at snapshots in time, one car is having its components assembled, another is having its bodywork painted, another is being tested, etc. At the end of an iteration, a car rolls off the production line and the start is fed with a load of bits. i.e. the production line doesn’t build all the engines at once, paint all the bodywork, test all the cars at the same time...
For software delivery too, the deliverable at the end of each iteration needs to be incrementally different. We are not typically building what has been built before. It's unique and has to be to justify the investment for the stakeholders. So the activity of a software project has to been seen as more akin to research and development. This takes time, experimentation and repeated iterations to achieve progress over time. A waterfall process assumes everything is known up-front.
Another unfortunate side-effect of the "industrialisation" mindset is the creation of capability silos. This is where people with similar skill-sets are grouped together to perform specfic roles for project teams. However, this leads to a number of disadvantages for software delivery. The people in each silo become highly specialised in their skills and experiences, unable to perform the generalising specialist role that is capable of stepping to help others whilst knowing how to support the overall team to their best ability. It can also lead to the physical separation of roles within the project team, resulting in formal document handovers, a non-collaborative work environment, and the tendency to "throw work over the wall" to the next role in the process chain. A perfect breeding ground for a waterfall mindset.
It's not just the organisation that sows waterfall seeds into the fertile minds of software practitioners. Many tools vendors and their training courses, when providing a context to their solutions, often do so by referring to the value added during the "testing phase" for example. They may do so to highlight to the value of their tools to a certain aspect of delivery, but this can result in a mindset that considers the tools to be the solution for a poorly performing project team. Test tools are a classic example, focussing the organisation or project teams on the reactive use of testing tools to identify defects just prior to deployment, as opposed to encouraging project teams to use iterative and agile practices to discover mistakes early and have them fixed at a time when it costs the least.
A focus on power tools supporting only one activity or role, as opposed to collaborative tools encouraging a whole team approach, can certainly cause problems.
Fear of mistakes
And finally, what's the biggest reason why the waterfall comfort blanket is so common-place? I believe it is fear. Fear of making mistakes, of not having thought of everything before trying to share that knowledge with others, of having to change their minds.
Too often team members are measured by the "completion" of documentation, encouraged to "gold-plate" their work product or knowledge prior to sharing it or having it reviewed. They may achieve beautiful examples of the perfect document or code, but they passed the point of diminishing returns for their efforts far earlier on, a point at which most of the consumers of that information would have had either enough to help them or the ability to provide valuable feedback to the author. This is a principle that applies to every member of team, but when measured by "completion", waterfall remains in charge.
So, in summary, whilst it is frustrating to see so many people still clinging tightly to their waterfall comfort blanket, it has to be expected. Many of the messages these people are given, together with the measures of "success" placed upon them, encourage a waterfall mentality.
As an industry, we have a long way to go before we see a thorough decline in waterfall activity. However, if we continue to tackle all of the above challenges, it can be achieved.
As the value of Agile practices becomes more understood, brave project managers are beginning to challenge the normal practices of the organisation and request the opportunity to take a more Agile approach.
In our animated movie "I want to run an Agile project" (by Carson Holmes and I), we follow the experiences of one such brave project manager, Luke, and his many different encounters throughout the enterprise, as he works to establish and deliver his Agile project.
After watching his sadly-entertaining journey, in this article we explain what is really going on, and start to highlight the reasons behind his troubles, and how an organisation can work to overcome them.
Scene 1 - The Stakeholder Establishing that there is a need for the business to invest in a project is just the start of working with the stakeholder. They may think they know what they need, they probably think they know what the solution looks like, but whatever they think they know, they have to work with the project team to make the project a success. This is where so many projects come unstuck. They assume they can define and clearly communicate their needs to the project team via a document of their perceived notions of what the system shall do: "a bucket of shalls"! This rarely succeeds. However establishing a Vision and a close working relationship with the stakeholder and their business users will allow the project to start quickly, collaborate with the stakeholders to identify critical needs, and work to quickly deliver a rapid return on investment for the business. Without this collaboration, wants will quickly turn to assumptions, spec's into speculations, and a high probability that any invested effort will not deliver what the stakeholder really needed.
Scene 2 - Procurement Establishing a business need and having stakeholders willing to commit to the project is a good start, but this typically leads to the need to complete funding procedures, whether they be external procurement teams, on internal programme offices. Those teams want to know what it is they are funding and what they will get for their money. Unfortunately, they typically operate to simplistic assumptions that the business knows the details of what they really need up-front, and that business needs will not change during the life of the project. Changing their mindset to one of committing only small investments and observing ROI prior to investing further sounds easy. But, when organisations have never had prior opportunity for early ROI before, they expect every project to be protracted and deliver disappointing results late. The procurement people need to be brave and ask some tough questions of the delivery projects: What if we have to cut funding half-way through the project? Can we look to establish early business revenue help fund the remainder of the project? Can we incrementally fund your project based upon demonstrated results? Our Agile project manager would be happy to answer these questions.
Scene 3 - Governance and Compliance There are good reasons why organisations set up these governance and compliance teams. Industry regulatory compliance rules need to be adhered to and organisations have been "burnt" so many times by failing, typically waterfall, projects that safety nets were required to protect the business from rogue and dangerous delivery projects. However, these governance rules can also prevent successful project practices from being used and enforce some of the practices that the governance rules are trying to avoid! Incremental improvements in delivery success can be achieved by enforcing draconian measures on projects, but to make distinct step-changes in delivery success requires a more significant change: incremental and agile delivery. Don't get us wrong, there is nothing wrong with asking a project questions such as: "Can you prove you understand our needs?", "Can you demonstrate you have a sound solution that will meet those needs?", "Do you understand the risks involved, and can you demonstrate how you will overcome them?", "Can you demonstrate that your solution meets the expectations of the stakeholders?", etc. However, most typically the governance team have asked for documentation to support the answers to these questions as opposed to concrete evidence that the team is actively delivering results. Getting back to the objective of any "control point" will typically allow an agile team to establish what measures of valuable progress they are looking for, and provide better measures of real progress than the typical documentary evidence has ever provided.
Scene 4 - Enterprise Architecture There is a lot of value that any project can gain from working with Enterprise Architecture (EA): understanding the strategic technologies of the organisation; establishing the non-functional and technical requirements of the project; aligning to the other systems of the enterprise; providing feedback on the technical vision of the project; to name just a few. However, EA should not be looking to define the detail of the solution that the team needs to determine and deliver. They should be like the stakeholder; helping to define needs, validating business value, and collaborating regularly with the team. In this way, they provide a valuable source of strategic technical governance and alignment to business strategy. Delivering a big design up-front only leads to speculation and proof by documentation that a technical solution will work. It's much better to demonstrate an executable architecture, and the EA teams will agree when they start to realise that it's possible to work that way.
Scene 5 - Development Team Not everything on the project will be technically easy. Many of the things asked of the project team will be new to them. The developers will have different experiences, and the best way to overcome the challenges for the team will be to encourage them to collaborate. Unfortunately many members of development teams have had careers where they have been encouraged to act as heroes. They have been measured by their individual performance and not by the success of their project team. When the regular delivery of demonstrable success becomes important, the team needs to recognise this and whenever technical challenges arise, work collaboratively to find a solution. This will be both more efficient and help build a sense of team. Pairing is one approach to achieving this, but it's not required all the time, only when anyone on the team finds a challenge and asks for help. Then a member of the team will volunteer their help, for as long as the challenge still remains.
Scene 6 - Deployment It's no wonder that deployment teams view development teams with caution. So often they have been on the receiving end of executable solutions that have been rushed into deployment, poorly tested, and not designed to support a production environment effectively. However, if treated as another stakeholder of the project, their needs can easily be catered for, and their fears allayed. They are also expected to protect the business, and when projects have rarely delivered to expectations, they are very wary of new solutions that are regularly delivered and expect regular deployment. Engaging the deployment team regularly in the project, allowing them to see successful pre-deployment testing, and demonstrating a successful deployment plan, will help the team to gain the confidence to schedule the Agile team’s regular deployments.
Scene 7 - Stakeholder Acceptance By the time the Agile project team is ready to deploy a solution that adds value to the business, there should be no surprises for the business about what it is going to get. Their continued involvement as members of the project team or through attending regular demonstrations, should provide them with ample opportunity to ensure that the project is delivering what they need, even if their need change, or they were unsure what they wanted until they saw it for the first time. However, even in the worst-case scenario of the stakeholders only seeing the solution at thei first point where a basic solution could be deployed, they are still able to change the course of the project far earlier than would be possible in a more traditional lifecycle.
Epilogue Our brave Project Manager, Luke, did manage to force his Agile project through the organisation, but whilst it was a tough journey, it was worth it. The customer did receive early value and Luke established precedents with many of the other organisational functions. Over time he will find that the rest of the organisation starts to recognise the value of the Agile delivery approach and barriers will removed or refined to accommodate the delivery of early value. However, this process will take time, and it will take more than just Luke’s endeavours to make that change.
One of the great privileges of my working life is to have the chance to share knowledge and encourage learning. I have been writing, deploying and teaching many software delivery courses over the last decade and most of the experiences have been positive for both the participants and myself, or so they tell me...
However, there have been instances when people just don't seem to want to engage or can't quite understand the significance of what we are discussing. When I dig beneath the surface of their previous experiences and why they feel they are attending the course, I typically discover that there is a common reason for their disengagement; they simply aren't ready to learn.
For an individual to want to make changes in how they perform, take on challenges or operate their daily activities there needs to be a compelling reason for them to learn how to make the change and to then apply that learning. Traditional thinking tends to be that we provide people with ‘just in time’ education so that they are prepared with the new skills required of the change. However, this tends to fail in ensuring that the skills are learned in a way that allows them to be applied effectively when the time for change arises. I much prefer to find ways of applying a ‘just too late’ learning approach. This can be achieved in a number of ways and here I will share with you just a few.....
New opportunities - "Things are changing, you have the opportunity to be part of it and reap the benefits."
This could be seen as the leader's strategic motivational speech but it doesn't have to be like that. Being honest within the organisation about what is coming in the industry, how this will mean change, but also new opportunities for people who want to come on the journey, is a great way to help people discover the need to learn. Of course they also need to be given the chance to participate, but those who want to be part of it will really understand why they need to participate in learning too.
Known unknowns - "Find out what you can, answer these questions, tell me what you discovered and question what you don't understand."
The above approach to encouraging learning is not for everyone. Instead, some organisations realise a need to develop new skills and their employees find that they are sent on training courses without really knowing why they are there. When that happens, it's mainly to the extent that they aren't aware of what it is they don't know, or how it could add value to them. In this and many other scenarios a ‘research first’ approach to education works well. Don't make the class sit through your interpretation of what they need to know, let them discover this for themselves. Give them a topic, ask them to find out what it means, why they should care, and have some questions to ask if they don't quite get it. It makes for a great discussion, with experiences shared and all the learning is driven by the participants.
New challenges - "Welcome to the project, this is what we are expected to deliver and this is how we will deliver it. Any questions?"
As project teams form they can often feel a sense of not knowing how to get started. Sure, someone tells them that they will follow a certain approach using specific techniques, but if that's even a little new to them it will be tough to get started with confidence. However, if the project can start with a whole team approach to learning using the project as the example to work on, people will see the value of the activity and be left with some progress on the project and a way of working to continue.
Retrospectives - "So, we did well in the last month, but could we have done better?"
This is the most common ‘just too late’ learning moment for iterative and agile teams, but what is most important is that the lessons are also applied to the future of the project. These actions may be a change of working practices for the project which may require additional education, but in this instance there will be a strong appetite to learn from the team who now realise where they need improve.
In each of the four examples above we are building an awareness of the need to learn. However, to establish the use of the techniques may require an organisational culture change which is rarely easy to achieve. This is because even though an individual chooses to learn because they know they need to change, the organisation around them may not have learned the same lessons and may not be ready or willing to change. To enable this ‘just too late’ learning at an organisational level requires a regular measurement, and a continuous improvement mindset. But that’s another story…
Software development project failures can prove very expensive and operationally detrimental for any organisation, often leaving the finance director at a loss as to why the project has failed to deliver its key business objectives. How often do finance directors invest heavily in a software development project, believing they will have the business solution they need, only to be disappointed with the end result?
You only have to look at the recent UK Government reports into their current relationships with IT suppliers, to realise the damaging effect of ploughing money into an IT project which fails to deliver the required business value. Of course, not all software projects result in expensive lawsuits or public outcry, but many can end up causing huge headaches for both IT managers and finance directors which could be prevented by building a greater level of collaboration, the ability to react to business change, and continual demonstrations of “real” progress, to develop a layer of trust between the IT supplier and the business.
This is most typically achieved by contracting with suppliers in a way that both encourages and supports them to deliver using incremental, iterative and agile software development practices.
Failing to build trust with the IT supplier
Building trust between the business and their supplier is vital for any software development project. FDs aren’t expected to have knowledge, expertise or experience in delivering IT projects, and rely on their IT department to manage the relationship to their IT suppliers. However, more often than not the supplier is forced into a fixed-price contract for a project that promises to deliver everything, if not more than, the business needs.
Suppliers also know they only have one shot at obtaining a contract to deliver, and the IT department typically has one opportunity to secure a budget for a system. This typically results in the IT department defining everything that they think a system might ever need to do, asking the supplier for the fixed-price quotation, resulting in a large budget allocated, a contract being secured with one of the cheapest bidders, to deliver a system at some point in the distant future.
As a simple approach it seems to provide a level of control and financial certainty to the business. Unfortunately it typically results in dissatisfaction for the business, a breakdown of trust with the supplier, and a solution that does not reflect the business’s future needs, or worse still, no solution at all.
So how can this happen? This failure tends to stem from a lack of trust, a flawed governance model, a false measure of progress, and a misunderstanding of risk. It’s fair to say that the IT industry does not have the best reputation for successfully delivering projects, whether success is defined as on-time, within budget, or delivering the business value expected of the system. This poor history has led to much of the distrust that exists today.
With such a lack of trust the business adopts a governance approach to oversee the project and keep the supplier honest, typically introducing “gateways” that want to see a measure of progress against governance objectives. Whilst sound in principle, these gateways are often misinterpreted and misused by those performing the governance, leading to measures that adversely encourage poor supplier behaviour, such as a focus on endless documentation, as opposed to demonstrating working solutions. This also results in some very poor risk identification and management, from a mistaken belief that the defined governance approach will ensure success.
Whilst a one-shot procurement model can seem like an attractive option for the FD, providing a defined budget request need with no expected increases, the risks that this can bring to successful delivery are extremely high.
Little consideration is made for how the project will manage the inevitable change of business priorities that the organisation will have during the lifetime of the project. Most likely the strategy of a business and its plans will have moved on and changed during this time, making some parts of any project obsolete.
Plus, by using a fixed price, scope and delivery method, it is impossible for an FD to envisage a return on investment before the end of the project. FDs are often led to think that the project is progressing well from measures of effort against plan. This lack of visible real progress would be highlighted very quickly by asking just one important question – what level of business value return on investment would I get on a project if I had to trim the budget and stop it half-way through?
Relieve pressure on cash flow during project lifecycle
FDs can stop this vicious circle of project failure by encouraging a more progressive funding style relationship with their IT suppliers. Before the project is contractually agreed, FDs should expect to see an agreement from suppliers where they plan to demonstrate a return on investment on a regular basis, or at least for certain stages during the project delivery. If a supplier can show the financial controller of the project some tangible benefits throughout the project, layers of trust gradually build between the two parties and the FD’s confidence in the supplier’s ability will grow as the project develops.
This method also puts FDs in control of the project’s budget and relieves the pressure on cash flow as their finance department shouldn’t need to commit to a large investment, before any of the results have been delivered. Even though there are strong signs that we are coming out of recession, FDs need to remain cautious and ensure that they receive a real value on any investment they make. Taking an agile, iterative approach to their software projects will help them maintain control on their budgets and deliver the objectives set out at the start of the project.
With many organisations looking to improve their IT efficiency and adopt agile project management methods, many will now start to explore this route to greater agility in their software projects – but achieving this goal is not easy.
Quicker and cheaper
Whilst it is an admirable objective to achieve quicker and cheaper results for projects, this should not be the primary objective for greater agility. The key objective should be for IT teams to be more predictable in their delivery, increasing stakeholder confidence and satisfaction, and delivering against business objectives. Most executives would be happy to just get what they are promised, for a change. Driving for cheaper and faster can also result in a reduction of focus on quality, which will of course be costly in the long-term.
Most governance rule sets were designed to provide stakeholders added confidence in the delivery of their solutions by the project teams. However, how they are interpreted has often led to the governance of projects by the production of documentation. Delivering Agile projects in this kind of environment can be very tough, and unless you can return to the principles of the governance rules, being allowed to demonstrate real value as opposed to false effort-spent, your agile project is unlikely to prosper.
The total cost of ownership for any system has a very large proportion being spent during its operational life, so the ability to run and maintain a system efficiently is a very important outcome for a project. However, there are dangers when supposedly-agile projects don’t produce any documentation or include the needs of the operational stakeholders that the system will not be efficient to maintain and run. By failing to consider the needs of operational stakeholders early in the life of a project, a perceived project success may turn out to be a costly long-term failure.
IKIWISI (I know it when I see it)
A common issue for any project team is that what the customer said they wanted is not necessarily what they later want or indeed need. IKIWISI (I Know It When I See It) is a factor that unless addressed and utilised throughout a project will put any project at risk. Agile projects should provide continuous insight into progress and regular demonstrations to the customer. If this visibility is lost or the customer is not engaged with the project to “see it”, then the opportunity to know if the right solution is being developed will have been lost.
To achieve an early ROI for an agile project, the customer needs to be engaged continually in determining the priorities of the project to ensure that this meets the prioritised needs of the business. Without this prioritisation effort, the project will have to start guessing, and make investments to develop solutions that potentially don’t add significant value to the business.
Value versus Risk and Strategy
Whilst agile projects should always remain focussed on the delivery of maximum value to their customers, there has to be a balance maintained against the need for an overall project strategy and the management of project risk. An agile project that maintains a sole focus on value as the customer sees it, may discover that they have overlooked technical risk and a strategic alignment to the customer’s enterprise environment.
There is a growing market for outsourced test services where rigorous testing practices are applied post-development, but whilst it may seem like a cheaper and more rigorous approach to determining quality, the separation of test activity from the development process will have a more costly effect over time. The agile team needs to have a test element to its activity within every iteration of delivery, maintaining quality, and removing the likelihood of major issue discovery late in the project lifecycle.
Command and Control
The legacy of waterfall and a misguided belief in up-front detailed Gantt charts, may cause some in the project organisation to feel they have lost control of an agile project team. This can be particularly true of the project managers (PMs) who have been used to defining every action of the team to deliver a result. There is a place in the agile organisation for the PMs but it’s not telling people how to do their job, it’s about helping remove obstacles and distractions from the team’s path, and ensuring that the customer remains engaged at all times. The old-school PMs will only create inefficiency, frustration and animosity in an agile project.
Skills and Discipline
A long-held myth is that agile projects are full of “hacker” developers, throwing away all processes of the past and exposing the organisation to great risk. However, the reality is that for an agile project to be successful the team does need skills, experience and discipline to do the “right thing” in a responsible manner. Unfortunately, not all organisations have invested in their people in a way that allows them to work in a successfully agile way.
One of the benefits of agile delivery is the ability to react to the changing needs of the customer. However, where this can go wrong is when the project team never has any stability in what it is trying to achieve at any point in time. Within any project iteration, the team must be able to focus on a stable set of requirements that have been previously prioritised by the customer.
If needs do change, then priorities for the next iteration can reflect these changing needs up until the start of the next iteration. Without this discipline of change management the delivery team can suffer from change fatigue and their productivity will certainly suffer.
In essence a “fragile” project environment is typically when an element of what is done is “agile in name only” and activities are performed without discipline and the appropriate level of rigour. However, when applied correctly an agile approach will deliver results and massively reduce the risk of a poor return on investment.