![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
Software projects in general have an unimpressive track record when compared to traditional engineering projects.
1
There are a vast number of reports documenting software projects that were seriously over budget, over schedule, or even cancelled. Even those projects that eventually manage to get a product out of the door will in most cases have to spend considerable time doing post release fixes and corrections.
This paper examines some of the key characteristics that make software development difficult and error prone. I will explore this topic primarily by contrasting software construction with construction of traditional concrete, physical engineering systems, such as buildings, bridges, ships, and aircraft. I would like to add that the difficulties I describe here are those specifically addressed by the IBM® Rational® approach to iterative development, which you can read about elsewhere in this issue and in the archives of The Rational Edge ezine. But even if you have enjoyed success through the application of iterative development techniques in your software projects, I believe it is useful to explore the various reasons those techniques are so critical.
The failure of a software project becomes more likely as the size of a project increases. The larger the project, the more likely is it to become late, over budget, or even get cancelled. Fundamentally, the initial cost and schedule estimates are frequently way off. This has led some people to believe that the root cause for failures in the software industry is inadequate management practices for large-scale software development projects. However, even though many of today's software projects are quite large in terms of budget, duration, number of people involved, etc. most of them are in fact relatively small when compared to traditional engineering projects, such as bridge building or dam construction.
Given the far-better track record in terms of staying on plan and on budget for traditional engineering projects -- even those that are orders of magnitude larger than the largest software projects -- it is tempting to conclude that the problems of the software industry could be mitigated by applying identical project management techniques and processes as those used in traditional engineering. The problem is, the similarities that seem apparent on the surface might only go skin deep. There are key differences that make software projects much more difficult to plan, estimate, and complete according to plan than traditional engineering projects.
Major problems in software projects
Commercial software projects 2 are frequently over budget and late, often in the extreme: Budget and schedule overruns by a factor of two are not unheard of. Many projects seriously in trouble are also cancelled at a fairly late stage, thus having incurred significant economic loss as well as lost opportunity costs. Once they are made generally available (GA), software systems often fail to perform according to specification. Typical shortcomings include missing functionality, poor runtime performance, and bugs. For most software systems or applications, it will take quite some time, and lots of frustration and effort, from initial GA until the application eventually stabilizes, hopefully by then behaving approximately as specified. The typical variance of software projects, in terms of plans versus actuals, is thus very large.
Contrast this situation with that of traditional engineering projects -- even the very largest of traditional engineering projects, with time spans over many years, multi-billion dollar budgets, and involving thousands of people are commonly completed very close to the initial plan, both in terms of schedule and budget. Thus, the variance for these projects is limited to a fairly small range.
The obvious question becomes: Why are software engineers so incapable of accurate estimations?
A key difference between software projects and traditional engineering projects is the proportion of time and cost spent on the "creative" phase, that is, architecture and design, versus on the production or manufacturing phase. In traditional engineering, the bulk of time as well as of the cost is spent in production, while the architecture and design phase only contributes a minor part of time and cost. For instance, for a housing construction project, or for development of a new model of a car, the "creative" part of the work might involve a limited number of architects and designers, taking only a couple of months, while the actual production might involve thousands of people, or even automations, such as robots, and goes on for many years.
On the other hand, for software projects, the relationship between architecture and design versus production is not only inverted, but in fact approaches infinity: since the production or manufacturing costs for software are almost nil (see below), almost all cost for a software project comes from the workflows dealing with the creative parts -- that is, the invention and design of software.
One can thus define traditional engineering projects as being primarily "production projects," while software projects are fundamentally "design projects."
Characteristics of production projects
Typical production projects are characterized by being quite standardized in terms of the tasks, processes, and methods involved. In addition, production projects typically operate on standardized building blocks, such as the prefabricated construction elements of a building. Production projects are well suited for mechanization and automation, and they can be predictably managed, traditionally by Tayloristic approaches of Scientific Management, where each job, task, and step can be described in detail and accurately measured and optimized for time and cost. These types of projects, with well-defined mechanistic processes and tasks, are also easy to optimize for cost -- e.g. by relocating or outsourcing to a low-cost provider.
Characteristics of design projects
Design projects are completely different. The jobs and tasks involved in design projects are far from mechanical, and do not easily lend themselves to Tayloristic or other types of a priori, fully predictive approaches. Instead, design projects demand long periods of discovery and invention, involving a very skilled staff capable of performing qualified tasks. Developers must possess deep and broad knowledge of both the problem and solution domains, they must be capable of intellectual work, and they must be able to perform well in team communication and collaboration. Design projects also demand a different approach to planning and management.
The "factory approach" to software development is bound to fail
Many large software development organizations fail to see or acknowledge the difference between design and production projects, and attempt to use business and management approaches that might be well suited for production and manufacturing projects, but which will not work well for software projects, which are non-mechanical, far from trivial, and fully dependent on the quality of the people involved.
Compared to production projects, design projects demand a fundamentally different approach. Costs can ultimately not be kept down by hiring low-pay "human automations," capable only of well defined mechanical tasks. Time-to-market cannot be improved by adding more of these "automations" to the project, or by asking them to work "faster." Most of the tasks involved are simply too complex for lending themselves to mechanization. For design projects to be successful, all staff involved must be very knowledgeable, skilled, and experienced in a broad range of software engineering topics, ranging from understanding the problem domain and customer needs, all the way down to the specifics of the implementation platform. Furthermore, they also need to be motivated.
The common attempts to use business and management practices from traditional production projects on software projects -- focusing on cost reductions and standard business optimizations of workflows which are far from mechanical, quantifiable, or trivial -- without understanding the complexities involved in a typical design project, has led many software development organizations to invest way too little on the key analytical and creative workflows of their projects. For instance, methodical use of prototyping, modeling, or simulation is rare. This often results in architectures and designs being less than adequate, making implementation troublesome, and more expensive than planned, because a large number of "fixes" are needed at a later stage. This typical pattern of "quick and dirty" analysis and design, followed by "jump to implementation," accompanied by "trial and error" based quality control would be totally unthinkable in traditional engineering projects, where the cost of not getting the architecture and design correct in the first place is totally prohibitive. Few organizations involved in traditional engineering can cope with vast numbers of their products being recalled due to malfunctions. Unfortunately, this pattern is very common in software development: "We'll fix the problem in a later release."
Too little emphasis has been placed on the quality of staff responsible for the truly innovative, creative, and hard aspects of software development, which unfortunately comprise most of the workflows of a typical software development project. Even though some software projects, particularly those in maintenance phase, can be successfully done by a team of "average Joe's," any new development efforts are likely to be dependent on very skilled staff for their success.
The planning of design projects needs to be adaptive in order to cope with the many unknowns and changes. It is impossible, particularly in the early phases, to accurately estimate the effort involved in developing any non-trivial piece of software.
"Supporting" versus "Doing" disciplines
For the rest of this discussion, it will be useful to distinguish between two categories of activities involved in any development or construction project: "supporting" and "doing." The former deals with traditional project management type of activities, such as planning, scheduling, staffing, estimation, monitoring and reporting, while the latter includes all those activities that deal with the actual construction of the system, that is, the various "crafts" involved in designing and building something, such as analysis, design, construction, and implementation.
On the surface, it might appear that a software project shares the same supporting activities and workflows as a traditional engineering project. Activities such as requirements gathering, project organization and planning, staffing, estimation, monitoring, and reporting all occur in both types of projects. There are numerous methods and tools available for supporting these activities, and traditional engineering projects have a proven track record in completing their work more or less as planned.
The software industry has also focused on the supporting disciplines for quite some time. There is a plethora of processes, methods, and tools specifically targeted towards the software industry, and most organizations have made significant investments in shaping up their supporting disciplines.
Still, few software projects perform as planned; initial as well as later estimates tend to be way off, demanding frequent re-planning, which unfortunately also tends to be overly optimistic.
What is it then that makes planning of software projects so hard?
Abstract versus concrete things
Obviously, it is among the doing disciplines that we find the most obvious differences between software and traditional engineering. Even though there are some similarities -- particularly in the architecture workflow, where architects in software as well as in traditional engineering typically use models -- the remaining activities are very different: While engineers in traditional disciplines often operate on standardized physical entities, software engineers operate exclusively on intangible, abstract things. Instead of welding pieces of metal, software engineers join stuff by writing code. Instead of building physical models to scale, or using mathematical models, software designers typically "model" by writing high level code. Instead of placing building blocks under measured forces to test their structural integrity, software engineers write code for their testing. Regardless of workflow, software engineers are all the time working with non-tangible, abstract, and flexible things -- pure mindware.
Method of construction
Software systems differ from traditional systems in their methods of construction or implementation. Physical systems are to a large extent constructed by a process of assembly of standardized components. Software systems, on the other hand, are fundamentally built by a process of discovery, invention, and creation of new components. Even though the software industry has been pursuing reuse, standardization of interfaces, and componentization, the success has so far been limited.
A physical system -- a building, a bridge, a car, an aircraft, a ship, or a computer -- exists in the physical world. Besides occupying 3-dimensional space, their "atomic structures" -- that is, the individual components or parts of these types of systems -- are constructed from other physical entities. Regardless of whether we consider the physical systems themselves or their individual constituent parts, they share some common physical properties, such as weight, length, width, height, etc. They also have in common their tangibility: We can see, touch, feel, and move them around, and we can also easily play and experiment with them, using whatever means the physical world and its laws makes available for us.
We can also create scaled models of these physical systems, and subject these models to experiments and tests governed by the same physical laws that the real, full-scale systems will be governed by, thereby gaining valuable information aiding us in building the real thing. This is partly because the laws governing physical systems, as well as the systems themselves, also primarily exhibit continuous characteristics. This makes the behavior of these systems reasonably predictable, quantifiable, and measurable to a much higher degree than for software systems, which are fundamentally discrete, where a small change to an input can result in a hugely disproportionate change to the outputs.
Humans are quite familiar with physical systems -- there's no need to be an engineer in order to understand the practicalities of physical systems and laws. From the day we're born, we have personal experience in coping with the physical world. We learn first-hand about the principles of gravity the first time we fall down, and we experience many other physical laws early on in our lives.
Software, on the other hand, is not a physical, tangible thing. Instead, for all practical purposes, software is a purely abstract, conceptual thing, existing only in the minds of humans. Sure, we can place the atomic structure of software -- its individual bits -- onto media such as a disc drive or CD, thereby creating a physical representation of it, but a software system itself is purely conceptual and intangible. As such, it's not subject to any of the stringent laws governing physical systems. This lack of laws or first principles makes construction of software systems as much an art as a science. For software development there are few if any useful analytical methods available. Instead, most of software development is fundamentally based on synthetic approaches, where experience, skill and use of "best practices" in chosen methods and processes become the only available determinants for success or failure.
In addition, being intangible greatly limits our possibilities to "play and experiment" with our software system -- we cannot touch it, feel it, nor move it around -- all we can do is to think about it.
Lack of first principles (laws)
In software, there are no first principles or laws governing the creation and assembly of software. In contrast, traditional engineering, for instance building construction, is constrained by well defined and well known physical laws. This lack of laws makes software the ultimate in flexibility, that is, there are very few constraints on what you can do with software, which is a good thing. Unfortunately, the almost infinite degrees of freedom resulting from the lack of laws, and the inherent flexibility also means that software professionals have very little assistance in terms of a priori constraints on how to do their work. A construction engineer is fully aware of the laws that the physical world places on his designs, but the situation for the software engineer is very different: he or she operates in a world which is essentially free of any a priori limitations -- the only constraints for the designs in our world are those imposed by ourselves and the methodologies, processes, and tools we use.
In a sense, creating software is far more closely related to writing a novel than to any traditional engineering effort -- an author is free to construct whatever type of story, limited only by his imagination and whatever facts may be necessary to the narrative. Furthermore, the author must construct the architecture of the story in his head, and to make sure the design and implementation of the story is consistent with the overall architecture. The quality of the author's product is to a high degree based on whether he succeeds in keeping the details of the story consistent with the architecture.
The abstract system that an author constructs exists only in his mind, and the physical manifestation -- the book -- is only a convenient means of storage and distribution. This is very much the same situation as for software.
Traditional engineering systems have a significant marginal cost. For instance, having just successfully built a bridge, building an identical one will cost almost as much as the first. This is very different for software, where the marginal cost for creating a copy of an existing system is almost nil. The fact that marginal cost for software is almost nonexistent establishes two other key differences between traditional engineering and software:
- Many software systems solve unique problems never tackled before.
- There is a strong tendency to extend existing software systems beyond their capabilities.
Actually, the bullets above are two sides of the same coin: because the marginal cost of software is nil, it's very tempting for organizations developing software to solve a new problem using a legacy system as a foundation -- that is, extending and modifying an already existing system to cope with the new problems. Within reason, this is a sound and effective practice, but when taken too far, this approach can eventually lead to a practice of patching and hacking, quickly destroying the intended architecture, making further extension and maintenance impossible.
Secondly, because there is a strong temptation to extend legacy code to deal with the creation of similar but extended systems, the remaining software systems actually created from scratch are necessarily going to be first-ever systems dealing with previously untouched problems, demanding the organization to stray into the unknown, by stretching existing technology, as well as methods and tools, to their limits. These first-ever projects have many similarities with pure research projects, where it is very difficult to estimate and predict the outcome. This tendency for software projects to constantly work on the edges of technology is caused by the seemingly never ending steady increase of the capabilities of modern digital technology, which tends to push the envelope for what organizations will attempt to do with software. Furthermore, another consequence of this law of Moore's is that our foundation, that is, the platforms and fundamental technologies we use on top of the platform, are never stable -- they are constantly changing, demanding the same rate of change in our software development projects. Therefore, software development teams must learn to live in, and cope with a very dynamic world where everything constantly changes 5 .
Flexibility -- a blessing or a curse?
The inherent flexibility of software in combination with the ever increasing power of digital technology allows us to continue solving unprecedented problems each day using software. Because of its flexibility, we can mold software into any form to do almost anything. Flexibility thus expands the "what" we can use software for almost indefinitely. Unfortunately, flexibility also is one of the chief culprits of the problems for software projects, because it applies equally well to the "how" of software engineering. That is, software can not only be molded into almost any form, but we can also perform the molding process itself in almost any way we like. Because there are so few constraints governing software, the number of possible ways to do something is almost unlimited. Therefore, software development organizations have almost infinite possibilities for choosing methods, tools and processes to use, and often there exist many different ways of working within the same organization.
As an analogy, contrast building something using Lego, versus building something using Play-Doh. Even though we can combine the individual bricks of Lego in almost infinite number of ways producing a large variety of "systems," we are still constrained under the very strict "laws" provided by the Lego framework and its inherent principles on how Lego blocks can be put together. The same situation exists for physical systems. Play-Doh, on the other hand, is very similar to software -- not only can it be used to construct almost any structure, but it also allows a constructor to do it in almost any way he pleases, there are very few constraints on the "how's" of Play-Doh.
Requirements for prediction and estimation
In order to accurately predict and estimate a project in terms of duration, effort, etc., all the various tasks and steps involved must not only be identified and scheduled, they also need to be accurately quantifiable in several dimensions, such as duration, difficulty, resource consumption, and dependencies. Basically, accurate prediction and estimation requires the existence of an algorithm, or a step-by-step problem solving procedure with a finite number of well defined steps. For traditional engineering, where there typically exists vast pools of experience building similar or identical systems, under the familiar physical laws, the typical tasks and steps are well known, and estimation can be fairly accurately done. However, for software projects, particularly the first-ever projects, such data simply does not exist. Nor is there a vast pool of applicable analytical or quantitative methods we could use.
Software also lacks a well defined atomic unit of work -- even though we can apply various measures, such as lines-of-code, number of classes, number of function points, number of bugs, etc. The problem is, these measures are neither atomic nor comparable in terms of effort. That is, a sequence A with 10 lines of code might have a very different degree of complexity and difficulty compared to sequence B of 10 lines of code written in the same language. Similarly with bugs -- some bugs are orders of magnitude more tricky to identify and correct than others, but we really have no way to consistently identify ahead of time which ones will be the tricky ones.
Estimation and prediction of software projects becomes guesswork, where only conservative assessment and previous experience can reduce the variables in play. This is very similar to pure research projects, where many different paths towards a solution are tried and abandoned before a successful path can be found.
It has also been proven that the productivity of individual software developers has very large variance. Productivity between individuals in the same development team can differ by orders of magnitude.
Furthermore, because of software's inherent flexibility, we can build a system in almost any way we like, using whatever means we like, further exacerbating the problems of accurate estimation.
All these factors combined contribute to making estimation and prediction less than accurate. In essence, fully accurate prediction and estimation of projects or tasks demand a Tayloristic approach of Scientific Management, which only is applicable to trivial, mechanical tasks. Typical software projects are, for reasons discussed above, far from being trivial or mechanical.
Software is hard for a number of fundamental reasons:
- Software projects are design projects, not production projects.
- Because software is pure mindware, software development is more of an art or craft than a science or engineering discipline, making the "doing" disciplines of software development difficult to perform, monitor, and measure. Productivity differences between individuals are huge.
- Software lacks laws and first principles.
- Software lacks a measurable standard unit of work.
- Given software's flexibility, the "supporting" disciplines of software projects -- planning, estimation, prediction and observation -- are quite difficult.
- Moore's law makes the foundations of software ever changing -- there's no "permanent" platform to build on.
Even though it might be tempting to consider software engineering as "just another engineering discipline," and thus jump to the conclusion that software can be developed using the same business, management, and engineering principles as traditional engineering, there are some unique characteristics that make software and software development not only special, but also more difficult to estimate and monitor than traditional engineering projects.
Software development, with its inherent characteristics, existing in a very dynamic world governed only by Moore's law, will remain more art than a science for a long time to come. We can only hope to tackle the problems in software development by acknowledging that software is different, and understanding the consequences of these differences.
Any attempts to regard software development as just another form of traditional engineering will generally not deliver the desired results. Only by having a good understanding of what makes software different, can we start addressing the unique issues associated with software development. Iterative development, modeling, requirements management, and early-and-often stakeholder involvement are among the techniques available to software teams who recognize this critical difference and are looking for a new way forward.
1 Examples of successful engineering projects, at least with respect to schedule, are all the construction projects for Olympic Games -- as far as I know, all Olympic Games have started on time, in fundamentally new premises. Another successful example, the Sears Tower: built 72-74, 160M USD, 60 subcontractors, 2400 workers at peak. More info, see http://skyscraperpage.com/cities/?buildingID=5
2 Interestingly, there's some evidence indicating that non-commercial software projects, even very large scale ones, such as the Eclipse Open Source project, or the GNU/Linux projects, have a much better track records than fully commercial ones.
3 A car factory assembly line is a good example. Most of the tasks are now performed not by humans but by robots.
4 http://en.wikipedia.org/wiki/Scientific_management
5 Panta Rei
- Participate in the discussion forum.
- A new forum has been created specifically for Rational Edge articles, so now you can share your thoughts about this or other articles in the current issue or our archives. Read what your colleagues the world over have to say, generate your own discussion, or join discussions in progress. Begin by clicking HERE.
-
Global Rational User Group Community
Tommy Lennhamn spent his formative years as a systems programmer with Philips Elektronikindustrier, then the leading Swedish Aerospace and Defense contractor. He joined Rational Scandinavia AB in 1991, as a Technical Consultant, and has exclusively focused on embedded, real-time, and event-driven systems, ranging from small, microcontroller-based gizmos to huge, multi-million line distributed systems with several hundred developers and hundreds of processing units. He is currently a member of IBM Rational's Ericsson Corporate Account Strategy Team, working very closely with Ericsson R and D and working as a liaison officer between Ericsson Software Research and IBM Research, looking into future directions of systems and software development.
Table of contents
Local resources
Tags
Use the slider bar to see more or fewer tags.
For articles in technology zones (such as Java technology, Linux, Open source, XML), Popular tags shows the top tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), Popular tags shows the top tags for just that product zone.
For articles in technology zones (such as Java technology, Linux, Open source, XML), My tags shows your tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), My tags shows your tags for just that product zone.
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
