 | Level: Introductory Gary Pollice, Professor of Practice, Worcester Polytechnic Institute
15 Dec 2004 from The Rational Edge: Regular columnist Gary Pollice looks at changes and progress in software engineering over the past two decades. Using instructional books published in 1985 and 2005, respectively, as a guide, he examines changes in each discipline within the software development lifecycle.
The end of the year is a good time for reflection, so this month's column will take a look at the discipline(s) of software engineering to assess how far we've come in the last couple of decades. My starting point is approximately twenty years ago, when I was a part-time student at the Wang Institute of Graduate Studies (WIGS).
WIGS was one of the first institutions of higher learning with a graduate program devoted to software engineering. It was a special place. Although WIGS was not tied directly to the Wang Corporation, it was indirectly tied to the fortunes of Dr. An Wang, who had the vision to found an institution that excelled in teaching the software engineers of the future. His personal fortune and philanthropic efforts were directly affected by Wang Corporation's performance: When the company suffered, so did the Institute. But in that brief, shining moment that spanned 1980 to1987, WIGS had a profound impact on many of us who were fortunate enough to be part of the experience.
The method
To assess how far software engineering has come in the past twenty years, we need specific points of comparison. Let's begin with the body of knowledge presented in college texts. I dug up my copy of Software Engineering Concepts, written by Richard Fairley in 1985,
1
to use as a starting point. Today there are many software engineering textbooks and articles to choose from, but let's use Software Engineering: A Practitioner's Approach, Sixth Edition, by Roger S. Pressman
2
as our main source for comparison. Currently in its sixth edition, this book has weathered the test of time and has become a standard reference in software engineering.
If we were to use volume as our only indication of maturity, we could stop right here; Fairley's book weighs in at a mere 364 pages, while Pressman's -- like many other popular software engineering texts -- is a hefty 880 pages.
However, as a more sophisticated comparison will be more meaningful, we will use the IBM® Rational Unified Process,® or RUP,® as our guide. RUP describes a software engineering process as activities, roles, and artifacts distributed across several disciplines. We will use the main disciplines of requirements, analysis and design, implementation, and test as our framework for comparison.
Fundamentals
The term software engineering came into common use in 1968, when a workshop on growing problems in software technology was held under the auspices of NATO in the town of Garmisch-Partenkirchen, West Germany. What does the term mean? In 1985, the IEEE Standard Glossary of Software Engineering terminology defined software engineering as: "The systematic approach to the development, operation, maintenance, and retirement of software." Software included computer programs, procedures, rules, and possibly associated documentation pertaining to computer system operation.
In 1993, the organization revised this definition to become: "Software Engineering: (1) The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software. (2) The study of approaches as in (1)."
As an alternative, Fairley offers a distinctively optimistic definition in his book: "Software engineering is the technological and managerial discipline concerned with systematic production and maintenance of software products that are developed and modified on time and within cost estimates."
Of course, there are other definitions too numerous to mention here; the Wikipedia
3
says that software engineering is "... the profession concerned with creating and maintaining software applications by applying technologies and practices from computer science, project management, and other fields." This simple definition nicely encompasses the breadth of software engineering. Let's agree to use it for this article.
In my opinion, you cannot discuss the state of software engineering without discussing process. If software development is, in fact, a form of engineering, then there should be a well-understood process that tells us how to practice the discipline. We can certainly see significant changes in the state-of-the-practice over the last twenty years. In 1985, almost all projects with a well-defined Software Development Lifecycle (SDLC) used the waterfall model, shown in Figure 1. Anyone who has taken a course on software engineering knows the problems with this model, and I will not reiterate them here.
Figure 1: Waterfall software development lifecycle
In 1988, Barry Boehm introduced a spiral model for software development (Figure 2) that became the root of all modern software methodologies. This model recognizes the pervasive nature of change in software engineering and describes an iterative, incremental approach to dealing with change and reducing risk. This change represents perhaps the biggest advance in software engineering over the last twenty years.
Figure 2: Spiral model of the software development lifecycle
However, we have seen one other fundamental paradigm shift during the past two decades. In 1985, we were just beginning to understand the power of object-oriented systems; we were also starting to see development languages and environments that supported these systems. Fairley's book does not discuss object-oriented analysis or design, but today, an object-oriented approach is the de facto standard for developing software.
Requirements
We have long known that if we want to build software that solves a problem, we need to define the problem. In 1985, we were looking at many different ways to define a problem by writing good requirements, and a software requirements specification (SRS) was the main vehicle programmers used to record them.
The SRS was a large document; modern processes, such as RUP, call for distributing this material into several, more specific artifacts. A typical monolithic SRS, as described in Fairley's book, consisted of twelve sections:
- Product overview and summary
- Development, operating, and maintenance environments
- External interfaces and data flow
- Functional requirements
- Performance requirements
- Exception handling
- Early subsets and implementation priorities
- Foreseeable modifications and enhancements
- Acceptance criteria
- Design hints and guidelines
- Cross-reference index
- Glossary of terms
RUP includes several of these sections in other artifacts, such as the Glossary, Software Architecture Document, Development Case, and so on. Software engineers in 1985 didn't give much thought to usability requirements. The personal computer explosion had not occurred yet, and computers were still for specially trained users.
In 1985, the software development community knew nothing about use cases, which today is probably the most widely used form of specifying functional requirements. Use cases give us a vehicle for specifying requirements that customers and developers can understand and review. This represents a major advance in software engineering. In the past, SRS documents that were written for many projects, especially large government systems, filled several volumes and sat on shelves, unread and unused, except by people whose job it was to ensure that the requirements were formatted according to standard.
A large portion of the chapter entitled "Software Requirements Definition" in Fairley's book is about different specification methods. It covers data flow diagrams and several formal specification techniques, such as relational notations, state notations, regular expressions, and so on. In contrast, Pressman's book relegates the formal aspects of requirements specification to a chapter in the back of the book, in the Advanced Topics section. We certainly have not given up on formal methods, but we realize that they are still quite difficult to understand, require specialized knowledge, and do not have great tool support. Much of our current exposure to formal specification comes with our use of languages, like the Object Constraint Language (OCL), to provide rigor to our specifications and UML diagrams.
4
Pressman's chapter on requirements engineering discusses the process for managing requirements, how to elicit and negotiate requirements with stakeholders, how to write use cases, and how to develop a good analysis model. It focuses on the interaction between project staff members rather than specific technologies or formal techniques. In the last twenty years we have become more aware that software development is a team-based endeavor that relies heavily on the people involved. The agile movement of the last five years has reinforced the need to deal with people on a human level, rather than treating them simply as human resources.
Overall, the requirements discipline has come quite a way in the last two decades. We have a better understanding of two things that are critical to delivering good software systems:
- Change is a constant. We know things change and, unlike twenty years ago, we plan for change. We manage our requirements in a way that facilitates change.
- We know that requirements evolve iteratively and incrementally. We use processes that help us get closer to the stakeholders and work with them constantly to validate our understanding of requirements and ensure that we are meeting their expectations.
Our understanding of just these two things has helped us to produce better systems that deliver more value than ever before.
Analysis and design
In 1985, requirements specification was part of analysis, which was the basis for design. If we were lucky, there was a feedback mechanism in place, but more often we arbitrarily decided when we had created all the necessary artifacts and could forge ahead to designing the system. Most of the really good software developers I know worked iteratively. They thought up a solution for part of the problem, implemented that solution, and made sure it worked with the solutions for other parts of the problem. However, this was all "under the covers"; they didn't want to let their managers know what they were doing, because iterative efforts didn't fit into the well-defined process.
When Fairley wrote his book, various industry segments were using quite a few different design methodologies as well as Computer-Assisted Software Engineering (CASE) tools they bought at hefty prices. Project teams stored all of their software artifacts in one repository and used proprietary vendor tools to work on them. These tools were typically resource hogs -- slow and prone to error. We also found that the artifacts the tools supplied weren't that useful; many vendors neglected the most important artifact in a software system: the actual code. Nevertheless, this was a fertile time for the industry; people were figuring out what might be useful in an integrated development environment (IDE).
The prevailing paradigm for programming was functional decomposition with procedural languages. The major programming languages of the day were COBOL, C, Pascal, FORTRAN, and Ada, a complex language. Most Ada compilers were either slow or ran on specialized hardware, such as Rational's RS/6000 with its APEX environment designed for Ada development. Over time, Ada users began evolving a new approach to developing programs, based upon cooperating components.
At that time many popular design methodologies were tied to CASE tools, including the Jackson Structured Programming (JSP) method and its counterpart, Jackson Systems Development (JSD), both developed by Michael Jackson, who continues to be a thought leader in design methods. JSP and JSD, often taught in university software design courses, had a fairly rich notation and accompanying semantics for constructing a system model. However, these notations were by no means a standard. Combining JSP with another design method, such as Structured Systems Analysis and Design (SSAD), entailed a lot of wasted effort and opportunities for error.
Fairley's chapter on design concentrates on modules and structure, stepwise refinement, and abstraction -- methods that have been either superseded or greatly improved upon since then. The notations included data flow diagrams, structure charts, HIPO diagrams,
5
procedure templates, pseudo code, structured flowcharts, structured English, and decision tables. Veteran readers may remember these, but, like my slide rule, they have become almost obsolete.
Today's analysis and design methods and tools are vastly improved. One of the biggest advances is widespread use of the Unified Modeling Language (UML). However you use it -- informally, formally, extensively, or casually -- UML is the lingua franca of software development. The unification of notation and semantics, along with major power boosts in hardware, have allowed software tool companies to create powerful IDEs that provide much more value than the older CASE tools, which have mostly gone by the wayside. In addition, whereas in 1985 we often found ourselves force-fitting projects to a particular process or toolset, today's open environments -- such as Eclipse and NetBeans -- allow us mix and match tools from different providers to suit a particular project.
Design itself has undergone deep changes as well. Component-based development was just starting to appear in 1985. Although some thought leaders understood how to create a good, robust, flexible architecture that would adapt to user needs, they had not yet expressed their understanding in a form that mere mortals could understand. Eventually, Grady Booch and others published books and papers that helped the rest of us understand what a good design looked like and how to produce one.
6
Today, undergraduates are fully conversant with those ideas and learn to apply them quite easily.
A second major design advance was the pioneering work of Erich Gamma on software patterns. His 1991 dissertation at the University of Zürich laid the groundwork for the explosive development of such patterns, and he served as lead author of the classic book, Design Patterns, which he cowrote with the other "gang of four" members.
7
Patterns gave designers and implementers a common language for expressing concepts, and they quickly developed a life of their own. Today there are myriad books, conferences, and workshops devoted to patterns.
Looking at Pressman's book, we see further evidence that a lot has happened in analysis and design. The five chapters devoted to this discipline -- on analysis modeling, design engineering, architectural design, component-level design, and user interface design -- take up almost 200 pages.
Implementation
Fairley devotes one chapter to what he calls "implementation issues," and one to modern programming languages. Pressman did not feel that implementation was worthy of its own chapter. The same is true for another modern software engineering book I checked: Software Engineering, by Ian Sommerville.
8
These omissions reflect a trend in software engineering to push implementation out of the software engineering domain. In my opinion, this is a mistake. RUP includes an implementation discipline for good reason. During implementation, all the work that went into understanding the underlying business problem and designing an effective solution comes together into a tangible result. A balanced treatment of software engineering should treat this critical discipline on a par with requirements, analysis and design, and so forth.
According to Fairley, implementation issues in 1985 consisted of:
- Structured coding techniques
- Coding style
- Standards and guidelines
- Documentation guidelines
Though this list is fairly small, it encompasses many contemporary implementation issues. However, it does not include today's refactoring techniques, unit testing tools and techniques, and so on.
Fairley's chapter on modern programming languages highlighted major features we considered to be modern in 1985. Today these features are so common that we would not think them worthy of a separate chapter. We all know about type checking, separate compilation, user-defined types (classes today), and abstraction. However, the sections on exception handling and concurrency mechanisms are still valid as advanced topics. I've found that many developers still do not understand these things very well.
To be fair to Pressman and other modern authors, they do discuss many implementation issues in other chapters. For instance, Pressman discusses debugging in his testing strategies chapter and abstraction and object-oriented methods in his design chapters. However, he presents the issues at a higher level than one would normally expect for implementation. In fact, even RUP provides few low-level details about implementation. The prevailing attitude seems to be that systems are built and deployed by knowledgeable professionals, so implementation is not a big challenge. I tend to disagree.
Test
Test has been a major part of software engineering since we coined the term; if we were claiming to be engineering professionals, then we had to produce high-quality products. Since Glenford Myers published his classic work, The Art of Software Testing,
9
in 1979, software testing has blossomed into a full-fledged profession. It fits well into engineering since it deals with empirical results: how many defects are in the product, how much has the testing covered, and so on.
Fairley's book has one forty-page testing chapter covering:
- Code reviews and inspections
- Unit testing
- Static analysis and symbolic execution
- System testing
- Debugging
- Formal verification
Some of these procedures were quite formal but little used. Even today, formal verification is mainly a research area.
Pressman has two chapters on software testing that cover eighty pages. The first one, on testing strategies, covers the concepts of verification and validation, organizational issues, strategies for different types of software, debugging, and so on. The second chapter is about the specific techniques of white-box testing, black-box testing, and several other methods that can be applied at different levels.
Discussions about testing often include software metrics and advocate a broader, quality assurance program. Fairley mentions source code metrics in a short section, but Pressman has an entire chapter on software product metrics. This illustrates how far we've come in bringing empirical methods to software engineering. This approach has led to more formal efforts to achieve software process maturity, such as the Capability Maturity Model (CMM), and agile methods such as Extreme Programming and Scrum. We now want the ability to measure the effectiveness of an organization or team -- and even individuals.
10
In 1985, the best known metric model for project management was Barry Boehm's Constructive Cost Model (COCOMO) for predicting effort and cost. At the code level, Maurice Halstead's software metrics and Tom McCabe's cyclomatic complexity model were widely accepted. All of these models are used today in some measure. COCOMO has evolved to COCOMOII, and a more robust version of McCabe's metrics is included in many commercial and open source analysis tools.
The testing discipline has experienced perhaps steadier growth in terms of knowledge acquisition, theory, and application than the others we've looked at. It may not have advanced proportionately, but it certainly has received continual attention over the past two decades.
Conclusion
Software engineering has come a long way in the last twenty years. Yet, there is so much more we have to do. We still suffer from a higher rate of late and cancelled projects than most engineering disciplines. Although software has become ubiquitous and critical for our daily existence, the fundamental laws of software development, if they exist, have yet to be discovered.
Software engineering is not computer science -- it is bigger. It is worthy of both serious, scholarly study and effective application. The road ahead is filled with possibilities for improvement, and we can expect the next twenty years to be exciting for software engineering researchers and practitioners. However, along the way we can also expect to travel down side roads that lead to nowhere. That's what it takes to innovate and progress. What is the major trend in software development today? As I see it, theory is being translated into practice at an ever increasing rate. And where will that lead us? Be sure to check the December 2024 issue of The Rational Edge, which will have my retrospective on software development in the first two decades of the twenty-first century.
Notes
1 Richard Fairley, Software Engineering Concepts. McGraw-Hill, 1985, ISBN: 0070199027. Fairley was a professor at WIGS at the time the book was published.
2 Roger S. Pressman, Software Engineering: A Practitioner's Approach, Sixth Edition. McGraw-Hill, 2005, ISBN 007301933x. The first edition of this text came out in 1982. If I had a copy of that edition, I would have used it instead of Fairley.
3 http://en.wikipedia.org/wiki/Software_engineering
4 For an introduction to OCL, see "Formally Speaking: How to Apply OCL," in the July issue of The Rational Edge.
5 HIPO stands for Hierarchical Input Process Output.
6 Grady Booch's landmark object-oriented design book, Object-Oriented Analysis and Design with Applications, was published in 1990 by Addison-Wesley.
7 Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides, Design Patterns. Addison-Wesley, 1995, ISBN: 0201633612.
8 Ian Sommerville, Software Engineering, Seventh Edition. Addison-Wesley, 2004, ISBN 0321210263.
9 G. Myers, The Art of Software Testing. Wiley Interscience, 1979.
10 Metrics for individual performance are described well in the Personal Software Process (PSP) developed by Watts Humphrey. See http://www.sei.cmu.edu/tsp/psp.html
About the author | 
| | Gary Pollice is a professor of practice at Worcester Polytechnic Institute, in Worcester, MA. He teaches software engineering, design, testing, and other computer science courses, and also directs student projects. Before entering the academic world, he spent more than thirty-five years developing various kinds of software, from business applications to compilers and tools. His last industry job was with IBM Rational software, where he was known as "the RUP Curmudgeon" and was also a member of the original Rational Suite team. He is the primary author of Software Development for Small Teams: A RUP-Centric Approach, published by Addison-Wesley in 2004. He holds a B.A. in mathematics and an M.S. in computer science. |
Rate this page
| |
