Evolutionary architecture and emergent design
Emergent design in the wild
The when, why, and how of emergent design
This content is part # of # in the series: Evolutionary architecture and emergent design
This content is part of the series:Evolutionary architecture and emergent design
Stay tuned for additional content in this series.
Many of my examples throughout this series rely on the Lean Software concept of deferring decisions to the last responsible moment. But how do you know when that moment has arrived? And how do you know how far into the future you can push design decisions before a better path emerges? What kind of projects benefit the most from emergent design? This final Evolutionary architecture and emergent design installment answers those questions, then reviews some of the main points from the series as a whole.
To set the stage, I start with the poetry of Donald Rumsfeld.
There are known unknowns. That is to say there are things that we now know we don't know. But there are also unknown unknowns. There are things we don't know we don't know.
—Donald Rumsfeld, Former Secretary of Defense for the United States of America
Rumsfeld distinguishes categories of unknowns, starting with the unknowns that we know exist but haven't found. But he also acknowledges the existence of things so beyond our knowledge or experience that we don't even know to look for them: the "unknown unknowns."
Rumsfeld was talking about the uncertainty of war, but he could have been talking about software. If you have written any nontrivial software, you can relate to the unknown unknowns problem — one of the biggest problems in software design. You think you have a firm understanding of the issues you'll encounter as you implement a solution but, inevitably, unexpected problems arise. Interacting with an open source framework, say, isn't as straightforward as you thought, or a problem has more nuance, requiring much more careful thought, than it originally seemed.
Unexpected design issues arise constantly because software has low failure tolerances — much lower than physical systems. For example, consider the building around you. Constructing the building was a multiperson, multimonth (or multiyear) project. Now consider a multiperson software project constructed over a similar timeframe. Those projects have a similar scale, yet the physical building is vastly more tolerant of structural defects. If a light-switch plate doesn't completely cover the hole in the wall that houses the wires, the building won't fall down. Yet a small defect in software can make it crash. Compared to atoms, the failure tolerance of bits is unforgiving. Of course, software (being soft) is also easier to fix: we can spackle in the hole (fix the bug) and instantly remanufacture the house.
If a wing falls off an airplane, the forensic engineers will look first at where the wing attaches to the airplane: in physical systems, proximity of error to function is frequently high. But in software, it is common to execute a line of code that causes an instability that doesn't manifest for hundreds or thousands of seemingly unrelated lines of code away.
Design up front for software is difficult because each part interacts with each other part in myriad, sometimes unexpected ways. Predicting the implications of those interactions is beyond our current capabilities. Emergent design embraces the inevitability of surprising complexity and tries to mitigate the damaging effects of change.
Spectrum of design
Emergent design isn't a binary state. You can't definitively say that your design is 100 percent agile or 0 percent agile; it exists on a spectrum, as shown in Figure 1:
Figure 1. The spectrum of design
On the far left in Figure 1 you have traditional Big Design Up Front (BDUF), embodied in many common development methodologies. BDUF, in its finest straw-man form, embodies ivory-tower architects creating design artifacts, dropping them through a hole in the floor for hapless coders to implement with no changes. In nonparody form, this design methodology diligently tries to discover everything interesting before coding starts. It is a predictive, proactive model of designing software.
The right-hand side in Figure 1 represents the kind of coding you did in high school: you hack on it until it works, then continue to hack to make changes. This works well at small scales (basically, as long as the problem is small enough to fit inside your head) but doesn't scale much beyond that.
Emergent design falls in between these two extremes, but more toward the right than the left. Emergent design is a responsive, reactive way to design software.
It's puzzling that so many organizations continue to use BDUF in the face of so many failures and underachieving projects. I'm not suggesting that you can't use BDUF successfully. (In fact, I've done many such projects in the past.) But the track record for that development methodology is poor and has been for decades. Fred Brooks' seminal book Mythical Man Month discussed the problems with building software in that style, and it was published in 1975 (see Related topics).
It's not surprising that teams attempted this style of development because, metaphorically, it matches how design works in more traditional engineering. If you are making widgets or iPods, you must do all the design up front because you can't refactor atoms. Consider the original Intel Pentium processor. After it was released, a bug in the floating-point math unit was discovered, requiring every operating-system maker to write special code to accommodate the problem. Once the manufacturing process is complete, you can no longer make changes to hardware. Software is the polar opposite: most of the life of a software project takes place after the initial release, via enhancements, bug fixes, and other "maintenance" activities. We deal with bits instead of atoms, and bits are infinitely more malleable.
How much design up front?
Agile design doesn't try to ignore design at the beginning of a project or phase. It tries to do as little as possible early in the process, when you know least about the real nature of the problem. I'm frequently asked, "How do you decide how much design is appropriate at the onset of a project?" Different kinds of projects have different criteria, and they vary in type and complexity more than most nontechnical people realize. Basically, you're trying to balance two things: up-front design works when your early decisions are accurate enough to avoid change in the future and the cost of later change is large enough that the benefits matter. Thus, you need good knowledge in the early stages combined with development techniques that make subsequent change expensive. Agile design objects to this notion because people tend to overestimate their ability to make accurate decisions early, and they suffer the ever-expanding consequences later.
Here are some examples of projects for which you should do more design up front:
- Those with extremely stable requirements, with no changes expected on the order of years.
- Ones for highly isolated environments (for example, space travel, underwater exploration), for safety reasons.
- A project for which you've written the exact same piece of software before, with the same group of people, with no scope changes. (You'll get a deadly accurate estimate for this one.)
- Projects for highly constrained environments, such as embedded systems, to make sure that you consider the constraints with that environment. (I would still try to allow behavioral functionality to emerge as much as possible.)
Types of projects for which you should not do too much up-front design include:
- Projects with highly variable, changing requirements (on the order of months or weeks), like most business applications.
- Those that need to respond to external factors, such as market conditions.
- Projects for which you're not yet sure about many of the technical or business details. Note that this encompasses virtually every project, harking back to Rumsfeld's "unknown unknowns."
- Projects that benefit from a subscription to development rather than the artificial distinction of "done." No software project is ever done, so you are always really buying a subscription, and the sooner you realize it, the sooner you can act like it.
Choosing the right time to make decisions is difficult but important. All projects are unique, making specific advice useless. However, some general guidelines exist:
- Notice when "the simplest thing" that works has been working for a while, yet the problem it was solving takes a significant jump in importance or complexity. For example, let's say that you have been using a database as a simple messaging queue for basic background, asynchronous behavior. Now, though, a couple of new requirements will add to the variety of asynchronous behaviors at the same time that the performance is starting to suffer. Now is the perfect time to revisit your "simplest thing that works" decision, because your solution no longer matches that criterion.
- When looking at the overall scope of the problem, try to isolate pockets that will tend toward emergent design techniques. For example, let's say that you are working on an application that needs geocoding support, and you've never worked with a geocoding library before. You should perform a few spikes (simple, directed research-and-development projects) to make sure you have enough understanding for estimation purposes. Try to prevent making any architectural (hard to change later) decisions based on incomplete understanding, and revisit the interface point between the rest of the application and this isolated part to see if you can make improvements based on better understanding.
- Try to keep interaction points between applications fluid. One of the undesirable side effects of protocols like Simple Object Access Protocol (SOAP) is their insistence on rigid structure and strong types. The secret to flexibility is less specificity, not more, the realization of which is driving so much interest in Representational State Transfer (REST) and related technologies. When building an API, try to accept the most generic kinds reasonable, which applies to integration as well.
I'll end the series with a summary of the main themes from the previous 18 installments.
Code as design
Back in "The Relationship between code and design," I cited an article by Jack Reeves that suggests that software design consists of the entire source code for a project, not just the artifacts we normally think of as design (white-board diagrams, Unified Modeling Language, and so on). He compares source code to the plans created by engineers that specify everything that a manufacturing team needs to convert the design to atoms. Our plan is the source code, which the compiler converts to bits. If code is design, then the computer languages and frameworks we use define the raw material for what we can design.
The leverage provided by more-powerful languages offsets the scariness of their advanced features. Although it's nice to choose and standardize on languages more than a decade old because you have a ready supply of low-cost developers, you must also accept the fact that you cannot move as fast as your competition who are using more-modern languages and tools. Many organizations place too much emphasis on standardization at the expense of both innovation and sanity. I've seen more than one project forced to use a standard set of frameworks that were ridiculously inappropriate for the problem, which harms any kind of design discovery because it is obscured by the accidental noise.
This doesn't mean that you should allow every developer to choose his or her own tool stack. It does mean that you have to keep a close eye on the real value provided by standardization and consider reasonable alternatives. For example, perhaps you want to keep using Java technology for your projects, but you can start using more advanced tools for build, testing, and other developer tasks. Code appears everywhere in software projects, and it all embodies design, either conscious or not.
Many methodologies try to avoid uncertainty. If you are building with atoms, uncertainty is bad because it's expensive to change configurations of atoms once they've been forced into a certain shape. But when you're building with bits, change is easy and highly desirable. Avoiding change is both difficult and undesirable in software.
Agile methodologies try to find ways to make change easier, using techniques such as unit testing, refactoring, Continuous Integration, and iterative development. Emergent design embodies agile philosophies about design. When design decisions arise, ask yourself:
- Do I need to make this decision now?
- Can I safely defer this decision?
- What can I do to make the decision reversible?
If you're in an environment in which you can easily refactor your code, making a temporarily suboptimal decision isn't so scary, because you can fix it without too much pain. If you set up your projects to adapt to change, deferring decisions isn't damaging, because you have optimized for course corrections. Embracing change requires the ability to look at decisions with ruthless objectivity and to change the ones that are making things worse.
Harvesting idiomatic patterns
Another aspect of emergent design is the ability to see useful design elements that are already in your code and harvest them as idiomatic patterns, which I defined in "Leveraging reusable code, Part 1" as effective solutions to past problems. These discovered patterns are the crown jewels of your company because they are design elements that capture something useful about the way the company does business. A depressingly small amount of the code you write encapsulates unique value: the rest is mostly just pushing and pulling things into databases, building HTML, and so on. Idiomatic patterns are more valuable than ones cooked up in Joint Application Design (JAD) or ivory-tower design because they've already met one of the critical criteria for usefulness: they have already been used to solve an actual problem.
You can harvest these patterns in a variety of ways. You can create an API, which is mechanically easy but doesn't stand out much — it looks like all the other APIs you use. You can also capture some categories of idiomatic patterns using metaprogramming and attributes (see "Leveraging reusable code, Part 2"). I also discussed capturing domain patterns using domain-specific languages (DSLs) in "Using DSLs," "Fluent interfaces," "Building DSLs in Groovy," and "Building DSLs in JRuby."
Always try to recognize useful design elements, and try to avoid decisions that make it artificially hard to do so. For example, adding architectural elements to your application (such as layers for extensibility, or services in an service-oriented architecture) makes your code more complex as soon as those elements are added to the code base, not when you start using them. Make sure that you are adding complexity at the right times, because it is accidental complexity until you start using it.
One of the goals of this series was to force myself to look at design in different ways and document that trip. I hope you've enjoyed riding in the passenger seat. I'm not giving up my sojourning just yet, just changing the subject. Stay tuned for my next developerWorks series on functional thinking.
- The Productive Programmer (Neal Ford, O'Reilly Media, 2008): Neal Ford's most recent book expands on a number of the topics in this series.
- The Poetry of Donald Rumsfeld: Some of Rumsfeld's pronouncements have been put in verse form and set to music.
- The Mythical Man Month, 2d ed. (Fred Brooks, Addison-Wesley, 1995): This book is a seminal work in software development, exposing some of the unintuitive but true quirks of software projects.