![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
For most of us, complexity is an "I know it when I see it" (IKITWISI) kind of thing. In software development and delivery, we use often use the term "complexity" without any agreed, technical definition. This is understandable, since complexity is hard to define. The wide range of technical definitions or measures from different fields,
1
including software design,
2
show there is no easy, single definition.
As I discuss in this article, complexity is both unavoidable and costly. In fact, for a variety of reasons discussed below, complexity is increasing in our personal and technical lives. It is fair to say that complexity, in its various manifestations, is one of the key technical and business concerns of our time.
Yet, there are general ways to diminish the impact of complexity. That is the topic of this article.
Generally speaking, something that is complex has many interconnected, interacting parts. With that in mind, the first step in diminishing complexity is to understand the degree of complexity involved; i.e., we need a way to measure it. Here are a couple of pragmatic criteria we can use to identify complexity and reason about measuring it. These criteria apply to software, systems, project efforts, and business organizations -- actually any entity that has interacting parts:
- Predictive. The difficulty of knowing how the entity will behave in the future from its current state or history. This is important for project efforts and organizations.
- Retrospective. The difficulty of understanding how the entity got into its current state. This important for system and software and system debugging and underlies most system and software complexity measures.
One of the key insights of the last half-century is that by these measures, complexity arises remarkably quickly. 3 Simple things interacting with simple rules quickly become complex by either of the above perspectives. For example:
- Swarms, such as those created by ant or termite nests, bee hives, and bird flocks, each consist of individuals who follow remarkably simple rules, yet collectively, their behavior is impossible to predict and the history of swarm behavior cannot be reconstructed.
- The Internet is made up of many millions of relatively simple nodes. Yet it has generated entirely unanticipated social and business behavior.
- A large software program, even if well designed, is complex.
However, another century-old lesson is that simplicity is desirable. A chief goal of architecture and design is to find the simplest solution possible for an entity that meets stakeholder needs. Simpler things are generally more robust, less likely to break, and easier to repair. This principles holds for business and organization design. Unfortunately, computer science teaches us there is no practical way to know if you have the simplest design.
The moral is, as long as the entity has interacting moving parts, expect complexity.
In our professional lives, complexity applies to:
- The systems and software we deliver
- The teams we participate in
- The business organization we belong to
In all cases, there are two sources of complexity:
- Context. How the entity interacts with external entities.
- Design, Architecture. How the internal entities interact.
Contextual complexity is important because of our desire to anticipate the behavior of the world the entities listed above will interact with. If we can understand the future behavior and needs of the members of the context, then we can:
- Plan and design systems with confidence in their ongoing utility
- Create organizations that will provide stability for the business and the employees
However, because of the Internet or some other network technology, all of the entities in the context are likely connected to each other, as well as others outside the given context, even if indirectly. The connections among these context entities create a huge global web of possible interactions. The implications of all this connectivity have received lots of real and virtual ink. For instance, there are growing indications that development teams of the future will behave like members of a social web. The point is that all of the entities are members of a sort of swarm and so context complexity is inevitable.
The force driving design complexity is different. Technology is permitting us to have increasingly stateful devices and systems on which we host increasingly capable, stateful software. Examples of such devices include VHSIC (Very High Speed Integrated Circuits) Hardware Description Language (VHDL) modules in commercial products, electronic control units in vehicles, personal computers, enterprise blade servers, and mainframes with virtualization. The expectation is that the software will be distributed across these devices to provide functionality as often as needed or to balance processing to meet unpredictable loads. This sort of complexity is especially evident in enterprise software with service-oriented architecture (SOA) implementations, but could occur in all distributed software architectures. These entities exhibit both kinds of complexity. Hence, it is very difficult to debug or predict the performance of these entities.
As discussed above, complexity can't be avoided or even proven to be minimized. However, it can be dealt with. Within the space of entities that provide the same services and meet the same needs, some entities are simpler than others, and finding simpler system and organization designs has significant economic benefit. Left unchecked, complexity leads to systems that are very expensive to maintain and adapt and so are soon obsolete. These systems deliver less value than simple systems. Complex development organizations suffer from diseconomies of scale; most of their efforts are spent with internal communications, not with providing stakeholder value.
So while you cannot defeat complexity, you need to continue the struggle. In this section, I will review some key techniques.
There are two established techniques for managing system and software complexity:
- Architecture and design
- Simulation and testing
I discuss each below.
Architecture and design
Much has been written about the role of architecture in enabling engineers and developers to reason and communicate about a system. Two architectural attributes are
- Encapsulation -- hiding the internal data and methods
- Cohesion -- how the encapsulated entities are connected
Figure 1 shows two diagrams representing two entities with the same number of components. In the upper right there is no encapsulation and high cohesion. The lower left, the components are encapsulated into three larger components with less cohesion. This entity is less complex. Good architectures in all domains strive to defeat complexity by applying this principle. Achieving the right amount of encapsulation and cohesion is still an art. Too much cohesion and the system suffers all the ills of complexity. Too little, and the system becomes stovepiped and brittle.
The second way architecture can help manage complexity is to anticipate change -- i.e., to design the system knowing it will have to adapt to a changing context. Architecting for change consists of two activities:
- Designing the model so that it includes the encapsulated system and what it interacts with (the "actors"). The system attributes and services are traced to system requirements. With this in hand, one can quickly analyze the impacts of changes in the context on the system.
- Regularly testing the architecture for its ability to react to the inevitable changes. An example of this is to measure the amount of change in the architecture engendered by a change of interface, added capability, or new context entity. Note that attacking complexity takes effort and investment, but the cost of complexity is generally much greater.
Figure 1. Both upper right and lower left diagrams represent a similar complex entity, but the architecture illustrated in the lower-left diagram shows how encapsulation reduces complexity.
Testing and simulation
Since, for complex systems and contexts, it is impossible to anticipate the behavior and performance of the context and systems in place, one needs to analyze, simulate, and test the behavior to get at least a statistical view of the system-delivered function and performance. Early in the development lifecycle, one uses simulation and analysis tools. As the system comes together, one replaces the simulations with functional and load performance testing. Note that this approach works best when using some form of iterative development.
Like a system, we can model how an organization interacts with its context. This is the realm of business modeling. A particularly compelling use of business models arises in the SOA practice. SOA encapsulates the organization as a service provider; it interacts with other organizations by providing and consuming services. The goal of SOA is to deploy services that allow an organization to cope with context complexity by being able to respond quickly to unforeseen changes.
Operational governance 4 provides some methods for dealing with internal complexity. In particular, setting the Responsible, Accountable, Consulted, and Informed (RACI) framework provides a means for dealing with organization cohesion. One effective way to set the RACI framework is to identify the decision points in organization processes, especially when process artifacts change state. At these decision points you then identify or determine who, by role, is responsible for the decision, who in the management chain will be held accountable, who must be consulted, and who should be informed. If this structure is too dense, then there is too much cohesion and the organization is likely to be too complex to be effective.
Development teams have special complexity concerns. Their context includes a set of stakeholders (requirement setters, users, financial control officers, etc.) often with conflicting and evolving concerns. Sometimes the stakeholders who set requirements gain better understanding of their needs and update requirements in flight; sometimes their needs actually change because of changes of their context. In any case, development teams need to adopt team methods that anticipate change. Expecting context stability for development leads to delivering less than desirable systems. This is a key premise for agile development 5 .
Also, development teams scale in size from small to medium and large, and they become complex in their own right. The behavior of the team itself becomes hard to predict. 6 Again, encapsulation and cohesion applies. Following Conway's Law, 7 the optimal way to partition the team to achieve encapsulation is along architectural boundaries. By setting teams to work on architectural components or subsystems, you achieve high cohesion when you need it, working on tightly coupled modules, with less cohesion across teams to better negotiate capability and interfaces. You also encapsulate the teams so that the discussions can be restricted to those aspects of the encapsulated components that are externally visible, yielding the optimal amount of communication. Look back at Figure 1, and imagine each node is a person and each diagram is a team design. Which team would you prefer to be on?
Conclusion -- An old and apt metaphor
The ocean is complex by any definition; in this case the entities are water molecules. Yet, some live for interacting with its unpredictable motions; they are surfers. For a while, "surfing" has been a metaphor for the capability of not only dealing with complexity, but even using it for ones own purposes. We surf the Web. In the 1980s, we surfed the changing business environment. We understand the future belongs to those who can metaphorically surf. They do not fight complexity, they revel in it. Being good at working with complexity, reducing it when you can, riding it when you can't, is essential in today's business world.
In following issues of The Rational Edge, I will discuss some specific methods for dealing with complexity. So, relax, stay responsive, and enjoy the ride.
1 Ty a Google search "Define: Complexity."
2 My favorite is still the McCabe measure. See McCabe, Thomas J., "A Complexity Measure," IEEE Transactions on Software Engineering, SE-2 No. 4, pp. 308-320, December 1976.
3 There is a broad literature on dynamical system theory, chaos theory, and cellular automata theory, all of which apply. See for example, the journal, Complex Systems, http://www.complex-systems.com/about.html
4 See http://www.ibm.com/developerworks/rational/library/may07/cantor_sanders/
5 See, for example http://agilemanifesto.org/ and the writings of Scott Ambler, http://www.ambysoft.com/scottAmbler.html
6 M. Cantor, Software Leadership, Addison Wesley, 2001.
7 See http://www.melconway.com/law/index.html and http://en.wikipedia.org/wiki/Conway%27s_Law
- Participate in the discussion forum.
- A new forum has been created specifically for Rational Edge articles, so now you can share your thoughts about this or other articles in the current issue or our archives. Read what your colleagues the world over have to say, generate your own discussion, or join discussions in progress. Begin by clicking HERE.
-
Global Rational User Group Community
As a leader in the IBM® Rational® field services group, Murray Cantor promotes and extends Rational best practices, including the IBM Rational Unified Process® for Systems Engineering (RUP®-SE). Murray has been named Distinguished Engineer both for his contributions to RUP-SE and for his successes with client enterprise transformations. He has published two books and numerous papers, and plays a key role on standards committees relating to the Unified Modeling Language (UML) and RUP. He received his Ph.D. in mathematics from the University of California at Berkeley in 1973.
Table of contents
Local resources
Tags
Use the slider bar to see more or fewer tags.
For articles in technology zones (such as Java technology, Linux, Open source, XML), Popular tags shows the top tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), Popular tags shows the top tags for just that product zone.
For articles in technology zones (such as Java technology, Linux, Open source, XML), My tags shows your tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), My tags shows your tags for just that product zone.
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
