Why is Toyota the Number Two automobile manufacturer in the world, and steadily growing?1 How has the U.S. military improved equipment maintenance turnaround an astonishing 15,000%?2 The same way more and more forward-looking companies driving out inefficiencies in their supply chains and delivering better quality and lower prices to their consumers: Lean Thinking. Lean Thinking (also known as "Lean") is a set of revolutionary principles that guide us to reexamine our definitions of quality and instead focus on our customers' definitions of value.
Mary and Tom Poppendieck's book Lean Software Development3 translates Lean's manufacturing-oriented principles into software development terms. In this article, I'll introduce these principles and discuss how the lessons Lean has been teaching manufacturers can be applied to a software delivery organization. I'll also go a step further to explore how the IBM Rational Unified Process®, or RUP®, can provide the technology support required to implement a Lean approach in your software development organization.
Efficiency on the assembly line
Since the advent of the assembly line, the world of manufacturing has experienced evolution and revolution in best practice thinking. Exacting productivity from resources and core operations, connecting product development to customers, and improving quality are all unremitting goals each new best practice has sought to achieve. Lean Thinking is another such concept that for years has been making its way through the management ranks. It has been responsible for transforming enterprises, delivering higher quality products and services to consumers, and making Toyota the Number Two auto manufacturer in the world. Lean focuses on eliminating wasteful activities from the manufacturing supply chain. From the start of the value creation process until the product is delivered to the consumer, eradicating what the Japanese call muda (waste) is Lean's primary objective. As is also true for many other manufacturing concepts, the principles of Lean have real implications for those of us tasked with improving software delivery.
Let's look at what Lean Thinking is and how it applies to software development. The Lean movement, initiated by Toyota executive Taiichi Ohno, seeks to "do more and more with less and less -- less human effort, less equipment, less time, and less space -- while coming closer and closer to providing customers with exactly what they want."4 More specifically, Lean Thinking focuses on eliminating seven forms of waste (see Table 1) that prevent the organization from achieving the "more with less" goal. Waste is defined as any activity that does not directly add value to the finished product, with value being defined by the ultimate consumer of the product or service.
In software development, eliminating waste is by no means a new idea. However, we look at waste most often from the viewpoint of defects and throwaway designs. Lean Thinking advocates regard any activity that does not directly add value to the finished product as waste. In the case of software development, the authors paraphrase Winston Royce in stating that "every step in the waterfall process, except for analysis and coding, is waste."5 The spirit of this is clear; Lean Thinking should be applied in order to minimize the labor required to demonstrate executable code. Yes, we need testing and configuration management, too, but all other activities and practices are necessary evils that have evolved due to the limitations imposed by technology, communication, and management.6
| Waste in Manufacturing | Waste in Software Development |
|---|---|
| Inventory | Partially done work; e.g.: intermediate work products, usually documents and plans, components not integrated into the integration stream |
| Extra Processing | Extra processes; e.g.: paperwork, status reports |
| Overproduction | Extra features; e.g.: functionality that is not used by the customer, is not really necessary, and does not add real business value |
| Transportation | Task switching; e.g.: working on too many projects simultaneously |
| Waiting | Waiting; e.g.: waiting for sign-off, waiting for a build, waiting for completed test results |
| Motion | Motion; e.g.: moving from one development tool to the next |
| Defects | Defects; e.g.: requirements, design, or code related defects |
As long as there are business bookshelves to be filled, we will have a continual barrage of new ideas from management gurus and profiles on successful CEOs. So what makes this Lean concept different? How exactly can embracing Lean positively affect our organization? Here are some of the ways:
Reduced cycle times. Just as it can dramatically reduce cycle times in manufacturing, resulting in more inventory turn, Lean Thinking can speed up delivery of high-quality software. Software doesn't really have a true equivalent to inventory, but a software build or configured component comes close. These are the "physical" manifestations of the software delivery supply chain. I will discuss how Lean Thinking and a supportive development infrastructure can facilitate this software inventory turn a bit later in the article.
Higher quality solutions. Getting a software solution that matches the business needs of its stakeholders -- particularly while those needs are still evolving -- is obviously challenging. Organizations that have adopted Lean Thinking and the corresponding management systems are finding that quality can be delivered while simultaneously reducing cycle time and costs.
Selling the value of iterative and agile development. Advocates of iterative and agile software development methods often struggle to explain these concepts to their business counterparts. Given the need for tight coordination between the business and development disciplines, buy-in by both these stakeholder groups is critical for successful adoption of these practices. Yet resistance is often quite intense. I have personally encountered instances where the very mention of words like "agile" and "iterative" conjured up visions of developers hacking away until they finally got it right. Fortunately Lean and agile have a lot in common, as I'll explain below. "Lean Lingo" offers a way of conveying agile development concepts in a business vernacular. This is powerful because it has immediate value in bridging the communication gap between business and software organizations. It helps convey what iterative and agile methods are really about -- alignment, control, and efficiency in software delivery.
Key Lean principles for software delivery
While Lean Thinking as we know it today got started in manufacturing, many of the principles we associate with Lean have been applied to software development for some time, in a variety of guises. Software delivery is a supply chain much like any other. There are inputs, outputs, and value-added activities. Seeking to minimize the steps in the process that do not add value is just as valid a goal in software delivery as it is in manufacturing or services. That said, not all of the concepts of Lean apply to software, and some apply differently than their manufacturing counterparts. This section will introduce a few of the most important concepts of Lean and discuss how they manifest themselves in the world of software and systems delivery.
An integral idea in Lean Thinking is the concept of value flow, or simply flow. In the simplest of terms, flow seeks to eliminate the seven wastes elaborated earlier by producing and delivering products and services in a just-in-time manner. The authors of Lean Thinking warn that in order to really understand value flow you have to "rearrange your mental furniture." It is almost counterintuitive because it challenges most of what we have been taught. Understanding flow requires that we seek to improve the interactions between software delivery steps -- not just maximize efficiency of the steps themselves.
This concept is difficult to grasp immediately, so let's put it in a familiar context. Usually work is organized into a series of specialized activities performed in departments. For years, the belief has been that this is the best way to get efficiencies and economies of scale from your operations. One department optimizes their work to maximize efficiency for their piece of the product, and the next department does the same. Work is done in batches and sits in queues until the next department is ready to accept the input and process it further.
If you're building a car, for example, imagine one work team responsible for producing doors, and another team responsible for assembly of the entire car. Traditionally, the management team responsible for producing doors would strive to find the most efficient means of producing doors. They petition for equipment that enables them to produce the most doors for the least cost per unit. They optimize their work to ensure their team remains productive. In order to keep their team busy, they might even unknowingly produce more doors than there are frames to attach these doors to. Why would this be done? Not to be too cynical, but managers are usually rewarded for making a numerical efficiency target. A good way to make the numbers is to "produce in large batches in order to achieve high asset utilization and low cost per individual step in the process."8
In the world of software development, the result of this batch mentality is the "overproduction" of soft assets. These software assets are intermediate work products that are used to develop software, such as documents, reports, mockups, prototypes -- anything that is not ultimately used in the finished product. Frequently, we produce these artifacts, but they go unused by downstream development team members. For example, product management and analysts spend time writing requirements that are never implemented for various reasons (feasibility, cost, etc.). Time spent meeting, discussing, and documenting these detailed requirements is waste. Analysts often spend considerable time on the aesthetics of their documents. Developers implement complicated designs or plan for scalability and flexibility that may never be needed. Unknowingly, testers implement test cases for functionality that has been descoped. All of this is overproduction, hence waste.
Reducing this waste requires seeing the whole value chain, not just each function's responsibility. This "seeing the whole" is a requirement to achieve flow. As the Poppendiecks point out, "a mature organization looks at the whole system; it does not focus on optimizing disaggregated parts."9
Flow also requires a production rhythm. At Toyota, a new car rolls off the assembly line every 55 seconds. Each car started less than 20 hours prior as component parts. Each step in the production process has exactly 55 seconds to complete. That means there can be no waiting for parts to arrive from a supplier or different department. The upstream tasks cannot be delayed.10 This continuous flow of assets down the production line mandates that waste be eliminated.
Flow exists in the software business as well. From the IBM Rational perspective, the best practice of iterative development is the essential element that enables software flow. Iterations are short sprints to functionality that can be tested with the end user community prior to building more functionality. With iterative delivery of software, there are two dimensions of flow: the way code moves through the development cycle, and the limited timeframe and scope of the individual iterations.
Early review and testability
In the first dimension, the solution moves horizontally from its conception to a skeleton or architecture of the solution, on through the fully developed and then deployed solution (see Figure 1). Code moves through the development cycle much like an automobile moves through a production line. This allows a testable version of the product very early in the development cycle. This early version is not deployed for full use; rather, it is shared with stakeholders and serves as the basis of subsequent layers of software. Just like you could take an incomplete car off the production line and take it for a test drive even if there weren't doors, painted frame, or even seats installed, early versions of the application wouldn't be fully functional, but they are reviewable and testable.
Figure 1: Software flow in an iterative development process
Determining value by inspection
The same rhythm that is generated in Toyota's production system can be accomplished through a series of these iterations, in which layers of functionality are incrementally added. The key is that these iterations are time boxed. In other words, each piece of functionality (i.e., each elaboration of the architecture) must be completed within a preplanned time window. At the end of the iteration, either you have accomplished the goals of the iteration (a certain amount of tested functionality implemented in code) or you have not. You have instant validation of real, tangible progress. Just as in Toyota's system, you cannot hide lack of progress. Either the doors were attached to the car moving down the line on time or they were not. Either we have functioning code that passed the test cases or we do not.
This approach ensures we are adding value -- which is what concerns Lean Thinking executives. We can always inspect the state of the product to answer the question, "What value have we added?" This is counter to the conventional approach in which the status of the product is determined by determining which non-value-add activities have been completed. For example, "Has the requirements specification been signed?" or "Is the design document complete?" These deliverables don't directly add value to the solution.
Forcing issues to the top
Iterations have the added advantage of forcing issues to rise to the top; another important concept of Lean Thinking that is known by the Japanese word jidoka. In manufacturing, if there is a defect, the line is stopped to correct the issue. The traditional view of this would be that stopping the assembly line is bad. In software, we have the same concern with stopping a project. Often we brush over important design issues or technical risks in order to keep the project seemingly moving forward. With iterative development, this false progress is not possible because in the short period of the iteration, you must have working, tested code that yields some value or elimination of a technical risk in order to prove delivery of a result. The logic here is simple. It's easier to address issues before more development and other downstream activities are completed.
Small batch production
The second dimension of flow in an iterative development lifecycle is somewhat analogous to the supply of component parts that must be delivered for assembly on a car flowing through the line. The elements that are used to build a software component include the business requirements, the technical requirements, a defined design that fits enterprise architecture standards, and the tests used to validate the component's operation. Each team gathers a narrow set of requirements in every iteration. Hence, they implement only the business needs that are within the scope of the current iteration, thereby reducing development and testing efforts. Manufacturing calls this small batch production.
The key is that the teams work to build just what is needed for the current iteration and that the work products are delivered just in time to the next team in the delivery chain. Remembering that the only things that add value in development are the results-oriented tasks of the iterative method, information delivered to the downstream activity should be in a format that is most consumable by the downstream team. In most organizations, what is produced is a static -- usually paper -- document. What we know from significant industry experience is that static documents are a poor method for communicating. A preferred approach is to use a set of dynamic models: digital assets that are shared among the entire development team. The Object Management Group (OMG) calls this a model driven approach to development.11
Model-driven flow
Without diving too deeply into the semantics and various levels of models prescribed by the OMG and the IBM Rational Unified Process, let's look at how the models typically used to describe software architecture fit together (see Figure 2).
An analyst builds a representation of the software requirements in a use-case model, which is a set of requirements written in natural language from the end user's perspective. That use-case model is then used by the architect to drive the analysis and design models. These models can explicitly use content from the use cases to drive architecture. Next, the developer takes these digital models and turns it into an implementation model, or code. All of these models build on top of each other and ultimately drive the delivery of functioning code.
Figure 2: The relationship among models used in software development
So how does this flow of digital assets differ from just another flow of documents and specifications from one department to the next? Remember that the only two things that drive value in the software delivery value chain are results-oriented tasks: analysis, coding, testing, change management, and builds. When an analyst spends time documenting requirements in a form that is not directly usable by the developer, that activity does not add value to the final product. If an architect creates a design document, developers must then manually translate those architectural decisions into code. Similarly, testers must pore through requirements and design documents to create test cases that best suit their needs.
To create real flow, the analyst, the architect, the developer, and the tester should be using work products that connect their work. In short, they should be building digital assets that flow between them rather than producing somewhat useful documents that must be manually transferred into each team's required activity, preferred method, and tool of choice.
Another Lean concept that promotes flow is heijunka, which refers to the leveling of production or averaging of both your demand volume and product mix to smooth out production. In traditional manufacturing shops, the amount of production is driven by swings in demand. One day, the company may be responding to an order for 100 Model XYX cars, and the next day, demand shoots up to 400. This change ripples through the supply chain, making it difficult for suppliers to provide the required parts and causing problems forecasting labor needs. With the Lean approach, the firm strives to manage the flow of orders to match a near constant level of production.
In software, level production is accomplished by selecting an even amount of work that can be accomplished in a single iteration. The project manager and the architect select an attainable number of use-case scenarios that can be detailed, designed, coded, and tested within the scope of the iteration. An iteration usually takes six-to-eight weeks depending on a number of factors. Working in this manner creates a rhythm that keeps the team from working at a fluctuating, disrupted pace.
The authors of Lean Thinking suggest that the concept of pull is about simply ensuring that nothing in a value chain is produced upstream until the downstream customer has asked for it (i.e., the downstream request serves to "pull" production from upstream resources). As with Flow, there are a couple of key dimensions of this within the realm of software.
First, project managers need to make sure what is being delivered to the customer is of real value. According to the widely read Standish CHAOS report, more than 65% of software features are either never or rarely used.12 These features are elaborated into lower-level requirements, then developed and tested, yet they deliver little or no value. Why do these ever get off the drawing board? There are several common reasons. Poor collaboration between the end user and product management contributes to a shotgun approach to development. We brainstorm ideas, fire them off to development, and launch them into the market hoping users will derive value from the new functionality. How software projects are funded is another factor. A "big bang" approach that attempts to solve too many problems at once makes it difficult to pull the real value through the development process.13
The next dimension of pull has to do with how software development teams collaborate to deliver functionality. As explained above, development teams often work to deliver batches of artifacts. Analysts produce bulk requirements in the form of a specification delivered to development. Development delivers a completed build for those requirements to testing. Each member of the team spends significant time waiting on some upstream decisions to be made or work products to be delivered. That wait time is waste. Iterative development supports pull and reduces waste.
My colleague Murray Cantor points out in a recent article14 in The Rational Edge, that "At the outset of a project, nothing is known for sure. For example, the cost, effort, and duration of a project are at best educated guesses." This includes the requirement that will ultimately satisfy the business needs and the right design that will match the needs. Conventional wisdom tells us to try and get all the requirements in place first and then make design and production decisions. But forcing decisions about requirements that are not knowable at the outset of the project sets up the project for inevitable scope creep. The project team spends significant time responding to changes, reworking requirements documents, test cases, and even code.
The Lean approach to combat this malady is to "decide as late as possible."15 Put off decisions until the last reasonable moment. Let's look at an example: In the 1980s the cost of making a design change at any of the Big Three auto manufacturers in Detroit was 30-50% of the total die cost. In Japan it was 10-20%. To get this result, Japanese tool and die makers actually started roughing out the die at the same time the car was being designed, not beforehand. There was a constant collaboration between the design engineers and the manufacturing engineers. You do not get the same results if you attempt to rigorously control changes too early or attempt to get it right the first time -- in software development, too much can change.
Frequently, there is significant waste because the design "doesn't fit the needs of the next specialist down the line."16 In manufacturing, this results in problems, like the radio not perfectly fitting the dash. At Toyota, the design and manufacturing teams combat this by rigorously looking for technical risks in their designs. They then build in tolerances for their design specifications. This lays out acceptable variance and gives the downstream team some flexibility.
In software development, overplanning leads to requirements that are often not cost effective to implement given technical constraints. For example: "We can't deliver real-time data because the backend is a batch system." True collaborative engineering would suggest that joint development of some of the requirements and some of the design would approach a solution that can be economically delivered.
Let's continue with our car analogy. The traditional approach to developing software requirements is the equivalent of asking customers at the outset of development what color car they want, what kind of stereo should be in it, what kind of tires need to be on it, and what's the standard air pressure for those tires, etc., etc. All of these are questions that must be answered -- but not all at once and not all at the beginning of the design process. That leaves very little flexibility for change.
A preferred approach is to first understand the purpose of the car. Is it simply to get back and forth from work? Or is it to haul ten kids to soccer practice? The answer to that question must be known early because of the implications for the "architecture" of the car.
Just as tighter integration between design and manufacturing engineers yields results for Japanese car manufacturers, the IBM Rational approach to development centers around the idea that tighter collaboration between the end user and the development team will result in software that best meets the real needs of the consumer. Murray Cantor suggests that the collaboration yields "useful team knowledge" that directly and systematically reduces the amount of technical and performance risk a project faces. He mathematically asserts that this tighter collaboration creates knowledge that can be reused to create new knowledge. In other words, value is transferred from one team member to another in its more pure state.17 This approach manifests itself in the rigorous implementation of use cases as a method for facilitating this collaboration. Use cases and user stories are methods for writing requirements for a software system from the perspective of the customer or end user of the system.
Executing Lean software delivery
What does it take to implement Lean software delivery? Here, I have set forth a list of priorities to transform your organization, and discuss the supporting IBM Rational technology as appropriate.
Regarding my proposed list of priorities, it is important to note that the order in which they will be implemented varies with different organizations. For instance, every development group need not completely establish pull among all stakeholders before beginning to provide transparency.
The first step in instituting Lean in your organization is to establish pull from your end users and key stakeholders. At an organizational level, that means selecting the right mix of projects to execute. At a project level, it means making sure you select the right mix of features to yield real value. Lastly, it means establishing a high level of collaboration that ensures upstream team members are not producing intermediate work products that downstream team members do not really need.
Select the right projects
Regardless of whether the software you deliver is for commercial use or you work in an IT development shop, strategic alignment of the software you develop is essential. Selecting the right software projects that will deliver the most revenue, reduce cost the most, or make customers the happiest is the ultimate objective. This strategic alignment is a primary goal of good governance practices. IBM Rational Portfolio Manager® (see Figure 3) supports these practices by helping to automate the project proposal and approval process in a manner that facilitates using objective criteria to find the right mix of projects that will deliver the most value to stakeholders.
Figure 3: IBM Rational Portfolio Manager
Connecting business value to software functionality
Once the right projects are selected, you have to find the right functionality for a product or solution that will deliver the most value. This is what software engineering refers to as requirements management. IBM Rational's approach to managing software requirements seeks to reduce waste by reducing the production of documentation that is not useful. This is accomplished in several ways.
First, the requirements must be written in a non-technical way so that customers and non-technical stakeholders can understand them. A very common problem in software development occurs when customers sign off on technical requirements specifications that they do not understand because they are, in essence, told, "If you don't sign this, we can't move forward with development." To keep the project on track (or at least support the illusion that it's on track), they sign. This breeds distrust, and worse, often delivers a solution that does not yield value. The IBM Rational alternative is to embrace requirements written from the user's perspective, not the system's perspective. These user-centric requirements take the form of use cases. The use cases drive the project. They are used to plan the work, derive the system architecture, and jump-start the test cases.
Second, we need to ensure that technical requirements can be connected back to the value they drive, in the form of use cases. This is called requirements traceability. IBM Rational's software requirements management product, IBM Rational RequisitePro® (see Figure 4), allows the product team to automate traceability. This automation makes it significantly easier for the team to spot overproduction, often in the form of creeping functionality.
Figure 4: A traceability matrix in IBM Rational RequisitePro
Priority 2: Create value flow with iterations and digital models
Once the organization has implemented some level of pull, and thus improved its customer value by selecting the right projects and the project-specific value using use cases, it's time to make sure that value flows through the delivery cycle.
As was suggested earlier, in software delivery, flow is created by developing software iteratively. The IBM Rational Unified Process prescribes a framework within which an organization can execute iterative software development projects. As pointed out above, in order to maximize the value of the deliverables that are handed off between roles, teams should deliver intermediate work products that can best be used by the downstream role; that is, digital models. The Unified Modeling Language (UML) provides a standard notation for those interactions. The RUP prescribes a method by which that notation can be used to deliver digital assets for assembly.
The IBM Rational Software Development platform is built on top of the open source Eclipse integrated development environment (IDE). This platform provides the technology that facilitates the passing of digital assets from one role to the next.
Connecting the concepts of pull and flow with iterative development is depicted in Figure 5. Requirements are pulled from the users and written from their perspective. Those requirements are selected for implementation in a single iteration based on the value their implementation delivers. This downstream pull feeds the "assembly line." When the right requirements are delivered just in time to the delivery team and that team builds digital assets rather than paper based specifications, we enable true flow.
Figure 5: Supporting pull and flow with iterative development
Priority 3: Reduce waste with collaboration and automation
Adopting Lean practices requires a different way of collaborating between your clients and stakeholders. In order to institutionalize the concepts of Lean, especially flow and pull, it is necessary to have tight collaboration amongst the stakeholders. To enable this collaboration, the organization should implement a platform that supports value development rather than creating more overhead and waste. Just as Lean manufacturers seek to eliminate non-value-add activities in their manufacturing lines through deploying the right equipment, software engineering teams need the equivalent infrastructure. The IBM Rational Software Development Platform is integrated so that key project information is made available automatically to the various team members. For example, the analyst does not have to explicitly tell the tester that a requirement has changed -- the platform provides this notification. An integrated delivery platform helps you eliminate non-value-add activities, and that's the name of the game if you hope to achieve flow.
Priority 4: Provide transparency to the software delivery process
Visual control over the production process is a requirement for Lean. IBM Rational provides this transparent visibility by allowing management to look at the technical and performance metrics for a project currently under development or maintenance. Using the IBM Rational Portfolio Manager® (see Figure 6), executives, program managers, and project managers can spend less time producing manual status reports by quickly accessing real-time data.
Figure 6: An IBM Rational Portfolio Manager dashboard showing schedule, budget, and scope information
Priority 5: Speed "inventory" cycles
As discussed earlier, software does not have an actual concept of inventory, but the builds that are the physical manifestation of the software are the closest thing we have. IBM Rational Build Forge® automates the process of completing a software build and deploying that build into integration and testing environments. This automation has enabled clients to go from ten to fifteen builds in a week to well over three hundred. That decrease in build cycle time means that defects are found and fixed earlier in the development cycle. This means an elimination of several types of waste.
Lean Thinking can have a tremendous impact on the speed and quality of the software your organization delivers. It requires a shift from the traditional (i.e., "waterfall") mindset about development productivity, but the results of its application in other domains speak for themselves. The IBM Rational Unified Process (RUP) and the IBM Rational Software Development Platform provide the technology your organization can use to actually begin executing Lean Thinking -- thereby moving from theory to practice.
Special thanks to Murray Cantor, Jesse Tilly, Kurt Bittner, and Rachael Rusting for their thoughts and recommendations on this article.
1 M. Duvall, in Baseline (http://www.baseline.com), September 2006.
2 S. B. Donnely, "Lean and Mean." Time Magazine, August 2006.
3 M.A. Poppendieck and T. Poppendieck, Lean Software Development: An Agile Toolkit. Addison Wesley, Boston, 2003.
4 J.P. Jones, Lean Thinking. New York Free Press 2003.
5 Poppendieck, 2003.
6 B. Boehm, 2005 Some Future Trends and Implications for Systems and Software Engineering Processes. Wiley Interscience, Hoboken, New Jersey, 2005, p. 2-3.
7 Poppendieck, 2003.
8 J. Womack, Lean Enterprise Institute. From www.lean.org, September 2006 (http://www.lean.org/WhoWeAre/LEINewsStory.cfm?NewsArticleId=38).
9 Poppendieck, 2003.
10 Duvall, 2006.
11 OMG Model Driven Architecture. Object Management Group, from http://www.omg.org/mda/, September 2006.
12 The Standish Group, CHAOS Report. 1994 (http://www.standishgroup.com/sample_research/chaos_1994_1.php).
13 M. Lines, "Effective Governance Practices for Iterative Development." The Rational Edge, February 2005, pp. 1-7.
14 See Murray Cantor, "Estimation variance and governance." The Rational Edge, March 2006, at http://www.ibm.com/developerworks/rational/library/mar06/cantor/index.html
15 Poppendieck, 2003.
16 Jones, 2003.
17 Murray Cantor, Op. cit.
Comments (Undergoing maintenance)






