 | Level: Intermediate Pal Krogdahl (pal.krogdahl@se.ibm.com), Senior IT Architect, IBM
Gottrried Luef (gottfried_luef@at.ibm.com), Senior IT Architect, IBM
Christoph Steindl (christoph_steindl@at.ibm.com), Senior IT Architect, IBM
26 Jul 2005 Part 1 of this two-part series takes a look at the fundamental principles of agile development encapsulated within recognized methodologies like Scrum, Extreme Programming (XP), Crystal, Dynamic Systems Development Method (DSDM), and others that focus on Lean Software Development (LSD). Today, businesses have to respond with flexibility and speed to ever-changing customer demand, market opportunities, and external threats. For these businesses to be successful, development of a Service-Oriented Architecture (SOA) is a way to architect IT systems with the goal of adaptability -- the ability to respond to changing and new requirements. Part 2 of this series details their suitability for developing a SOA. Introduction
Service-Oriented Architecture (SOA), as defined and documented by respected thought leaders like Mark Colan (see "Service-Oriented Architecture expands the vision of Web services, Part 1"), has gained considerable ground as a recognized emerging style for architecting adaptable, enterprise-wide IT systems. Although the goal is clear, the way to reach the goal is not, since there is no broadly accepted SOA methodology. However, there are some contributions (see Methodology for SOA).
Everyone who is responsible for architecting a SOA faces two main challenges: 1) How can I design a system such that there is a good fit between business processes, business goals, and IT architecture, and 2) How can I build an architecture that is able to respond to future changes? The second challenge deals with agility and is generally being discussed in the context of agile software development methods. In this article, we show you how to extend the ideas of such methods to the design of a SOA. We will first review some of the most common agile development methods, before taking a closer look at the principles of "Lean Software Development" (LSD). Lastly, we discuss an initial analysis of LSD within the process of building a SOA.
SOA: A primer
What is SOA?
IT architectures, especially enterprise-wide architectures, were always constructed with the goal to support the business processes and to respond to changes in the business. In the last few years, new approaches to system architecture emerged, which aim to structure a complex system around units of capability, called services.
In the most recent understanding, there are four basic aspects around a service:
- Provision
- Consumption
- Description
- Brokerage
A systematic and standards-based support of these aspects occurs with Web services. Therefore, Web services have an unbeatable agility advantage. For instance, Web services infrastructures allow for changing the service provider at runtime without affecting the consumers.
A system that calls itself SOA-based should have the following properties:
- Business processes are mapped to software services; therefore, business can be tracked by the software.
- There is an infrastructure in place that supports the four different service aspects given above. This enables a high degree of agility on the service level.
- Services are the units of monitoring and management. Therefore, one can track operational properties and problems to business processes.
SOA applications
Figure 1 shows the elements of an enterprise-wide SOA from the perspective of applications. Business processes are partly or fully supported by user interface applications and service applications. A step in a business process is either manual or is supported by a user interface application. User interface applications implement a lot of micro workflow, and they also consume services that implement business functionality.
In the service choreography layer, composite services are defined by means of a choreography language, such as BPEL (Business Process Execution Language). The choreography of composite services defines their flow and composition from elemental services. The choreography layer should be supported by a choreography tool that allows graphical specification, such as IBM WebSphere® Business Integration Modeler and IBM Rational® Application Developer.
The elemental services, used by the service choreography layer and also by user interface applications, are implemented by service applications. In turn, the service implementations might call other services, often from other service applications.
Figure 1. Elements of a SOA, adapted from the CBDI Service Oriented Architecture Practice Portal
Figure 1 was adapted from "Understanding SOA" by David Sprott and Lawrence Wilkes, on the CBDI Service Oriented Architecture Practice Portal.
Methodology for SOA
What should a development method look like that leads to a sound SOA? As can be seen in the previous section, there are business processes, applications, and services. It is obvious that modeling the services is a main task that such a method must support. Another important aspect is to ensure the link between business processes and services (see What is SOA).
The article "Elements of Service-Oriented Analysis and Design" shows how existing models like Object-Oriented Analysis and Design (OOAD), enterprise architecture frameworks, and business process modeling techniques contribute to SOA design. The article also states that you need additional method elements for a SOA, such as methods and techniques for service identification and aggregation, business traceability, integration of existing assets, and reuse.
In another IBM developerWorks article, "Service-oriented modeling and architecture", a method is described that offers answers for many of the above questions. The article concentrates on the modeling of services, which is done by techniques such as domain decomposition, existing systems analysis, and goal services modeling.
Both of the cited contributions bring up a lot of questions that we still need to answer -- for instance, the question of SOA governance. Another question we want to pose is: What rules and practices do you need to incorporate in a SOA development to ensure that the service model is able to respond to future changes? Here we can resort to the various agile software development methods.
Agile software development
Agile software development started to take off during the late nineties when Kent Beck promoted Extreme Programming (XP) with a set of values, principles, and practices for planning, coding, designing, and testing software. (For an introduction to XP, see Extreme Programming: A gentle introduction.)
All agile software development approaches have several values in common, such as:
- Frequent inspection and adaptation
- Frequent delivery
- Collaboration and close communication
- Reflective improvement
- Emergence of requirements (incremental), technology, and team capabilities
- Empowerment and self-organization
- Dealing with reality, not artifacts
- Courage and respect
From these values, the various agile methods in use today emphasize different practices.
In February 2001, the Agile Manifesto was defined, valuing individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. It is the foundation of all agile methods in use today.
For the benefit of this article, we briefly describe the most commonly used agile methods, since many of them are useful for developing a SOA. We are aware that SOA is about more than software development -- it is about business and IT architecture as well. Therefore, if we look at software development practices, we always have to assess their suitability for SOA. We accomplish this assessment in the second part of this article series.
Scrum
Scrum appears simple, yet has practices that deeply influence the work experience and capture key adaptive and agile qualities. Scrum's distinctive emphasis among the methods is its strong promotion of self-directed teams, daily team measurement, and avoidance of prescriptive process. Some key practices of Scrum include:
- Self-directed and self-organizing team
- Daily stand-up meeting with special questions (what have you done, what will you do, and what problems do you have)
- Usually 30-day calendar iterations
- At the beginning of each iteration, client-driven adaptive planning
- Demo functionality to the stakeholders (at the end of each iteration)
For enterprise-wide engagements, it is important to understand and manage the inter-project dependencies. This is well-covered in Scrum with a "global backlog", which is an enterprise view of the user-valued functional and non-functional requirements. This global backlog is globally prioritized. Each project takes from that global backlog the most important parts that fall into the scope of the project. Inter-project synchronization is also covered with a "Scrum of Scrums", a bi-daily (or weekly) meeting with representatives from each team in order to synchronize across teams.
XP
XP (http://www.extremeprogramming.org and http://www.xprogramming.com) emphasizes collaboration, quick and early software creation, and skilful development practices. It consists of a couple of primary practices (sit together, whole team, informative workspace, pair programming, weekly iterations, slack, ten-minute build, continuous integration, test-first programming, incremental design, and so forth) and a couple of corollary practices (real customer involvement, incremental and daily deployment, root-cause analysis, shared code, single code base, negotiated scope contract, pay-per-use, and so forth).
Crystal
Crystal is a family of methodologies with a common genetic code that emphasizes frequent delivery, close communication, and reflective improvement. Crystal's genetic code is made up of:
The priorities common to the Crystal family are safety in the project outcome, efficiency in development, and habitability of the conventions (in other words, the developers can live with them). The project team steers toward seven safety properties (the first three are core to Crystal and the others can be added in any order to increase the safety margin):
- Frequent delivery
- Reflective improvement
- Close communication; personal safety (the first step in trust)
- Focus
- Easy access to expert users
- Technical environment with automated testing
- Configuration management
- Frequent integration
Dynamic Systems Development Method
The Dynamic Systems Development Method (DSDM) provides a framework of controls for building and maintaining systems, which meet tight time constraints and provides a recipe for repeatable Rapid Application Development (RAD) success. The method not only addresses the developer's view of RAD, but also that of all the other parties who are interested in effective system development, including the users, project managers, and quality assurance personnel. Listed below are the governing principles of DSDM:
- Active user involvement is imperative.
- DSDM teams must be empowered to make decisions.
- The focus is on frequent delivery of products.
- Fitness for business purpose is the essential criterion for acceptance of deliverables.
- Iterative and incremental development is necessary to converge on an accurate business solution.
- All changes during development are reversible.
- Requirements are baselined at a high level and prioritized (to gain scope flexibility with low priority items).
- Testing is integrated throughout the life cycle.
- A collaborative and cooperative approach between all stakeholders is essential.
Lean manufacturing
The manufacturing industry has realized that there are two kinds of manufacturing problems -- "predictable manufacturing"and "new product development". The former is characterized by the ability to know requirements with surety in advance, thereby making deterministic scheduling possible. The latter is characterized by the inability to nail down requirements up front, making it necessary to continually adapt and re-estimate as a project progresses.
Craig Larman compares these problems in his book Agile and Iterative Development with the following results:
Table 1. Comparison of predictable manufacturing and new product development
|
Predictable manufacturing
|
New product development
| | It is possible to first complete specifications, and then build. | It is rarely possible to create upfront, unchanging, and detailed specifications. | | Near the start, one can reliably estimate effort and cost. | This is impossible near the beginning. As empirical data emerge, it becomes increasingly possible to plan and estimate. | | It is possible to identify, define, schedule, and order all the detailed activities. | Not possible near the beginning. Adaptive steps driven by build-feedback cycles are required. | | Adaptation to unpredictable change is not the norm, and change rates are relatively low. | Creative adaptation to unpredictable change is the norm. Change rates are high. |
As an example, Toyota started in the 1980's to revolutionize the automobile industry with their approach of "Lean Manufacturing", with the intent to eliminate waste, to streamline the value chain (even across enterprises), to produce on request (just in time), and to focus on the people who add value.
LSD
Mary and Tom Poppendieck have transferred the principles and practices from the manufacturing environment to the software development environment (see their Web site for more information) with the goals to identify and eliminate waste through continuous improvement, and to reduce defects and cycle time while delivering a steady stream of incremental business value. The foundations of LSD lie within the standards set for Lean Manufacturing by organizations such as Toyota, Dell, and Wal-Mart.
One key point to note is that LSD is not a development method per se, such as XP, DSDM, SCRUM, and others, but rather offers a number of fundamental principles which should be applied to improve software development, regardless of the development or project method that is used. In the remainder of this article, we quickly review the seven principles and some of the tools (for example, ideas written in italics in the following text) of LSD.
Principle 1: Eliminate waste
Eliminating waste is the most fundamental lean principle. It is the one from which all the other principles follow. The first step to implementing lean development is learning to See Waste (anything that does not improve the quality of code, does not reduce the amount of time and effort it takes to produce code, or does not deliver business value to the customer). The second step is to uncover the biggest sources of waste and eliminate them. A large study showed that up to 45 percent of features defined in early specifications might never be used after the solution is completed and delivered.
Principle 2. Amplify learning
When an organization faces software development challenges, there is a tendency to impose a more disciplined process on the organization with more rigorous sequential processing. Just as control theory predicts, this generally makes a bad situation worse. When a problem develops, the first thing to do is to make sure the feedback loops are all in place. The next thing to do is to increase the frequency of the feedback loops in the problem areas, since short feedback loops enhance control.
Whenever several individuals work on the same thing, a need for "synchronization" occurs. The need for synchronization is fundamental to any complex development process. "Iterations" are points of synchronization (the team and the customer see what has been accomplished) and force decisions to be made.
Principle 3. Decide as late as possible
Establishing requirements before development begins is commonly thought of as a way to protect against serious errors. The problem with sequential development (for example, waterfall) is that it forces designers to take a depth-first rather than a breadth-first approach. But the easiest way to make big mistakes is to drill down to detail too fast.
With concurrent development (for example, all work is done within the iteration -- analysis, design, programming, and testing), you start programming the highest value features as soon as a high-level conceptual design is determined, even while detailed requirements are being investigated. That exploratory approach permits you to learn by trying a variety of options before you lock in on a direction that constrains implementation of less important features ("Options Thinking"). But concurrent development requires developers with enough expertise in the domain to anticipate where the emerging design is likely to lead, and close collaboration with the customers and analysts who are designing how the system solves the business problem at hand.
Principle 4. Deliver as fast as possible
Customers like rapid delivery, which often translates to increased business flexibility. Rapid delivery is an options-friendly approach to software development. It lets you keep your options open until you have reduced uncertainty and can make more informed, fact-based decisions.
It must be clear to every person, at all times, what she or he should do to make the most effective contribution to the business. You can either tell them what to do ("command and control") or set things up so they can figure it out for themselves ("self-organization"). In a fast-moving environment, only the second option works. For effective self-organization, methods for local signaling and commitment (for example, using information radiators) must be developed to coordinate work in the form of Pull Systems. No schedule can make effective and fine-grained work assignments in a complex environment with modest variability, anyway. Creating detailed plans early means carving some decisions into stone. Plans and predictions are not bad, but making irrevocable decisions based on speculation is to be avoided.
The benefits of rapid development are usually larger than you might expect. Create a simple economic model of a new product for the next few years, basically a profit and loss statement (P&L), and use that economic model to drive development decisions. Get from marketing good estimates about what delay will do to sales volumes (for example, loss of early high pricing) and market share (for example, long-term loss of market share). The model shows what the difference in revenue and market share will do to profits.
Principle 5. Empower the team
All too often, improvement programs do not change the reality of how work gets done. They increase the intensity of factors leading to job dissatisfaction (policy, supervision, and administration) instead of increasing factors that contribute to job satisfaction (achievement, recognition, and responsibility). Lean thinking capitalizes on the intelligence of frontline workers, believing that they are the ones who should determine and continually improve the way they do their jobs. While software development cannot be successful without disciplined, motivated people, experimentation and feedback are more effective than trying to get things right the first time. Principle 6. Build integrity in
A product has conceptual integrity if its components match and work well together; the architecture achieves an effective balance between:
- Flexibility
- Maintainability
- Efficiency
- Responsiveness
To achieve conceptual integrity, deemphasize control mechanisms in favor of:
- Face-to-face discussions
- Small batches
- Speed
- Flow
Instead of trying to predict the future trends from the beginning, take a breadth-first approach and get the basics right. Then, let the details emerge and plan on regular refactoring to keep the architecture healthy. Refactoring means to stop the line (for example, stop adding new features) when you detect a "smell", and take the time to find and fix the root cause of the problem before proceeding with more development.
Refactoring is only possible if there is a tight safety net of tests (testing). Tests should be automated as much as possible and run as part of the daily or continuous build. Maintain a set of comprehensive tests throughout the life cycle of the system. Then, the system can be safely repaired and refactored throughout its useful life. If there is not enough time for testing, the first thing to do is reallocate the effort used in requirements documentation to writing customer tests.
Principle 7. See the whole
The more complex a system, the more temptation there is to divide it into parts and manage the parts locally. Local management tends to create local measurements of performance. These local measurements often create system-wide effects that decrease overall performance (sub-optimization). Lance Armstrong won the Tour de France each year from 1999 to 2005, yet he won only a few of the daily stages. Just like the Tour de France, optimizing every task is often a very bad strategy.
It is very difficult to measure everything that is important with knowledge work, especially when each effort is unique and uncertainty reigns. If you cannot measure everything that is important, partial measurements are very likely to turn into local management. If you can't measure everything that is necessary to optimize the overall business goal, then you are better off without the sub-optimizing partial measurements.
From a project manager’s perspective, there are four variables that one can adjust when managing projects: time, costs, quality, and scope. From these four variables, fix time, cost, and quality -- but not scope. Prioritize features, but don't specify in the contract the fixed set of features to be delivered. Move from a fixed scope to a negotiable scope: By delivering high priority features first, it is likely that you deliver most of the business value long before the customer's wish list is completed.
Agile architecture
No big design up front
Software development methods have different approaches to developing the architecture of a project: Rational Unified Process (RUP) addresses architectural risks early ("If you don't actively attack the risks, they will actively attack you"), whereas XP calls for a "no big design up front". Scrum defines just enough of the architecture up front, but then delivers the architecture incrementally -- prioritized in the same way as the remaining functionality.
From an agile point of view, it is not advisable to understand all the requirements at the beginning of a project, analyze them, and then develop the architecture for the whole system like in a waterfall model. It is important to balance the need for early architectural stability and the reality of change. Of course, it is costly to revise important architectural decisions again and again. On the other hand, if the decisions are bad and don't fit to the changed requirements, it is better to change the decisions earlier, rather than later.
Validation instead of verification
Whatever approach you use to develop an architecture, it is necessary to validate the architecture against the users' expectations (and not against the requirements). This kind of validation can only be done by developing a part of the solution ("walking skeleton", "spanning application", or "tracer bullet") to demonstrate that the architecture is viable. The architectural skeleton is then incrementally fleshed out with details.
When to do architecture work
Traditionally, it has been a decision of the development team as to how much architectural work is done in early project phases. The approach of some agile development methods, such as XP and Scrum, is to put functional and non-functional requirements on a common backlog, and then let the customer prioritize the requirements. While the customer prioritizes the requirements for business value and criticality, the development team estimates the effort and cost, and brings in the expertise where the architectural risks lie. Thus, the architecture is developed incrementally with the requirements.
Recently, the decision about when to do architecture work has been put on a sound basis where the decision is not just a technical decision any more, but becomes a business and financial one as well. The Incremental Funding Method breaks down the requirements into Minimum Marketable Features (MMF) and the architecture into Architecture Elements (AE). Based on heuristics, the MMFs and AEs are sequenced to optimize various financial targets (for example, maximized ROI, minimized cash injection, earlier payback, and so forth). The client can understand and evaluate the various architectural options and their financial implications. The development team understands the financial aspects of developing all the architecture up front or delivering the architecture incrementally.
Traditional and early agile approaches both took a "greedy approach": Traditional approaches attack the riskiest parts first and early agile approaches attack the most valuable parts first. IFM allows the customer and development team to select the approach that is most rewarding.
Continuous refactoring of services
In the context of SOA, one application is never seen as a standalone application. As we stated in SOA applications, there are various users of service applications, so there are various inter-application dependencies, clearly described as service dependencies in the service model. Any service should be considered for potential reuse by other applications. Thus, service applications must be seen as reusable products. In effect, this means that agility definitely becomes more important, because the number of consumers to the applications increases. There are not only the end users, but also the other applications that become consumers/customers to an application. As the number of customers increases, the number of changes will certainly also increase, since not all uses can be foreseen.
There is one central aspect in a SOA that drives agile development: The service model -- the model of services, their dependencies, choreography, and flows. As a first-order deliverable, the service model evolves over time as well as the definition of the service interfaces. Companies will recognize that the current version of their service model has weaknesses because responsibilities of the services are not defined the right way. They will have to shift responsibilities from one service (or one version of a service) to another one, and change services interfaces. The refactoring of the service model cannot be avoided, and as in agile software development, we encourage such continuous refactoring.
Refactoring services means changing interfaces by shifting responsibilities from one service to another. This can be done in a controlled way using the services model. The model of services becomes an essential tool for agile development in a SOA, because without it, it is very hard to control service refactoring. As we have the service model, we are ready to extend agile software development to the SOA level!
There is another piece of good news on the agile SOA front: Agility also becomes easier to manage. You can capture a change in business processes by changing the service choreography of composite services. A change in the choreography facilitates more easily than a change in the implementation of an elemental service.
Conclusion
In this first part of our two-part series, we introduced the concepts of SOA, agile software development, and LSD, and we tried to sketch out how the agile or lean principles and practices affect architecture, including service architecture.
The second part of this series attempts to mix each of the lean software principles with SOA development. This mixture, together with the chapter Agile architecture in this first part, results in a foundation for agile SOA development.
Acknowledgements
The authors would like to acknowledge the valued input and feedback of Olaf Zimmermann, Rick Robinson, Keith Jones, and Norbert Bieberstein.
Resources
About the authors | | | Pål Krogdahl is a Senior IT Architect and Method Exponent with the IBM EMEA WebSphere Lab Service team. He has been working for IBM since 1998 in various areas such as software development, technical pre-sales consulting, and solution architecture. His areas of expertise are in distributed computing, middleware, and Application Services Architecture, with focus on Enterprise Application Integration (EAI) and SOA. Pål has recently written extensively on the subject of SOA-based EAs, and their relationship to IBM patterns for e-business and the On Demand Operating Environment. To date, he has completed several residencies with the IBM International Technical Support Organization, where he has co-authored IBM WebSphere-related RedBooks with a focus on Web services and SOAs such as Patterns: Service-Oriented Architecture and Web Services (ISBN 073845317X). A large majority of his project experience has been predominantly within a Rapid Application Development environment. You can reach him at pal.krogdahl@se.ibm.com. |
| | | Gottfried Luef is currently working as an IT Architect and Consultant at IBM in Vienna, Austria. He supports various projects in the government area with expertise in Web services, SOA, and J2EE architecture. He participated in the development of Austrian government standards for Web services. Gottfried holds a doctoral degree in Information Management from the University of Vienna, Austria. You can contact him at gottfried_luef@at.ibm.com. |
| | | Christoph Steindl is a Senior IT Architect and Method Exponent at IBM in Vienna, Austria. He has been working for IBM since 2000 on a variety of software development and systems integration projects. His areas of expertise are in application development, software engineering, and methodologies. He is very knowledgeable about various agile methodologies and has given presentations about topics like LSD, agile project management with Scrum, supporting XP with Scrum, distributed agile, test-driven development, practical use-case modeling, and estimation in agile projects. He is an appointed lecturer at the Johnannes Kepler University in Linz (Austria) and at the University of Applied Sciences in Hagenberg (Austria), as well as a Certified ScrumMaster. He holds degrees for Computer Science and Mechatronics and a doctoral degree in technical sciences from the Johannes Kepler University in Linz. You can contact him at christoph_steindl@at.ibm.com. |
Rate this page
| |
