Kerrie Holley, IBM Fellow, is responsible for an IBM World Wide SOA Center of Excellence. Kerrie has a B.A. in mathematics and a Juris Doctorate from Depaul University. Outside of work, Kerrie enjoys spending time with his beautiful wife, two boys and a weekly basketball game with other dads.

SOA Failures

I was discussing the notion most recently with some clients who stated that SOA has failed and that many of their SOA deployments were a failure. I probed to better understand the nature of the failure and could not reach any conclusions about the specifics. It’s rare to hear someone say that the remodeling of my house was a failure. In fact most people would say I had a bad contractor, the project was not done correctly, the project was under estimated or the remodel failed to meet my expectations. In fact one would cite any number of specific reasons but not say the remodel was a failure.

When one does a renovation of their house or a room in their house is the goal renovation or remodeling? I do not think that is the goal, the goal for remodeling the kitchen may be to enhance its look and feel, to introduce superior appliances or any number of specific measurable goals. But the homeowner would never describe the goal as remodeling.

In a like manner, SOA should never be the goal. SOA is a means to an end, an approach, a strategy, a way of doing things. So how can SOA be a failure given this context? What were business the goals or aspirations? Were the specific benefits of adopting SOA defined? Were they measurable? Or did the project assume benefits of SOA would magically come about because the project was classified as a SOA project. What makes the project a SOA project? Are all SOAs created equally? Do all SOAs bring about the same value? The answer is clearly no.

In fact it’s not correct or rigorous to describe or discuss SOA failures as architectures cannot fail, their application and means of adoption might fail. It reminds me of the discussions I have had with developers who describes Web services as slow. Really? Isn’t that relative? What makes a technology slow? Compared to what?

My message is we ought to as software engineers apply some rigor to our thinking about SOA. I will discuss more on this subject latter.

Are all SOAs created equally?

I was speaking with some folks in the US government and the question of whether was asked, all SOAs are the same? Do they all provide the same value? Are all SOAs created equal? I pondered the question because it is insightful and a question worth exploring.

All SOAs are not the same, they are not created equally and they do not provide the same value. That is, what one deployment describes as an SOA implementation may offer different value than another. For example, I may deploy SOA technology (e.g., an Enterprise Service Bus or a registry) to improve integration flexibility. In one such implementation I might cause IT costs to rise because of the inability to share the infrastructure whereas in another deployment the shared infrastructure was addressed as a benefit to be achieved. In this simple example we see that the two ESB implementations provide different business values in the areas of cost.

Achieving business process flexibility is different than providing integration flexibility. Hence, an SOA deployment might address and provide integration flexibility without providing business process flexibility.

Different business goals (e.g., time to market, lower development costs or lower IT Costs or business flexibility) have both different metrics and require different SOA deployments. That is, the people, process and technology changes are different based on which motivation and goal is sought.

All too often we use platitudes to describe business goals with a measurable understanding of what the goal actually means and how is it actually achieved. Flexibility is the poster child for this platitude thinking. But we can define flexibility, we can measure it and we can achieve it. SOA can make a huge difference.

SOA and Web Services

Recently in a discussion with a senior executive at a financial services company he commented that SOA did not deliver the expected benefits. In fact, he went further to say SOA was a failure in his organization. Fortunately, I had the opportunity to gain pretty detail insights into the company’s initiatives around SOA as well as an excellent understanding of their past deployments.

Upon review it became pretty clear that the organization had done significant adoption of Web services. In fact hundreds of Web services were in place without concern for usage, performance and coarseness. It was clear that the organization was equating Web services and SOA as one and the same.

One of the clear lessons learned from organizations gaining business value of SOA is their understanding the difference between SOA and Web services; more on the differences in a latter blog.

SOA and Reuse

The highly touted benefit of reuse was discussed with several clients today as a benefit of SOA. So I asked the question of who was are target? Who are we trying to get to reuse things and why? Of course the response is we want programmers and developers to reuse code. More on this point in a moment. Another point was quickly raised as to whether reuse has anything to do with service orientation?

Of course a lot depends on how you define and see SOA and service orientation. That is, do you see SOA as purely an architectural style, is SOA the implementation of Web services, is SOA largely and solely for integration or is SOA about flexibility and adaptability to allow for rapid changes in the business whether it be business process changes, business model changes or other? I think the answer lies in all of the above; however, if we focus our attention on only one item we lose perspective. That is, we minimize the value and benefits of service orientation or SOA.

SOA has a business dimension namely promoting business agility, it has an architecture dimension which is both pertinent to business architecture and IT architectures, it has an implementation view, and an operating view. Limiting our view to just one, limits our understanding of service orientation and of SOA. Resulting in less business value and loss opportunity.

But lets get back to the question at hand which is about reuse. SOA is not about getting developers or programmers to reuse code, whether it be a subroutine, an object, a component or a service. SOA is about getting the organization or line of business to reuse things. Its about getting the business to share and to reuse business functionality across business boundaries, betweeen lines of business, within lines of business and across the enterprise where it makes sense.

If we look at why we are trying to achieve reuse its not because of reuse its because we want faster time to market, lower risk in implementation, seamless view of information or lower costs or improved quality where using the same functionality in multiple scenarios, multiple consumers, multiple channels, multiple applications helps. For example, we can actually effectuate faster time to market if organizations start reusing things a lot more than if developers start reusing things. The vast amount of elapsed time which affects time to market is consumed in figuring out what to do, integration and testing, areas where the developer role has minimal participation in reducing calendar time.

So the point is SOA and service orientation can help organizations and business reuse more things in their business but it requires that we use the construct of a service, not just a facade, not just Web services, but a service which represents a key activity or step of a business process. A service which is used as the structuring element of an application. A service which has the same care and feeding as applications. A service which has its own testing life cycle as an application, a service which has its own published software architecture, a service which is an asset the the business and is treated as such.

SOA Lessons Learned

Understanding lessons learned from SOA initiatives and projects provides insights on how to get started, creating business value and avoiding pitfalls. For example, some obvious lessons I discussed with a federal government client today included: (1) leading from an IT perspective; (2)treating SOA as same old architecture; (3) big bang approach; (4) equating SOA and Web Services; (5) ignoring or treating SOA governance as an after thought; (6) failure to integrate SOA and Enterprise Architecture thinking and (7) treating SOA as the what and not the how.

Clearly IT leaders should lead, whether its SOA initiatives or not. However, there ought to be business value proposition associated with SOA adoption, whether its flexibility, integration, cost reduction, integration, reducing process cycle time, or others. Of course the key is to take platitudes like flexibility and actually define what is meant. This is a longer discussion but the point is be clear and specific about your business intent and don't let SOA become just another science project.

Treating SOA as what we have done in the past or saying we did this 15 years ago missed the point. Its burying our heads in the sand. Its important to separate the hype but its also important to understand what is actually different from DCE, CORBA and other past approaches.

Obviously risk is lowered by taking incremental steps versus big bang. Organizations must adopt in line with their current maturity.

Many companies have seen the folly of thinking they will get SOA benefits by just using Web services standards and associated technologies. In fact for many companies they are seeing performance issues with their architectural decision to use Web services choices and in others little to no reuse. The point is the choice of Web services is an architecture decision.

Increasingly organizations are seeing that SOA governance must be planned and implemented in order to both achieve project benefits from SOA but especially to see sustained SOA benefits.

SOA can be used to jump start failing or ineffective Enterprise Architecture programs as SOA can be one way to increase the convergence between business and IT. However, SOA and EA are complimentary and SOA should be included in EA not the other way.

Lastly, the goal should not be to implement SOA but rather companies, organizations ought to have a vision where SOA is the how, as where there is no vision, people perish and in this case where there is no vision of how SOA will bring actual value, initiatives stall and fail and SOA gets blamed.

The Need to Show Business Value for SOA

MDA and SOA

While in Moscow working with a client the topic of Model Driven Architecture (MDA) and SOA was discussed. The client's view is that MDA in many regards is more advanced than SOA. It is of course an interesting perspective but one I disagree. SOA is not in conflict nor does it compete with any software engineering best practices. Suggesting that one is more advanced than the other places a value judgment on the use of best practices which I don't think promotes their usage.

MDA supports model-driven engineering of software systems. and provides guidelines for structuring various specifications as models. MDA also promotes the use of a platform-independent model (PIM) using an appropriate domain-specific language which can be translated to one or more platform-specific models (PSMs).

SOA's contribution to software engineering is under hot debate and discussion but I assert one of its impacts is in the area of both how we capture business requirements and how those requirements are communicated from business to IT. The notion of a service requires that business no longer push functionality to IT which is housed in silo applications, the function oriented approach. Instead services become first class constructs where they represent an application has more granular reusable assets called services. This changes the software engineering approach in several ways. One notable way is in how many organizations do use cases which have a large amount of behavior of things that occur on the glass, the user interface.

The change in development approach is that the service would be identified prior to use case development and as part of either understanding the business process, its re-engineering or its optimization. Separate use cases would be developed to represent the business functionality needs of the service versus the user interface work flow and design. This will allow for improved separation of concerns down stream during design and development.

But the big point is that this is complimentary to MDA as SOA builds or adds to best practices it does not conflict.

How to get started?

OKAY, I have been far too lazy in reporting back various experiences with clients as I discuss SOA. Very recently on a cold day and snowy in Michigan I met with a client who was singularly focused on what to do next. As I would mention motivations and rationales, he would politely communicate that we are convinced on the value and benefits of SOA we simply want to know how to get started or stated differently what have successful clients done to get stated.

So armed with that I articulated that there are different entry points and ways to get started but which one you select often depends on what will work in your culture (that is, your maturity in various aspects of software engineering), your appetite for change, funding models and of equal importance your business goals or motivations.

So I explained the six motivations should be understood so that we can determine the business value desired. That is, it is ill advised to pursue SOA as an end goal instead we pursue SOA as a software engineering best practice, as part of an enterprise architecture approach, part of an IT strategy or as part of a transformational effort (more on this in the future).

Six motivations must be understood because each will cause a different set of actions:

1. Flexibility
2. Improved business process (e.g., cycle times, optimization, etc.)
3. Revenue increase
4. Cost reduction
5. Risk and compliance
6. Integration

So if flexibility is to be something other than a platitude versus a durable statement that has a business and IT impact then one must be specific as to how they will know they have achieved flexibility when it occurs. Does flexibility include reuse? When we look at the SOA solution stack which (see http://www.ibm.com/developerworks/library/ar-archtemp/) which describes various layers in any SOA solution, we see potential reuse at several stages: services sharing, infrastructure sharing, etc. If flexibility is a motivation then we are embarking on a journey, an endeavor that will not be complete with one project but something that will require a program. It requires we address IT and SOA governance.

Improving the business process that involves 3rd party suppliers is an often seen motivation. Typically this involves extranets and is focused on using Web services as both an enabling standard and technology.

For some companies we see a desire to take their legacy assets (e.g., telephony functionality or reservation system) and make this functionality (i.e., services) available for 3rd party application providers or other consumers who build composite applications or sell the services as a part of their capability, resulting in a new revenue stream for the service provider.

Cost reduction often cannot be fully achieved without reuse of services. So often we are looking at using shared services as a way of retiring applications thereby reducing the total cost of ownership of the portfolio. In many cases one can achieve this goal without SOA, but at some point SOA may be necessary as the approach for re-structuring parts of the application and IT portfolio.

Risk and compliance can sometimes translate to exposing services to expose aggregated data or the orchestration of services in a workflow to address compliance.

Integration is a primary motivator for many organizations looking to adopt SOA. Integration has its own obvious benefits but in addition, the choice of applying SOA to integration also inevitably requires the organization decide if this will be a technology implementation or a transformational implementation. Organizations choosing the technology implementation may falsely conclude that they will achieve SOA benefits of reuse simply with the implementation of the technology where in reality they have solely remedied an integration issue. Getting the infrastructure to be shared (i.e. reused) across multiple lines of business (i.e. consumers) requires addressing more than tecnology, organization change management issues become apparent as an example.

SOA is not binary, there are different types of implementation, solutions and motivations. If one does not clearly define the motivation then its quite likely the result will not be met and SOA benefits will become another of IT broken promises.

So how do organizations get started? Of course the devil is in the details, but we have seen five entry points:

1. Technology. Examples are where the organization adopts one or more Web services standards or a product in the context of a sandbox to introduce the technology into the organization. Often there is also a project which can take advantage of the technology. This could also be the introduction of information as a service. For example, the aggregation of data from multiple sources might be best achieved using a service.

2. Business process management. In this example, the focus could be on improving or optimizing or developing automation for one or more business processes.

3. Connectivity. Integrating services across multiple applications inside and outside the enterprise for one or more defined business objectives

4. IT transformation. In some cases organizations have a "burning platform" which means that the technology or the application in its current state will not be able to meet foreseen business requirements presently and in the foreseeable future. Or the problem could be that IT requires a major improvement in its application delivery capabilities or it could simply be that IT costs have spiraled to unacceptable levels dictating IT simplification, IT optimization and other changes. Within this entry point multiple business motivations may be at play. Organizations could be trying to do more for less where they then examine how to increase the amount of reuse.

5. Business transformation. This is a case where due to regulatory changes, a public embarrassment, lack of sticky customers, changes of business models or whatever the business motivation, there is a need to do business and It transformation in parallel. Often this focuses attention on specific business scenarios that require addressing whether its people centered or process centered.

In addition we then discussed several case studies and I introduced the notion of a maturity model for providing a route map, a guide.

What is Different about SOA than previous approaches

Just recently in some meeting with clients in Budapest several questions were raised about what is truly different about SOA than previous approaches to improve IT agility? The history of COBOL (Common Business Oriented Language) being touted as a means to improve business and IT linkage was raised along with innovations in object-oriented development and component based development.

Rather than discuss the merits or pros and cons of each of these technology innovations I chose instead to focus on what is actually different with SOA. I described five differences:

(1) The ubiquitous presence of standards is actually different. The use of XML and WSDL or derivatives is a major difference. In addition, the way in which vendors author standards has also improved.

(2) A focus on interoperability versus integration is a key difference between SOA and prior approaches. Using the metaphor of applications as islands, integration is about creating bridges between these lands masses; interoperability is more like being teleported as in Second Life.

(3) The linkage between business and IT is different in that SOA makes the assertion that all businesses have a business design which describes how the business operates such as its business processes, the services rendered and consumed, business goals and objectives, business rules and policies. The business design ought to be a tool to communicate requirements between business and IT. Hence the use of business process models and BPM, where both process models and services have a distinct and clear relationship is different than previous approaches. Promoting the best practice of documenting as much as possible of the business design with tooling rather than the more common approach of using word processing or drawing tools is also different.

(4) The focus on reuse is also different in that with SOA we are focused on improving the organizational productivity versus a focus on developers and programmer productivity.

(5) Software design is also different. See Don Ferguson‘s blog (http://www-03.ibm.com/developerworks/blogs/page/donferguson) which I have included herein: Web services are more language independent than previous approaches. For example, CORBA was very C-like and was awkward in Smalltalk. EJBs and Java are, by definition, focused on the Java type space. XML renders more naturally into multiple languages like C, Java, COBOL, etc. XML and Web services are less fragile and better accommodate change. For example, it is possible to add or reorder elements in an XML business object without necessarily breaking code using older versions. The same applies to WSDL. Most previous approaches like RPC or CORBA were often a distributed version of a runtime model with offsets into data structures or function tables. So, additions and reordering broke code. Previous approaches really had three data models and type spaces. For example, a CORBA application had: 1) IDL, 2) IIOP for inflight messages, 3) SQL if the application was accessing a database. Distributed Java has a similar approach with serialized objects, the Java language and JDBC/SQL. Web services have one type space (XML) for interfaces, data applications are manipulating, XML databases and in-flight messages. It may not be obvious why one type space is better than three. The primary reasons are simplicity and flexibility. In Web services, there could be one basic approach for converting a data structure/business object from one format to another, instead of potentially three different type models and tools. This could enable a simpler development tool suite and API set. Moreover, having a consistent model enables flexible and more dynamic placement of business logic. The transformation code could run in an application, in an active database or in network intermediaries (proxies). We designed Web services to support both asynchronous messaging and remote procedure call. Previous approaches started with one or the other, and then grafted the other model later on. For example, implementing a layer on top of MQ for simple RPC is common. CORBA was originally RPC centric, but then added support for asynchronous messages. In most cases, the after the fact grafting of one technique on another was awkward and error-prone.

So when you add these all five things together you see there is something different. This is not a case of a shell game. In fact the only thing that is the same is the use of SOA as an architectural style, where we have the notions of consumer, provider, service description, broker, and a registry are similar to previous approaches. Of course many technical people see SOA as solely an architectural style and this misses the point entirely of service orientation.

SOA Governance

How do we fund SOA projects?