BPM & Service-oriented Architecture: Insights and Best Practices
Ali_Arsanjani 120000D8QB 1,025 Visits
Companies who want to collaborate and are in fact, business partners can do so in two ways: proprietary and standards-based.Standards-based is most flexible but covers a spectrum of industry standards such as ACORD for insurance to a standard XML Schema with pre-agreed tags. This is the spectrum of standards-based interaction. As we move from a few instances of collaboration and access to one another's business processes, the adoption of industry standards based on XML becomes more important: the rules of engagement are well-understood and have been well thought-out by a an industrty body or consortium. The downside of this is that often the industry standard is too broad and all-empassing making its use cumbersome and company's end up using only a fraction of the standard. Standardized message structures across Lines of Business within the enterpise and gradually across the eco-system become increasingly important.[Read More]
Ali_Arsanjani 120000D8QB 953 Visits
I have picked one of the favorite topics that I have received most emails on; namely SOA Governance, and would like to expand it to the topic of SOA Governance for the Service Eco-system.
But let's start small: inside the organization. Within the organizational dimension, we have the need for governance and specifically SOA Governance.
To ensure that the SOA Life-cycle is carried out correctly, we need the support provided by SOA Governance.
SOA Governance is more like an umbrellla activity that 1)oversees the functioning of the SOA Life-cycle and 2)defines relevant, currently apropos policies, 3) inserts control points and 4) communicates these policies, the values expected at specific control points to 5) ensure compliance with those policies with artifacts produced by activities conducted within the life-cycle.
Governance is more declarative; management is executive. SOA Governance declares policies and activities it expects to see, and delegates to management to carry out these actions to ensure conformance to policies, or sometimes, more importantly, to determine why a given activity, project or artifact must obtain an exception and WHY IT SHOULD NOT CONFORM to the governance policies.
George Galambos and I gave a talk last week at the annual IBM Technical Leadership Exchange about a topic that we have been talking about for some time now: the service eco-system. If you want to build a service eco-system you may need to look at some patterns that help you build your service eco-system .
As we move towards greater maturity in the adoption of SOA,there are a number of key challenges that need to be overcome. I like to call them the Grand Challenges of SOA. The first 9 are:
1. Business Case for SOA
2. SOA Maturity and Roadmap: What do I do next?
3. Service-oriented Modeling and Architecture end-to-end
4. Industry Specific SOA
5. Building composite applications
6. Monitoring and managing across the eco-system
7. Governing the SOA eco-system
8. Eco-system flexibility with declarative policies, service management and externalized functionality.
9. Service proliferation challenges quality of service
What do you think?
Ali_Arsanjani 120000D8QB 940 Visits
SOA Governance is about the governance of the three fundamental elements of SOA, namely, services, components and flows. So we are governing the process, the artifacts around services, components and flows. In the SOMA Method, the SOA Method used to model, analysis design services, components and flows, we identify, specify and realize these elements.
This relates to the monitoring of these elements throughout the life-cycle; putting in control and check points and policies around corrective action as we develop these.
Governance seeks to ensure adherence to/compliance with policy along the execution of a set of process steps that may start from the manual/human aspects of the life-cycle and continue onward into the runtime enviironment. It often accomplishes this goal by planning and instituting a set of check points or control points where process results are cross checked/validated with a set of standards (including permissible alternatives) as defined by policy.
SOA Governance sees to it that these elements are relevant to the organization (vitality), are being reviewed and validated by stakeholders and being communicated within the organization as the service model is being constructed within the life-cycle.
The governance of the service life-cycle is necessary to ensure that the needs of the business is supported by a set of flexibly re-composable IT services or components.
In order to do so, there needs to be policies, principles, checkpoints, reviews put into place from a process perspective; the execution of this is delegated to management Run-time governance of SOA on the other hand involves runtime monitoring of events and service execution to ensure compliance with the qualities of service declaratively defined by SOA policies.
Ali_Arsanjani 120000D8QB 1,130 Visits
Nasa has announced that one of its satellites have spotted a truly wonderful phenomena leading to the conclusion that it may have found evidence of liquid water reservoirs that erupt in Yellowstone-like geysers on Saturn's moon Enceladus.
After a long dry spell of blogging due mostly to heavy engagement on customer projects, I mark my midnight ruminations with the blog update. Multiple SOA projects in several industries: electronics, telecom and insurance. Sometimes I feel like what the satellite may feel like! Constant travel to meet with customers, find and fix their problems strategic and tactical in their journey towards SOA.
Like Cassini, I also discover some strange and wonderful things on client projects: the key challenges encountered on SOA projects. One of these is reklated to the human factor and the great influence of that factor versus the technical aspects of software engineering: the dynamics of organizations, people, communications, reporting, expectation setting and managing as well as mentoring others.
I play two roles: Chief Architect on client projects and university professor. As part of my latter duties I feel a need to convey some of my experiences to the next generation of software engineers that I teach in the following courses: Advanced Software Engineering, Dynamics of Software Architecture and Service-Oriented Architecture at the graduate level.
As I discuss how to become a software architecture with the masters in computer science I teach at MUM, one thing strikes me as very insightful to pass on to the students: that technology is many times the easy part! It seems to be the softer human skills that the architect needs to acquire for maximum effectiveness on SOA projects.
Ali_Arsanjani 120000D8QB 881 Visits
Many ISV's have professed new architectures for supporting SOA. For example , they are modeling typical business processes and exposing them as services.
Again, whether you are going down the SOA path by way of "engrain in company DNA", or "Natural Selection" or "Grand Design", you need to:
1. Integrate with legacy (operational systems layer), implies consolidation and multiple federated integration scenarios whihc would call for using best-practices in SOA design .
2. Ensure your consumer or presentation layer is loosely coupled
3. Then you can leverage all the SOA design sandwiched in between those two extremes.
4. BUT now, with the advent of the service eco-system , your company cannot maintain competative advantage unless is it at a maturity level that supports cross business partner interaction using, yes, you guessed right, services.
To create the eco-system, we need to understand what the business collaboration requirements of business partners are and what they are projected to be.
Often it's not only hard to obtain requirements, but also to get the business to say what they really want, in what order and by when; instead a general wand is waved(inbued with the most expensive peacock feather, as only those who hold purse strings can have).
But no matter what you gotta get those requirements translated into IT speak. The dW Architecture posed its second question: "How do I translate business needs into IT requirements?" I think it's mostly a matter of empowering business analysts with the ability and means to express their requirements properly and to help them do that, we need a teeny bit of rigor.
Increasingly, technology is helping make life easier in a large-scale, objective fashion albeit more focused in certain geographic regions. On the other hand, companis are also focusing on providing the consumer market with intensely subjective experiences which may indicate a trend of increased personal isolation: for example, a company has developed newwearable display for video iPods (typically costing 3x the cost of a typical iPod). We are accessible through cellphones, on the go; now we are accessible to entertainment providers, on-the-go.
Services on-the-go, tend to be context dependent; my choice in music and TV programs may be different from that of someone ten years younger. This may also be contingent on the time-zone I currently reside, to be able to get certain programming, and on the time of day. All of these point to a class of services, I have been referring to as context-aware services. An extreme version of this is when your cellphone announces to you that you are within walking distance of your favorite Chinese restaurant chain and it is near lunchtime in the current time zone.
Context-aware Services (CAS) is a convergence of a number technologies and paradigms: SOA, telecom, banking and personalization, to name a few. In some cases, context-aware services relate to the specific domain in which we are functioning: e.g., financial markets vs consumer banking vs insurance, etc.
This is what we will gradually see emerging on the horizon this year as the New Year dawns. Happy New Year!
Ali_Arsanjani 120000D8QB 945 Visits
One of the important aspects of SOA Governance that we have been discussion is that of Service Domains: rather than having a lines of business own systems, Service Domain Owners choose Service Domains which group a set of related services. These affinity of these services are often of a business nature, or originating from a buisness perspective at the very least.
Decomposing a business domain into its sub-domains and functional areas enables a set of natural boundaries that can be a good starting point for Service Domains.
The Service-oriented Modeling Approach uses this strategy to identify domains, decompose them and then put governance around them.
Every domain owner will provide a set of services and wil in turn, itself require a set of services for consumption.
Thus the publication of the servcie descriptions of both these sets of services are critical to get the service eco-system going in your enterprise.
This allows the Domain Owners to make decisions about implementation changes to Service Implementations without changing the Servcie Description, if the change does not affect the SLA (service level agreement) and QoS (quality of service).
A set of base services can be instilled into an organization as the primary business functions it provides and decide which ones it in turn requires from its business partners. (the DNA approach we discussed earlier , recognizing the fact that there are three approaches to this paradigm, each of which may fit your enterprise.
Instill, let go and observe the evolution of services as they are adopted at various levels of maturity (or not), exercise governance for communication of the services, obtaining feedback to ensure they are relevant and to check for compliance.[Read More]
Back to the discussion of Service Domains: changes and ownership through governance.
Two readers have responded with interesting considerations.
MMahesh, your response brings about an important element of SOA: that of the separation of description from implementation and to IoD's point, if the change in question is only one of functionality and not SLA change or QoS change, then the Domain Owner has less of a say in this.
It thus appears that the service consumer expects a certain function (Service description) and a certain QoS. If these are satisfied, then the consumer if typiclaly happy.
Thus the change might not even affect the consumer, in which case the question is whether non-consumer affecting changes should be reported.
In some cases, I would like to point out that this may be warranted: where there is a deviance from a reference architecture, and i.e., of compliance. Now if the change makes sense then it must be propagated through the governance process of vitality which ensures continued relevance of the compliance criteria (eg a reference architecture or reference model) .
1. provider and consumer roles have their own concerns and governance applies to each individually and then to both to maintain a healthy service eco-system.
2. Change the implementation if you want (provider) or change it if you have to (consumer) and you are not getting your QoS or functionality. This ability to have the power to choose from various providers who suit your needs (functioanl and operational (QoS)), is a key feature of SOA.
3. Service Domains define a sphere of responsiblity, both functionally, operationally and financially.[Read More]
Now you have convinced people to invest in building services for flexibility, redundancy elimination and business/IT alignment. Great.
Each business line goes about their way to achieve it, but soon will find out that there is one conspicuous issue: who is the Service Owner? If I build this service and it is used in multiple places, who pays for the changes if the changes are those I do not specifically "care about" in my line of business?
Thus appears the notion of Service Domains, where a domain defines a set of related services that some "one" can own, maintain, support and (obtain) fund (ing).
To make a chance to a service, contact the Service Domain Owner.
Here is a question for you:
If I don't need to change the service description, can I change the service implementation? If so, do I still have to go to the service domain owner?
Several folks have raised important points in this connection. Namely (assuming I have interpreted the comments correctly):
1. A question of SOA Governance:
"Do services come from some apriori (generalized, enterprise-wide or industry-wide) schema (Kant), or do they come from the specific local requirements (Hume)?"
2. Instill SOA into company DNA: "The DNA or "Nature" has to be "Nurtured" in an Information Technology (IT) still in its infancy. You could think of the IT DNA as the defined lower metrics required for any IT shop to deliver Information. The key performance criteria for Information delivery SOA as a roadmap to bridge the communications gap between the "IT think" people and the "Business driver think" people."
3. Natural Selection: The more we put in the DNA, the more restrictive the future generation of SOA will be, the less we put in the DNA the more fragile our SOA generation will be.
4. The Grand Design approach: "...grand design, and hints at a roadmap created by an external, pervasive entity ... the lingua franca for cross-platform coding...."
To me, project motivation comes from two very diametrically opposed sources: pragmatics of funding (business drivers) and the passion of professionals (to do "the right thing").
In some organizational cultures, we will have an entry level of maturity that is going to be project focused and natural selection may indeed rule.
At the other end of the spectrum we have neterprise wide transformation efforts that are occuring with some degree of SOA governance that follows a roadmap, "grand design". This allows the "instilling of SOA into Company DNA" in a spiral form : not necessarily top-down, not project by project, but by whichever comes first. The governance in place sees to it that where the passion of professionals or the pragmatics of funding are the drivers, that success is achieved within the designated , I repeat, designated scope.
As diverse projects weave their way to SOA, SOA governance sees to it that Variation-oriented Analysis is conducted across projects by the enterprise architecture board (with business and IT representation), Service Refactoring is done to factor out common services that eliminate redundancy (the third Service Litmus test).
Immanuel Kant, grandiose philosopher, maintained that we have "synthetic notions a priori"; notions prior to experience that are embedded, as we would perhaps say today, in our DNA. David Hume, the empiricist, on the other hand maintained that our mind is a "tabula rasa" -- a clean slate that experience writes upon and we have no notions prior to experiences.
Should SOA start to be embedded in the DNA of the Enterprise or is it something that should be left as best addressed piecemeal, project by project, on an "as need" basis?
What do you think?[Read More]
Adoption of SOA is a gradual process. While some organizations are dabbling into Web Services wrappering of applications as a means of exploring the world of SOA and deciding how to proceed, others are engaging in enterprise wide business transformation. Others are in between, and have defined their roadmaps, vision, strategy and criteria for assessment and governance. They are embarking on this journey by integrating application using Service-orientation. This latter is called service-oriented integration (SOI). This is the intermediate stage in which a lot of projects find themselves.
I have described this process and some patterns for embarking on this transformation to SOA using SOI .
Inside IBM, we have been developing and leveraging a Maturity Model and Process for SOA called the Service Integration Maturity Model (SIMM). This is designed to achieve desirable stages of maturity.
We have found it convenient to define seven levels of maturity that are defined by the level of de-coupling and flexiblity achievable at that stage of maturity:
1. Silo (data integration)
2. Integrated (application integration)
3. Componentized (functional integration)
4. Simple Services (process integration)
5. Composite Services (supply-chain integration)
6. Virtualized Services ( virtual infrastructure and integration through virtualization)
7. Dynamically Reconfigurable Services (eco-system integration)
Each have a detailed set of characteristics and criteria for assessment.
Here is an informal description of the highlights.
Level One. Organization start from proprietary and quite ad-hoc integration rendering their architecture brittle in the face of change.
Level Two. Organizations have moved towards some form of EAI (Enterprise Application Integration), albeit with proprietary connections and integration points. The approaches used have been to look at legacy systems and attempt to dissect and refactor through data integration.
Level Three. At this level, companies have componentized and modularized major or critical parts of their application portfolio. They have used legacy transformation and renovation methods to refactor legacy, expose functionality in a more modular fashion, built J2EE, or .NET based systems with clear component boundaries and scope.
The integration between components, is through their interfaces and the contracts between them.
Level Four. The company has embarked into the early phases of SOA by defining and exposing services for comsumption internally or externally for business partners; not quite on a large scale but acts as a Service provider none-the-less.
Level Five. Now the company has extended it's influence into the value chain and into the service eco-system. Services form a contract between suppliers, consumers and brokers that can build their own eco-system for on-demand interaction.
Level Six The company has now created a virtualized infrastructure to run applications. This is another level of decoupling achieved after the decoupled of the application, its servcies, components and flows. Now the infrastruture can be tuned and made agile through the notions of the grid and the grid service. Monitoring, management and events (e.g., common event infrastructure) have been externalized.
Level Seven This is a dynamically re-configurable software architecture. Services can be composed at run-time using externalized policy descriptions, externalized management and monitoring.
A simplified version of this has been discussed in 2003 in the Introduction I gave as guest editor of the Communications of the ACM on Enterprise Components and Services. Also, the four simplified levels have been discussed elsewhere as four adoption levels taken from SIMM .
Mappings to CMMI have also been done and are a natural way to try and understand maturity. The main challenge in a few of these efforts has been to be more than just a categorization according to some semblence of CMMI, and to justify why something is Defined, Repeatable or Optimizing, for example. This is often easy to do, due to the open and interpretable nature of the descriptions of each maturity level of CMMI.
The mapping to SIMM to CMMI is also quite straightforward; but the perceived value of this mapping seems distant other than the bandwagon factor of the success of CMMI. Service maturity relates to integration capabilities at various levels of business, methods, applications architecture and infrastructure.
CMMI deals with six levels of capability:
4. Quantitatively Managed
Each capability level consists of related specific and generic practices for a process area that can improve the organization processes associated with that process area.
Forrester has been discussing various notions of maturity, although from a different perspective.
Zapthink and Other analysts have discussed SOA roadmaps, which can be facilitated using SIMM.
Elements affecting maturity include coupling, standards usage, service identification that supports business needs, business models, goals and metrics that need to be supported by services, technologies, governance, infrastructure, tooling, skills and deployment capabilities.
Here is a developerworks article that provides more elaboration on this topic.
Ali_Arsanjani 120000D8QB 713 Visits
Of course with the current hype around Web services and SOA, there are a number of events hovering around this topic.
IBM is holding a series of SOA Days . The main SOA page for IBM products and trends can be found here .
Another event is the 2nd SOA and Web Services Best-practices conference in Chicago, and another is a workshop we are holding around the same topic at OOPSLA 2005, for the third year.
One of the main attractions of these and similar events are the battle stories we can exchange around SOA, which in the final analysis makes all the difference: what works in practice is the more important aspect of software engineering for clients.[Read More]
Ali_Arsanjani 120000D8QB 1,390 Visits
Summer vacation is over for most families with children. School has either started or is imminently going to... vacations are therefore usually over and we are back into the swing of things. We were talking about building the layers of an SOA, based on an SOA reference model. Let's discuss how to build the service component layer which is responsible for realizing the functionality and quality (adherence to non-functional requirements) of the service.
OK. So we assume know how to model and design an SOA. For that, we use some sort of Service-oriented Modeling method such as SOMA.
You have gone through Identification, Specification and Realization of Services, Service Components and Flows (processes).
During Realization you need to build the SOA. You need to map each service, service component and flow to a realization decision.
Once you have this table, you then have an idea of what you will:
3. Integrate (wrap, adapt)
4. Subscribe to (if event driven or publish and subscribe)
5. Transform (break apart a legacy system to get access to hitherto hidden nuggest of functionality not exposed in the orginal application)
Let's take number 1. Build. Building something is often about using a programming model to construct that piece of software.
In the 6th installation of the SOA Programming Model, Ferguson, Stockton and Nally talk about "A language-neutral, component-based programming model for service-oriented architecture (SOA) facilitates the implementation of Web services and their assembly into solutions."
This describes how to build the service components that will implement or realize your services.
The previous set of articles in that series can be found here.