The first installment of the Architecture in practice series, Realizing Service-Oriented Architecture, discusses IBM's Service-Oriented Architecture (SOA) Foundation Lifecycle (or SOA Lifecycle) and how it allows IBM customers to think of SOA in terms of a software development lifecycle. The four phases of the IBM SOA Lifecycle â Model, Assemble, Deploy, and Manage â are covered in detail.
This article focuses on the second of the eight SOA solution scenarios: the Service Connectivity scenario. Part 4 discussed techniques for creating services, and the motivation for reusing existing functions and exposing them as services. The Service Connectivity scenario takes this idea one step further, and builds upon the pivotal theme that it's better to reuse assets than it is to build them from scratch. In this article, learn about the various mechanisms that services both inside and outside the enterprise perimeter that can be connected to realize a service-oriented ecosystem. Learn about four different realizations of SOA connectivity, and how their implementations can be mapped to the phases of the SOA lifecycle.
For each example, IBM recommends tools and products that support its implementation. The main architectural construct that enables service connectivity is an Enterprise Service Bus (ESB). This article also examines some of the most commonly encountered topologies of an ESB, and how they may be used to define an end-to-end service connectivity solution.
Though many tools and products are mentioned, it's outside the scope of this article to explain all of them. See the Resources to further explore topics of interest.
Addressing the business challenge
To become a service-oriented business and a participant in an SOA ecosystem, today's enterprise must figure out how to integrate existing and new services into a well-orchestrated business process. The heterogeneity of multiple platforms, multiple transmission protocols, and diverse data formats in the modern IT environment make it imperative to address the seamless flow of information from anywhere at any time using anything.
The ability to build business processes that transcend traditional enterprise boundaries, and to build trusted relationships with business partners, is becoming a mainstream requirement. The challenge is to:
- Homogenize the connectivity of assets and services to support the business processes, and
- Simplify the connectivity between services and the SOA infrastructures to build a seamlessly connected platform that fosters a secure, reliable, and scalable flow of information in a federated environment.
The Service Connectivity scenario
The Service Connectivity scenario demonstrates the connectivity between service producers and consumers, fostering reuse of new and existing services across multiple delivery channels. For this article, we assume that the SOA solution scenarios were evaluated in the context of the business problem, and the Service Connectivity scenario was chosen for implementation. Briefly, the decision-making process relied on:
- Part 2 of this series, which provides a process for using the SOA scenarios. A critical part of the process is the criteria for selection. In Table 1 of Part 2, there are four generic use cases that fall under the Service Connectivity scenario. The first step is to map the functional requirements of a given problem domain to one of these four generic use cases.
- Realizations that were developed, which show how the scenario can be realized using the selected products. Realizations combine a customer scenario along with its solutions. Typical realizations have been provided that help customers jumpstart their SOA implementation.
The next section introduces the realizations of the generic use cases.
The four common realizations
At the heart of the Service Connectivity scenario are the concept, application, and implementation of the ESB. The features of the ESB are mapped against the requirements of a given problem, which in turn yields one or more ESB products that may be required to implement the solution. The four common realizations are:
- Secure connectivity to external third parties and trading partners
- Connect internal business systems based on open standards
- Deliver existing business processes through new business channels
- Adapt from third-party systems to Web services
Secure connectivity to external third parties and trading partners
The requirement to extend the enterprise business process outside the enterprise boundaries, by integrating services provided by third-party vendors or business partners, warrants a common solution that can be used by any client. An ESB gateway is usually recommended, to be used either standalone or in conjunction with an intra-enterprise ESB to provide secure service interaction between internal and external domain boundaries.
An ESB gateway can be used standalone when the sole requirement is secure access to resources. An ESB gateway can play a dual role by also providing some basic mediation and routing capabilities, based on the middleware and integration requirements. In such situations, the recommended product to use is the IBM Datapower XML Security Gateway XS40 (henceforth called the Datapower appliance). The Datapower appliance is a hardware appliance designed to provide: XML transformation, a secure access channel to invoke enterprise services, and basic mediation flow capabilities. They are chiefly designed to be message-aware network proxies for Web services traffic. The devices have extremely high performance in processing XML messages. They are also a message control point in an application's security architecture.
Usage of a gateway ESB for this generic connectivity use case should also follow the SOA lifecycle. The phases of a lifecycle are modular in nature; the execution of each phase provides its own benefits and is not dependent on the execution of the other phases to realize those benefits. Figure 1 maps the high-level implementation steps of this use case to the SOA lifecycle.
Figure 1. SOA lifecycle steps for service connectivity based on third party partner integration
Table 1 explains the activities for this use case during each phase of the SOA lifecycle.
Table 1. Service Connectivity based on third-party partner integration
|Model||There is nothing special about the high-level activities in this phase; they mainly focus on understanding and documenting the influence of the third-party services in the enterprise policies, roles, and performance metrics.|
|Assemble||The IBM Datapower Toolkit is used to configure the Datapower appliance with a simple XML firewall. Among the configurations, of importance are the configuration of the decryption and encryption of the request and response messages, and defining the rules associated with them.|
|Deploy||Configurations are defined in the Datapower toolkit and deployed onto the Datapower appliance. The appliance acts as the ESB gateway, enabling a seamless and secure integration of external services into the enterprise business process.|
|Manage|| Services that flow across the enterprise boundaries are
monitored for compliance to security, performance, and availability requirements.
IBM Tivoli® Composite Application Manager (ITCAM) for SOA is the recommended
product in this case. The Datapower appliance is seamlessly integrated with
ITCAM, and may be used to monitor the service invocations and
flows across the perimeter boundaries.
If enhanced security is required beyond the service message-level security provided by the Datapower appliance, the IBM Tivoli Access Manager (ITAM) may be used. ITAM is seamlessly integrated with the Datapower appliance to add extra security. If the infrastructure already has an implementation of a security mechanism (such as ITAM), the Datapower appliance may not be configured for message level security and may only be used for accelerated XML processing.
Connect internal business systems based on open standards
Sometimes an enterprise has a business and IT prerogative to use standards-based technology for connectivity between various services that orchestrate business processes with a WebSphere® Application Server-based middleware infrastructure. When the enterprise wants to reach out to multiple clients that support a multitude of standards-based service invocation mechanisms, the recommended ESB is the IBM WebSphere ESB.
WebSphere ESB provides a full-featured departmental solution for standards-based ESB implementations. It supports publish/subscribe and one to many flows, is fully transactional and exactly once delivery of critical business functions, and arbitrary integration logic wherever necessary. The mediation capabilities assist in transparent support for message formats and transport protocols required to support the service provider.
Figure 2 maps the high-level implementation steps to the SOA lifecycle.
Figure 2. SOA lifecycle steps for service connectivity based on open standards
In the Service Connectivity scenario, the focus is mainly on the assembly and deployment of services and business processes, which may be followed by the standard mode of practice for their management and monitoring.
Table 2 explains the activities for this use case during each phase of the SOA lifecycle.
Table 2. Activities for connecting internal business systems based on open standards
|Model||The main focus is on identifying and documenting the business channels that service providers use to access the processes. The business processes are simulated; metrics gathered from the simulations help develop the business case for automating and modernizing the business processes using standards-based connectivity between the services that realize the processes.|
|Assemble||The mediation flows that provide message routing and transformations are designed and implemented. IBM WebSphere Integration Developer is the recommended tool to build WebSphere ESB mediations and flows. The mediation flows help in processing incoming messages, logging content for auditing purposes, or decomposing the message if necessary before routing it to its final destination. See the Resources for more on this topic.|
|Deploy||The mediation flows that are assembled and built in WebSphere Integration Developer are
deployed to the WebSphere ESB run time platform. WebSphere ESB is built on top of WebSphere
Application Server, and inherits all of its standard and advanced security,
performance, clusterability, and fault-tolerance capabilities. WebSphere ESB provides:
|Manage||The mediations deployed on WebSphere ESB must be managed
and monitored to meet the specified service level agreements (SLAs). The
recommended infrastructure capable of managing and monitoring the WebSphere ESB
mediations is ITCAM for SOA. |
ITCAM for SOA delivers a comprehensive management solution for enterprises deploying SOA applications based on Web services. It can discover, monitor, diagnose, and control Web services implemented using open standards like SOAP/HTTP or SOAP/JMS. It helps identify and resolve problems that can occur around the deployed services.
Deliver existing business processes through new business channels
An essential tenet of an SOA ecosystem is to foster a seamless and transparent connectivity between service consumers and service providers. The heterogeneity of a typical business and IT environment poses a distinct set of challenges for integrating consumers and providers of services. The message format sent by the service consumer:
- Might not be standardized
- May vary between one consumer and another
- May be too coarse grained and need decomposing
Message decomposition, routing, and mediation are key requirements.
Even if message formats are resolved, and routing based on message information is implemented, protocols used by service consumers to initiate a service request may be different from the protocol expected by the provider. For example, a service consumer may initiate an HTTP-based service request, while the actual service may expect a JMS-based request invocation.
Security around service invocations is also a challenging endeavor. A service may require proper security context when invoked from outside the enterprise perimeter, but might not require any security when invoked by applications and systems inside the corporate firewall.
A service requester may request a business function that might be supported by one or more services that are composed together to realize the function. Such compositions need to be designed, implemented, and deployed on a middleware layer to facilitate seamless connectivity.
An enterprise with core business functions implemented in legacy systems and applications may need to unlock its business potential and open up. Both external customers and the enterprise can benefit from reuse across multiple applications in the IT portfolio. The ability to connect to standard and non-standard back-end systems through specific technology adapters becomes an imperative requirement.
An advanced ESB that performs, at a minimum, the following functions would be an ideal choice in this scenario:
- Conversion between communication protocols
- Conversion of the heterogeneous data formats into a canonical message model, with mediation capabilities
- Routing of service request to the proper service provider
- Composition of services to fulfill a service request
- Transaction management
- Messaging backbone
IBM WebSphere Message Broker (Message Broker) is suitable where advanced ESB functions are required. Message Broker is an option when Web services support is not critical, and quality-of-service requirements demand the use of mature middleware. Message Broker can support all the ESB capabilities above, and is not limited to interoperability based on open standards.
Figure 3. SOA lifecycle steps for extending business processes to multiple channels
Table 3 explains the activities for this use case during each phase of the SOA lifecycle.
Table 3. Activities for delivering existing business processes through new business channels
|Model||Do a deep-dive analysis for the business processes that cut across organizational boundaries. Services used in multiple business processes, based on the context, might need to support different business channels for interacting with the service consumer. Each of the protocols and message formats used by the service consumer must be supported. Requirements and findings in this phase help make a more informed judgment on the advanced features that the EBS may need to perform.|
|Assemble||The orchestration of services and associated message flows are designed and assembled. The recommended tool is the WebSphere Message Broker Toolkit, which is an Eclipse plug-in based on IBM Rational Application Developer. The tool assists in the development of message flows and message sets, and administers a message broker and its run time components (execution groups and the deployed message flows that run within them). It increases the speed and ease of developing message flows. Improved Web services support, including WSDL generation and use, provides key standards based service choreography.|
|Deploy|| A combination of at least three products are used. Message Broker is used as the ESB, providing a robust mediation support among other
capabilities, with WebSphere MQ providing the perfect foil as a persistent
messaging backbone. With multiple different legacy systems,
WebSphere Adapter technology provides the connectivity from the Message Broker to
the variety of different legacy-based applications. |
The WebSphere Adapter framework has more than a hundred technology adapters, providing connectivity and integrating with virtually any vendor and packaged products that one can imagine.
|Manage|| ITAM provides a
centralized, flexible, and scalable access control solution to control user access
to protected information and resources. Tivoli Federated Identity Manager (TFIM)
is used to manage the propagation of security credentials across security domains.
With Message Broker and WebSphere MQ as the ESB infrastructure, IBM Tivoli OMEGAMON XE for WebSphere Business Integration is recommended to manage and monitor the ESB infrastructure. Omegamon improves Message Broker and WebSphere MQ availability, proactively prevents problems, and simplifies management with a single tool. It also offers a Web-enabled tool for reporting real time availability and performance of the resources it manages.
Adapt from third-party systems to Web services
Adherence to open standards, such as SOAP/HTTP and JMS, is increasingly becoming a corporate mandate, yet the majority of enterprises have a medium-to-heavy IT portfolio based on legacy applications and systems. A fundamental principle of SOA is to unlock the business value embedded in old, existing, and third-party-based applications, systems, and technologies by adapting them to open standards based on Web service technology.
Reusing key functions in existing systems and applications, in a mechanism that realizes multiple business processes and reaches different business channels, is a fundamental value proposition of SOA. You can reduce costs of new application development, and costs throughout the development lifecycle, by reusing existing functions adapted by modern Web services technology. Adapting third-party and legacy systems using Web services technology is a relevant and important option during any application modernization initiative.
You can use an ESB to provide the connectivity to enterprise information systems (EIS) and legacy systems using the adapter framework. There are two main categories of adapters to consider:
- The WebSphere Business Integration (WBI) adapters allow heterogeneous business applications to exchange data between ERP, HR, CRM, and supply chain systems through the coordinated transfer of information using business objects.
- The WebSphere JCA adapter provides a mechanism to store and retrieve enterprise data in J2EE™. It supports enterprise connectivity to JDBC, flat files, and systems such as SAP, PeopleSoft, or Siebel.
Usage of the WBI and WebSphere JCA adapters assumes:
- The customer has, or is considering having, a WebSphere Application Server infrastructure.
- There are no WS-Security or other complex security considerations.
- There aren't many data synchronization requirements between the client and the EIS systems.
- The volume of requests is moderate.
Whether to use WBI adapters or WebSphere JCA adapters is an architectural decision that needs to be specific to customer requirements and the existing third-party systems. Some of the WebSphere adapters include:
- IBM WebSphere Adapter For Flat Files, Version 6.0
- IBM WebSphere Adapter for JDBC, Version 6.0
- IBM WebSphere Adapter for PeopleSoft Enterprise, Version 6.0
- IBM WebSphere Adapter for Siebel Business Applications, Version 6.0
- IBM WebSphere Adapter for SAP Applications, Version 6.0
Table 4 explains the activities for adapting from third-party systems to Web Services in relation to the SOA lifecycle phases.
Table 4. SOA lifecycle and activities for adapting from third-party systems to Web services
|Model||While mapping the activities in this Service Connectivity option, nothing unique is required.|
|Assemble||WebSphere Integration Developer is the development and assembly tool used for building mediation capabilities that run on the WebSphere ESB. Mediation functions in this realization include the enterprise discovery capabilities needed to incorporate the WebSphere BI or JCA adapters into the mediation applications.|
|Deploy||Both the WebSphere ESB and WebSphere Adapters comes into play. The WebSphere Adapters provide the EIS adapters required for connectivity to back-end and third-party systems. The WebSphere ESB provides the mediation capabilities to perform the XSLT transformation on the data, and includes a function to log messages as they flow through the mediation.|
|Manage||ITCAM for SOA is the recommended infrastructure component to monitor and manage the traffic between Web Services invocation. A federated identity management solution is needed to manage the propagation of security credentials across multiple tiers in the architecture -- specifically between the business logic layer and the EIS systems. IBM TFIM is the recommended solution to manage the identity and access to resources that span multiple security domains.|
Overview of mediations
Mediations, in the context of connectivity between services in an SOA, refers to:
- Ways in which the incoming service request message is transformed, routed, and augmented before it reaches the service provider.
- How the same techniques are applied to the response from the provider before it is finally sent back to the originating requestor.
WebSphere ESB is a run time environment capable of running message mediations. The technology used to define the message mediations is based on Service Component Architecture (SCA). A mediation module is a basic unit of deployment for message mediation. It contains mediation flow components that operate on the message requests and responses. In the final packaging of a mediation module, there is at least one export (defining the interface through which service consumers invoke the mediation) and one import (defining the connectivity to a service provider). The mediation flows and the modules are all designed and packaged using the mediation flow editor in WebSphere Integration Developer.
Mediations are developed following four commonly used patterns:
- Message protocol transformation
- The mediation module converts any standards-based protocol (HTTPS, HTTP, SOAP/JMS) used by the service requestor to any of the other standards-based protocol formats, thereby letting the service provider support any standards-based protocol used by service consumers.
- Content-based message routing
- The mediation module encapsulates the logic
and intelligence to dynamically choose one of multiple service providers to
fulfill the service request. The logic is primarily based on filtering rules that
may be applied on the message content, message header, service requester, and
so on to decide the best service provider at run time. This is a form of service
virtualization that keeps the requestor completely ignorant of which provider is
actually servicing the requests.
A Transfer Funds service is an example of this form of virtualization based on business rules and non-functional requirements. A premium customer might have funds readily available, but typical customers require a two-day period before the funds are available.
- Message format transformation
- This method comes in particularly handy when the service consumer uses a message structure that is not compliant with the
structure the service provider expects for its incoming request messages. In
such cases, the mediation module mediates the incoming request, detects the
differences, and translates the request message format into the response message
format before forwarding it on to the service provider. It also does the reverse
transformation from the service response message format to the format of the
An example implementation of this type of mediation module is an XSLT transformation from the requestor message schema to the provider's message schema.
- Message augmentation
- This type is used to add new parameters to an incoming request before it is sent to the service. Sometimes the incoming messages don't provide all the necessary information. The mediation module intercepts the incoming request message, performs functions such as database lookup, then augments the message with the required parameters before passing it on to the service provider. This pattern may also be used when the addition of optional parameters can speed up the processing of the service request.
The message protocol transformation and message format transformation patterns are used to resolve differences between the service consumer and service provider. The content-based routing and message augmentation patterns are used to implement a form of service virtualization, and to encapsulate the addition of intelligent logic to facilitate a more seamless connectivity and communication.
Choosing the right ESB
Three different ESB products have been discussed so far: WebSphere DataPower Appliances, WebSphere ESB, and WebSphere Message Broker. This section provides some guidance on choosing the right product to meet your ESB requirements.
The WebSphere DataPower Appliances is a set of three different hardware appliances for SOA solutions that redefine the boundaries of middleware. They extend the SOA foundation with specialized, consumable, dedicated SOA appliances that combine superior performance and hardened security for SOA implementations. This appliance may be used at the enterprise perimeter, providing accelerated (at wire speed) XML, and enhanced SOA security processing with XML-level protection and structured data handling. It also provides very basic mediation capabilities, and integrates seamlessly with the security infrastructure based on IBM TFIM, Access Manager, and so on. If you are looking for an ESB gateway that also performs some basic message mediation, then the WebSphere DataPower Appliances is recommended.
WebSphere ESB will connect applications with standards-based interfaces to power your SOA. Building an ESB that is solely based on WebSphere ESB is an option when Web services support is absolutely critical, and the service consumer and provider communicate only with open standards. Since WebSphere ESB is built on top of WebSphere Application Server, it leverages all the transport protocols that the latter supports. WebSphere ESB supplements Application Server with mediation capabilities that provide intelligent service connectivity options.
WebSphere Message Broker is the most advanced ESB solution in the WebSphere suite of products. It provides any-any connectivity for both standards and non-standards-based technologies and applications. Message Broker also facilitates an any-any message and data transformation framework, realizing an SOA solution in the most complex and heterogeneous environment. This product offers transactional processing, near-universal connectivity over multiple transports (including WebSphere MQ, JMS, and HTTP), and reliability and performance unmatched by others in the market. It supports a broader set of transformation and mediation features, and supports integration assets that may or may not be Web services-enabled. When fast throughput and quality-of-service requirements are critical, Message Broker is a far better choice than WebSphere ESB.
Coexistence of multiple ESB products
A WebSphere DataPower appliance is complimentary with either WebSphere ESB or Message Broker. Figure 4 shows how the different ESBs may be used together.
Figure 4. ESB products working together
DataPower and WebSphere ESB can coexist in an ESB infrastructure because they provide complimentary ESB features. While WebSphere ESB provides message mediation and transformation features within the enterprise firewall, a DataPower appliance can be deployed at the enterprise perimeter and configured to provide message security (denial of service, authentication, and authorization). DataPower can also provide basic message routing capabilities. In configurations with multiple departmental WebSphere ESBs providing services specific to line-of-businesses, the DataPower appliance can route messages to the correct WebSphere ESB run time, working as a federation hub for a multi-WebSphere ESB run time.
In situations where higher processing throughput is required in a WebSphere ESB-only run time, the power of DataPower appliances can be harnessed by offloading the XML processing overhead from the WebSphere ESB to the appliance, freeing the WebSphere ESB platform to provide higher service throughput. WebSphere ESB can also bolster a DataPower-only infrastructure. It can add full transaction support, complex message flow implementations, a persistent JMS messaging server, publish/subscribe support, and a wide range of application and technology adapters, including a robust support for IBM transaction processing environments.
DataPower bolsters a Message Broker solution the same way it does for WebSphere ESB. Because Message Broker is the most advanced ESB in the WebSphere suite, it adds more features to a DataPower-only solution than WebSphere ESB. Message Broker provides, among other things: a broader protocol support (any third-party JMS 1.1 support, telemetry, device integration); a tight integration with IBM transaction processing systems such as CICS and IMS; advanced messaging and complex event processing; a sophisticated scheduler and timer service; and a general purpose programming environment (Java, ESQL, C, and so on) to define arbitrary programming logic.
Each of the three ESB products can provide a targeted middleware solution. They can also work together to create a more advanced and end-to-end ESB based service connectivity solution. The configuration to choose is dictated by the requirements of a given problem domain.
There are eight different SOA scenarios that IBM identifies as the most commonly occurring in typical SOA-based IT projects. To help customers get started with SOA, IBM offers business-centric and IT-centric SOA entry points and scenarios, and provides comprehensive guidance on how each scenario should be modeled, designed, and implemented using IBM tools, products, and methodologies.
In this article you learned about Service Connectivity, the second SOA scenario. The Service Connectivity scenario is a realization of the connectivity entry point. The connectivity entry point allows you to effectively connect your infrastructure, integrating all of the people, processes, and information in your company. With flexible SOA connections between services and throughout your environment, without much effort you can take an existing business process and deliver it through a different business channel. You can even connect to external partners outside your firewall in a secure way. You can also adapt legacy and third party applications using Web services technology to unlock their business value and foster their reuse.
This article also explained how the SOA lifecycle can be applied to the realizations, and how the IBM product suite can address the specific design, development, and run time requirements for service enablement.
Read the other parts in this series:
- Part 1, Realizing Service-Oriented Architecture, discusses IBM's SOA Foundation Lifecycle (or SOA Lifecycle) and how it allows IBM customers to think of SOA in terms of a software development lifecycle.
- Part 2, An introduction to SOA solution scenarios, introduces the eight most common scenarios in a typical SOA-based IT project.
- Part 3, Top ten tips for writing great IT project proposals, highlights the major steps you should follow to ensure the development of a high-quality proposal.
- Part 4, Service creation options in a real-world SOA scenario, focuses on the Service Creation scenario.
- "Integrating WebSphere DataPower XML Security Gateway XS40 with WebSphere Message Broker" (developerWorks, Oct 2007) shows how the DataPower XS40 enhances a Message Broker flow with WS-Security, with security of the connection to Message Broker provided by HTTPS.
- The following IBM Redbooks are helpful:
- WebSphere Message Broker Basics provides an overview of the latest release of WebSphere Message Broker and the Message Brokers Toolkit.
- Implementing IBM Tivoli OMEGAMON XE for WebSphere Business Integration V1.1 covers the installation, configuration, and troubleshooting of IBM Tivoli OMEGAMON XE for WebSphere Business Integration on Windows, AIX, Linux, and z/OS platforms, with real life scenarios.
- Patterns: SOA Design Using WebSphere Message Broker and WebSphere ESB discusses patterns for integrating WebSphere Enterprise Service Bus and WebSphere Message Broker, and includes a scenario to help you design, develop, and deploy these products.
- "Twelve great things about the WebSphere Message Broker V6.0.2 Toolkit" (developerWorks, Jan 2007) explains how to take advantage of new features in the WebSphere Message Broker V6.0.2 Toolkit.
- Read "Getting started with WebSphere Enterprise Service Bus and WebSphere Integration Developer" (developerWorks, Jan 2006) to learn how WebSphere Integration Developer is used to build mediation flows for the WebSphere ESB.
- Learn more about:
- In the Architecture area on developerWorks, get the resources you need to advance your skills in the architecture arena.
- Browse the technology bookstore for books on these and other technical topics.