Using WebSphere DataPower as the foundation of the IBM ESB Backbone

The IBM® ESB Backbone (IEB) is a comprehensive, yet easy-to-use ESB solution that provides reliable messaging, message transformation, and message auditing. This article shows you how to use WebSphere® DataPower to package security, auditing, logging, mediation, transformation, and any-to-any protocol handling into an easy-to-manage endpoints file that does not require the involvement of experienced DataPower developers.

Share:

Leonard Flournoy (flournoy@us.ibm.com), Software Developer, IBM

Photo of Leonard FlournoyLeonard Flournoy is a Software Developer with IBM Software Services for WebSphere (ISSW). He has been working in the IT industry for 17 years. He is an IBM Certified DataPower developer, IBM Certified SOA Associate, MQ V7 Certified Developer, and a Blue Harmony and IBM ESB Backbone team member for the past 3 years. At IBM, he is part of a team of engineers focusing on emerging technologies in WebSphere products, and more recently, DataPower SOA Appliances. His interests include the design, architecture, and development of enterprise scale applications specializing in the use of WebSphere MQ, DataPower, and message data transformation.



Rick Goncalves (rick.goncalves@us.ibm.com), Software Architect, IBM

Photo of Rick GoncalvesRick Goncalves is an Enterprise Architect and Systems Integration IT Principal Consultant for IBM Global Business Services. He has over 19 years of experience in the IT industry, focusing on design, development, testing, and implementing enterprise integration systems. He applies a deep understanding of enterprise middleware solutions, systems infrastructure, application client server, and multi-tier architectures to help clients manage complex, real-time integration demands. His industry exposure includes insurance, utilities and energy, retail, food and beverage, electronics, financial banking, marketing, government, life sciences, and insurance.



Paul Gordon (gordonp@us.ibm.com), Software Developer, IBM

Photo of Paul GordonPaul Gordon is a Certified DataPower Developer, Software Engineer, and Project Manager with IBM Software Services for WebSphere (ISSW). His extraordinary combination of hardware and software skills have enabled him to play many different roles within IBM over the past 17 years. He is currently assigned to IBM's Blue Harmony and IBM ESB Backbone (IEB) and Application Development and Solution Architecture Technology build teams. His expertise includes large systems, clustering, distributed file systems, web technologies, high-end serial parallel and portable computing, maintenance, tuning, capacity planning and performance measurement, hardware and software evaluation, migration, retail, security, audit processing, and research. With years of team lead experience, Paul also offers skills in project management and resources allocation.



Deepa Badimala (badimala@us.ibm.com), Software Developer, IBM

Photo of Deepa BadimalaDeepa Badimala has been working in IBM as an Application and Integration Architect and Software Developer for 12 years with IBM Software Services for WebSphere (ISSW). She has previously worked in other small and large IT companies in India and the U.S. She has wide experience in the full life cycle of the application development process and has strong knowledge of J2EE technologies, IBM middleware, web services, web technologies, business processes based on Service Oriented Architecture (SOA), integration of processes and IBM products with partner products.



Rod Bradley (rodbradl@us.ibm.com), Software Developer, IBM

Photo of Rod BradleyRod Bradley is a Senior Enterprise Integration Solutions Architect working in the IT industry for more than 17 years. He has extensive experience in the strategy, design, implementation, quality assurance, and administration of enterprise-wide solutions in retail, manufacturing, finance, insurance, healthcare, energy, and government. He holds certifications in WebSphere Message Broker and applies his strong knowledge in application development, middleware solutions, web services, and security protocols to help customers solve real world solutions. He is currently serving on the Blue Harmony project and has been a DataPower and IBM ESB Backbone architect and developer for more than 3 years.



23 October 2013

Introduction

The IBM® ESB Backbone (IEB) is an Enterprise Service Bus (ESB) solution that touches on many of the day-to-day complexities with reliable messaging, message transformation, and message auditing. This article offers a detailed view into a specific implementation of the WebSphere DataPower device and how this solution evolved. Any architect, application owner, or distributed application developer who has had to endure the arduous task of point-to-point communication over heterogeneous protocols will appreciate this straightforward solution that has considered numerous points of failure.

The IEB is an integral piece to the IBM Blue Harmony infrastructure. The Blue Harmony project is responsible for IBM's go-to-market initiatives. At its core is WebSphere DataPower X150 and Blue Harmony developers have provided custom code designed specifically to be used with the Blue Harmony project. The custom code handles logging, message transformation, message routing, and security.

DataPower XI50 and custom code are combined to create the IBM ESB Backbone (IEB). This article provides application owners information about what is required for onboarding the IEB and its benefits. The article also covers case studies detailing the successes and failures our team has encountered during the application onboarding process.


Blue Harmony utilization of the IEB

Blue Harmony is a key element of IBM's strategic transformation to the globally integrated enterprise, laying the foundation for daily operations for the future. It enables the IBM go-to-market initiatives like a Smarter Planet, business analytics, and cloud computing. Blue Harmony allows its sales team and Business Partners to combine hardware, software, and services orders into an integrated solution for customers.

The technical challenge of the Blue Harmony project is enabling consumers anywhere to access services anywhere without having to incur the development overhead of considerations to protocol, wire message format, network security, and so on.

Consider this: Application "A" needs information to process an order from Application "B". When Application "A" is HTTP-based, Application "B" exposes a SOAP service and they exist in the same security zone without a firewall, all is well (see Figure 1).

Figure 1. A basic web service call
A basic web service call

Unfortunately, most enterprises consist of a heterogeneous collection of consumers and providers. What happens when Application B uses MQ only? Before SOA and the use of ESBs, the consumer would have to provide an MQ client. When other real life scenarios like those listed above are added to the equation, it is easy to see why the use of the WebSphere DataPower XI50 appliance makes perfect sense (see Figure 2).

Figure 2. Services being accessed via ESB
Services being accessed via ESB

Out-of-the box, the XI50 instantly added value by decoupling the consumer and provider and offloading many of the processing chores that were not directly related to the calling application.

The initial goals of the IEB were simple and paralleled to that of most the SOA ESBs, as described in the following sections:

Functional requirements

The functional requirements are:

  • The consumer receives the service that he or she needs.
  • The consumer dictates the format of the message.
  • The consumer uses the protocol that he or he desires, such as HTTP, HTTPS, and MQ.
  • The provider focuses on the business logic without having to deal with all different types of consumers accumulated over time as the interface and infrastructure continue to evolve. That is, the ESB must decouple the endpoints physically and protocol-wise, while managing the connections efficiently and complying with all the infrastructure and security requirements.
  • The IEB is easy-to-use, change, and manage.
  • The ESB offers solutions to common business interaction patterns (request and response, one-way, notifications, pub/sub, event processing) and transport protocols (for example, HTTP, HTTPS, MQ, and FTP).

Business requirements

The business requirements are to:

  • Reduce development time and save development costs.
  • Reduce the costs and time to expose existing legacy interfaces using standard based service interfaces.
  • Reduce the time to deploy for consumers and composite services and respond to business needs.
  • Improve manageability and problem determination.
  • Reduce downtime and improve availability.
  • Reduce security risks.
  • Reduce costs by reusing the common infrastructure services.

The solution

The solution provided by Dr. David Chen and his team was to use DataPower's XI50 Multi-Protocol Gateway (MPGW) service and a custom code base to satisfy the requirements. What they developed became known as the "IEB (IBM Enterprise Service Bus Backbone)". The IEB code base makes it possible for consumers to access providers without knowledge or configuration of the providers protocol. Consumers make all requests to the IEB and the IEB routes the request to the configured service provider. The IEB also managed all of the non-business related functions. The real values of the IEB were its design and the emphasis on the reuse of existing objects.

A pure DataPower solution requires the DataPower developers to create a new service for each consumer/provider communication (Blue Harmony refers to this as an interface). Each interface might require additional services such as transformation or data enrichment. The number of objects required to handle these individual task multiplied by the number of interfaces would result in a system that places heavy demands on human, software, and hardware resources.

The IEB developers had two major goals in mind when developing the code:

  • The product delivers "ease of use" and "ease of management". It was this forward thinking that shaped the IEB into what it is today.
  • The IEB configuration is the driving force of the IEB and allows all parties to focus and compartmentalize on only the tasks that are important to them. While the typical DataPower developer is burdened to create a service for each interface, the IEB developer merely provides an XML configuration by specifying the details of the interface.

The IEB provides all the power of DataPower XI50, while at the same time making it possible for anyone familiar with XML to deploy new interfaces or modify existing interfaces. A certified DataPower developer is no longer required to perform these tasks. This abstraction allows just about anyone to administer the IEB, by adding, modifying, and removing Blue Harmony interfaces. Regardless of the interface pattern that an interface may use, the IEB developer has the power of control.


Exploring the Blue Harmony IEB environment

IEB is made of WebSphere DataPower XI50 SOA appliances, Load Balancers (WebSphere Edge Servers, and Nortel Alteon devices), and Global Server Load Balancers (GSLB of Nortel Alteon), which provide ieb.ibm.com global domain names for internal IBM consumers, and connect.ibm.com domain names for external IBM consumers. Multiple DataPower XI50 SOA appliances form a DataPower cluster behind load balancers for high availability. Many such DataPower clusters are interconnected and deployed in the Blue Zone, Green Zone, and Yellow Zone of the Application Hosting Environment (AHE) Boulder and Poughkeepsie Service Delivery Centers (SDCs) to achieve 24x7 99.5% availability with no scheduled downtime.

Requesting applications can send messages to one of the DataPower clusters of the same zone, using w3-bz.ieb.ibm.com, w3-gz.ieb.ibm.com, and connect.ibm.com for Blue Zone, Green Zone, and Yellow Zone, respectively. The requests are authenticated or authorized as configured and routed to the target destinations. These DataPower SOA appliances host application-specific ESB logic to validate, transform, and route the messages.

The IEB implementation uses a two-hop CORP-BH IEB integration architecture with Corporate (CORP) IEB nodes and Blue Harmony (BH) IEB nodes. The CORP IEB nodes and Blue Harmony IEB nodes each consist of the DataPower appliances and a HUB queue manager that these DataPower appliances connect to. The HUB queue managers in different IEB nodes are connected to each other via standard MQ channels.

Figure 3 shows the portfolio systems connected to the nearest CORP IEB node via HTTP/S or MQ. The CORP IEB nodes handle the security and connections to the portfolio systems when receiving requests and delivering responses. The Blue Harmony systems interact with the Blue Harmony IEB nodes via HTTP/S or MQ. With MQ clustering enabled within the Blue Harmony environment, Blue Harmony systems, like WebSphere Message Broker (hereafter called Message Broker) and DataStage that connect to the Blue Harmony IEB nodes, can put or get messages on their own queue managers and the MQ cluster would then handle the message delivery. The two-hop IEB configuration greatly simplifies the need for queue objects required to exchange messages, eliminating the need for extraneous remote and for transmit queues and channels.

Figure 3. The IEB in a Blue Harmony environment
The IEB in a Blue Harmony environment

From the endpoints configuration file, the IEB determines that either the request is handled by the current IEB node or the request must flow to another IEB node to be processed. All IEB nodes can route requests to all other IEB nodes for both HTTP and MQ. Message delivery to and from the portfolio and Blue Harmony systems that is configured as a two-hop IEB configuration uses the IEB "servedBy" attribute to determine which IEB node is delivered to the backside destination, as shown in Figure 4.

Figure 4. The two-hop IEB configuration
The two-hop IEB configuration

The IEB endpoint configurations use the ingress and egress processing element nodes to identify which IEB node performs actions, such as message transformation and validation and Administration Reporting Measurement System (ARMS) logging. Transformation and validation and processing of all ARMS logging records in this architecture is performed by the Blue Harmony IEB node. For outbound interfaces configured in IEB, these actions are set up in the ingress processing node. For inbound interfaces coming from the portfolio, these actions are set up in the egress processing node.


Defining the IEB patterns specifications

The following sections outline the IEB interaction patterns that are used in the Blue Harmony project. These are general patterns that are supported by the IEB to describe the interface interactions. For each IEB pattern, an overview diagram and sample endpoint configuration XML entry is provided, along with the required information that is supplied for an interfaced to be implemented in the IEB.

Additionally, for the detailed general integration patterns that exist, all supported integration patterns are matched with the IEB patterns.

Supported integration patterns

The following list is not necessarily all the supported patterns. As the project progresses, there may be new integration patterns that are not included on this list. Additionally, the use of the supported integration pattern in the IEB may or may not indicate the utilization method of the IEB. For example, some interfaces may require that the use of this IEB is simply as a passthru or as a transformation engine.

The following integration patterns are supported by this IEB pattern:

  • BH.I01 – MQ to SAP BAPI/RFC
  • BH.I02 – MQ to SAP IDOC/RFC
  • BH.I11 – Complex ETL from MQ to SAP IDOC/RFC
  • BH.I16 – MQ to MQ
  • BH.O01 – SAP RFC Server to MQ
  • BH.O02 – SAP IDOC/RFC to MQ
  • BH.O13 – Complex ETL from SAP IDOC/RFC to MQ
  • BH.O19 – SAP Flat File to MQ
  • BH.O20 – SAP (ABAP) to MQ
  • BH.O24 – MQ to MQ

IEB-01 Pattern - MQ to MQ

Figure 5 shows the MQ to MQ IEB Pattern. There are two source applications in the diagram. The first application does not support the Rules and Formatting Headers (RFH), and therefore, uses mqAdapter. The second application does support the RFH header, so it puts the message directly to the IEB request input queue.

If the source application requires a reply message, then the target application replies to a common reply queue as shown in Figure 5.

Figure 5. IEB-01 Pattern – MQ to MQ
IEB-01 Pattern – MQ to MQ

IEB-02 Pattern - HTTP to MQ

Figure 6 shows the HTTP to MQ IEB Pattern. It is implied that the HTTP interaction between the source application and IEB will be a request or reply. The reply from the IEB is a simple HTTP 200 ACK reply. If the target application provides a reply message, then it is returned as shown in Figure 6.

Figure 6. IEB-02 Pattern – HTTP to MQ
IEB-02 Pattern – HTTP to MQ

IEB-03 Pattern - HTTP to HTTP

Figure 7 shows the HTTP to HTTP IEB Pattern. It is implied that the HTTP interaction between the source, IEB, and target applications will be a request or reply. The reply from the IEB is a simple HTTP 200 ACK reply. If the target application provides a reply message, then it is returned as shown in Figure 7.

Figure 7. IEB-03 Pattern – HTTP to HTTP
IEB-03 Pattern – HTTP to HTTP

IEB-04 Pattern - MQ to HTTP

Figure 8 shows the MQ to HTTP IEB Pattern. There are two source applications in the diagram. The first application does not support an RFH header, and therefore, uses mqAdapter. The second application does support the RFH header, so it puts the message directly to the IEB request input queue.

It is implied that the HTTP interaction between the IEB and target applications is a request or reply. The reply from the target application is a simple HTTP 200 ACK reply. If the target application provides a reply message, then it is returned to the source as shown in Figure 8.

Figure 8. IEB-94 Pattern – MQ to HTTP
IEB-94 Pattern – MQ to HTTP

Examining the Blue Harmony IEB components

This section describes the individual components of the IEB. The IEB is composed of policies, transformation modules, validation modules, and security and protocol modules to name a few (see Figure 9).

Figure 9. IEB components
IEB components

Endpoints configuration XML component

Figure 10 shows how a route entry is configured for each interface that integrates Blue Harmony with the external IBM enterprise systems. It is then read by the IEB and processed according to the configuration set in its elements. All configured route entries are stored locally on the IEB DataPower appliance in the Endpoints Configuration XML file.

Figure 10. IEB configuration entry
IEB configuration entry

Shown in Figure 9, each interface is uniquely identified by a service Uniform Resource Identifier (URI) in the endpoints configuration file, as shown below:

<url-path>/BH/OTC/DataPrivacy0001/IEB/IN</url-path>

Processing actions, such as message validation, transformation, and ARMS logging, are defined for request and response messages using specific elements. They are then placed either in the ingress-processing section or the egress-processing section of the XML (see Listing 1). The ingress-processing section represents the ingress node, where the request enters or the response exits the IEB cloud. The egress-processing section represents the egress node, where the request exits or the response enters the IEB cloud.

Listing 1. Configurations of the ingress and egress nodes
<ingress-processing>
	<request><callrule>Interface_01_Rule_01</callrule></
     request>
	<msglog msgDomain="O2C" sender="SAP" receiver="HKP2" documentType=
     "CN DD 0008 Shipping Information">
	...
	</msglog>
</ingress-processing>
<egress-processing>
	<request><MQMD>...</MQMD></request>
</egress-processing>

A message logging section (see Listing 2) is defined in the Endpoints Configuration file, which is used by the IEB to produce ARMS transactional logging messages for the following interaction points:

  • Request Message In (inbound request message)
  • Request Message Out (outbound request message after any IEB request transformation)
  • Response Message In (inbound response message)
  • Response Message Out (outbound response message after any IEB response transformation)
  • Error Message (exception handling message for any errors that occur along the way)
Listing 2. Configuring logging emit points
<msglog msgDomain="O2C" sender="SAP" receiver="HKP2" documentType="CN DD 0008 
 Shipping Information">
	<log-url>dpmq://qmgr-group/?RequestQueue=BH.IEB.MSG.LOG</log-url>
	<emit-point type="ack" msgFormat="XML" compression="n" codepage="1208">
     IEB_Req_MsgIn</emit-point>
	<emit-point type="ack" msgFormat="XML" compression="n" codepage="1208">
     IEB_Req_MsgOut</emit-point>
	<emit-point type="ack" msgFormat="XML" compression="n" codepage="1208">
     IEB_Rsp_MsgIn</emit-point>
	<emit-point type="ack" msgFormat="XML" compression="n" codepage="1208">
     IEB_Rsp_MsgOut</emit-point>
	<emit-point type="ack" msgFormat="XML" compression="n" codepage="1208"
	emailAddress="bhieb@us.ibm.com">IEB_Error</emit-point>
</msglog>

The inbound section (Listing 3) of the endpoints configuration XML depicts the two-hop IEB, A two hop implementation is where the request enters an IEB in one zone and is then forwarded to the destination IEB cluster in a different zone. The target or backend destination URL can be an HTTP/HTTPS web service endpoint URL, in which case the IEB performs a synchronous request or response interaction with the backend web service.

Listing 3. Defining the inbound node
<inbound secZone="IEB-BZ">
	<protocol idRealm="US" logPrefix="Interface1-WW">http</protocol>
	<address servedBy="server1.ibm.com>" processHTTPError="n">
	http://server1.ibm.com:7501/BH_SOAP_HTTP_Service1</address>
</inbound>

Or, the URL can be a MQ URL, and IEB puts the message on the defined local for Message Broker or remote queues for DataStage or WebSphere Process Server. The message is processed as an asynchronous datagram (fire-and-forget) message or as a synchronous request or reply (Listing 4). In both cases, the IEB performs success and exception handling as needed.

Listing 4. Defining the destination of the message
<address servedBy="server1.ibm.com" datagram="true">
	dpmq://qmgr-group/?RequestQueue=BH.OTC.WW.INTERFACE1.OUT;
</address>

Processing rules

Processing rules are created in the IEB domain and they define the actions to be performed by the IEB based on the outgoing or incoming traffic. Some of the actions handled by these processing rules are:

  • XML message validation
  • XML message transformation
  • Binary message transformation

A processing rule defined for a specific interface is then added to the route entry for that interface using the callrule element to be processed by the IEB as shown below:

<request><callrule>Interface_01_Rule_01</callrule></request>

Request or response XML message validation

The XSD or WSDL schema is used to validate the request and response XML messages in the IEB. The IEB performs success and exception handling based on validation results. The interface specific .xsd and .wsdl files are uploaded into the IEB domain and are pointed to in the route entry for that interface using the validate element as shown below:

<request><validate type="xsd">local:///BH/OTC/OM0003/Request/Validate/OrderLock.xsd
 </validate></request>

Request or response XML message transformation using XSLT

You can use XSLT to code the message transformation logic for request and response XML messages in the IEB when the transformation logic is simple enough for XSLT to handle and the messages are basically XML to XML formats. The IEB performs success and exception handling based on the transformation results. The interface specific .xsl files are uploaded into the IEB domain and pointed to in the route entry for that interface using the transform element as shown below. You can also add the .xsl files as actions in the processing rules.

<response><transform type="xform">local:///BH/OTC/Interface1/Response/Transform/ack_message.xsl 
 </transform></response>

Request or response message transformation using WTX binary maps

You can use WebSphere Transformation Extender maps to code the message transformation logic to accomplish complex mapping for XML or non-XML messages, such as MQ fixed length. The interface specific .dpa files are uploaded into the IEB domain and pointed to in the route entry for that interface using the transform element as shown below. The .dpa files can also be added as actions in the processing rules.

<request><transform type="xformbin-wtx">local:///BH/OTC/Interface1/Request/Transform/ScheduleUpdate.dpa 
 </transform></request>

SQL datasources

The SQL datasources are created in the IEB domain on the DataPower appliances and are then used by the message validation or transformation code to connect to databases and lookup data, as shown below.

<xsl:variable name="sqlResult"><dp:sql-execute source="BHAPPDB" statement="$selectSQL"/>
 </xsl:variable>

Crypto validation credentials

If the back-side target service configured in the Endpoints XML Configuration file uses the HTTPS protocol, a certificate obtained from the target server is loaded into the DataPower file system and a crypto validation credential is configured within the IEB domain. An example of a secure target service definition used in the configuration file is shown below:

<address servedBy="bhiebdp001.pok.ibm.com">
https://www-304.ibm.com/servers/econfig/act/services/GetAvailability_LeadTime_Port
 </address>

If mutual authentication needs to be set up between IEB and the source system, the client side certificate is obtained from the client system and loaded into DataPower's file system. A crypto validation credential is then configured within the IEB pointing to the client certificate (see the example below). The route entry in the endpoints XML configuration file is configured to specify that mutual authentication is needed between the IEB and the client system.

<protocol idRealm="CERT" certRealm="MA" logPrefix="PS1014">https</protocol>

The IEB also supports authorization processes via LDAP, CERTS, and so on. The Enterprise Directory (ED) or Web Identity (WI) is used to manage consumer's ID and password as shown below:

<protocol idRealm="ED" uri="ldaps://bluepages.ibm.com:636" logPrefix="CM922">https</protocol>

HTTP or HTPPs frontside handlers

The IEB frontside handlers support requests and responses from any application that communicates via HTTP using the SOAP/XML messages. To use the IEB HTTP frontside handler, the application needs to perform a standard web request (HTTP) or a secure SSL request (HTTPS). The IEB web request URL has the following format:

<PROTOCOL>://<HOSTNAME>:<PORT>/<SERVICEURI>

For example:

https://iebserver1.pok.ibm.com:445/BH/OTO/PQ0004/SOVA/OUT

MQ frontside handlers

If the MQ-based applications can send MQ messages to the IEB with a custom MQRFH header, then the IEB uses it to look up the source URI value in the header to identify the interface route entry in the endpoints configuration file. For other MQ-based applications that cannot send MFRFH header value to the IEB, the custom MQ frontside handlers are configured in the mqAdapter gateway service in DataPower to pick up the arriving message and attach an MQRFH header to it, setting its URI path to the name of that custom queue's frontside handler object. The IEB service receives messages from the mqAdapter and processes the message. In this way, the mqAdater service acts as an IEB pre-processor.


IEB validation and transformation processing

There are various schema validation and mapping transformation methods that are available for any interfaces using the IEB. These optional methods can be configured to be invoked during request and response processing and will perform validation and transformation of the incoming request and response message.

If the message fails during validation, then the IEB throws an error exception and produces an ARMS error message.

Note: For validation and transformation processing, the schema and mapping files need to be locally accessible by the IEB, available on its local file system. The files are uploaded to the DataPower appliance and referred to in the Endpoints Configuration file.

IEB validation methods

There are two validation methods available in the IEB: schema or WSDL validation.

  • XSD schema validation: To use this method, the endpoints XML must be set up with the following entry samples:
    • For request message processing:
      <request><validate type="xsd">local:///BH/domain/intid/Request/Validate/SchemaFile.xsd
       </validate></request>
    • For response message processing:
      <response><validate type="xsd">local:///BH/domain/intid/Response/Validate/SchemaFile.xsd
       </validate></response>
  • WSDL schema validation: To use this method, the endpoints XML must be set up with the following entry samples:
    • For request message processing:
      <request><validate type="wsdl">local:///BH/domain/intid/Request/Validate/WSDLFile.wsdl
       </validate></request>
    • For response message processing:
      <response><validate type="wsdl">local:///BH/domain/intid/Response/Validate/WSDLFile.wsdl
       </validate></response>

IEB XSLT transformation processing

For simpler mappings, which do not require extensive transformation logic and are basically XML to XML formats, XSLT can be used by the IEB to accomplish the transformation.

To use this method, the endpoints XML must be set up with the following entry samples:

  • For request message processing:
    <request><transform type="xsd">local:///BH/domain/intid/Request/Transform/transform.xsl
     </transform></request>
  • For response message processing:
    <response><transform type="xsd">local:///BH/domain/intid/Response/Transform/transform.xsl
     </transform></response>

DataPower supports the standard XSLT mapping transformation. Additionally, DataPower supports XSLT extension elements, extension functions, and variables. The extension elements and functions provide experienced XSLT users with a means to take maximum advantage of the capabilities offered by the DataPower XSLT processor. Transformation extensions can be used to develop XSL stylesheets that are tailored to local, site-specific, processing, or security requirements.

For more information about the documentation related to the DataPower XSLT extensions, see the Extension Functions Catalog.

IEB Transformation Extender processing

For more complex mappings, which requires extensive transformation logic and are not necessarily XML to XML formats (such as XML to MQ fixed length), WebSphere Transformation Extender maps can be used by the IEB to accomplish the transformation.

To use this method, the endpoints XML must be set up with the following entry samples:

  • For request message processing:
    <request><transform type="xformbin-wtx">local:///BH/domain/intid/Request/Transform/wtxmap.dpa
     </transform></request>
  • For response message processing:
    <response><transform type="xformbin-wtx">local:///BH/domain/intid/Response/Transform/wtxmap.dpa
     </transform></response>

Note: For the IEB to use a WebSphere Transformation Extender map, the map must be compiled with the DataPower Runtime map setting using WebSphere Transformation Extender Version 8.2.4. Version 8.3 is not support by the current DataPower firmware. This will produce a WebSphere Transformation Extender map with an extension of *.dpa.


IEB ARMS capabilities

The IEB uses a tool called Administration Reporting Measurement System (ARMS) to log the application business log data. This data is retrieved and viewed in the ARMS GUI for analysis, auditing, reporting, and problem determination purposes.

ARMS logging

The ARMS logging capability is built into the IEB functionality and is fully configurable via the IEB endpoint configuration XML. For an interface running through the IEB, configurable parameters can be set for a given ARMS logging message in the msglog section of the endpoints XML file, as shown in Figure 11.

Figure 11. Logging configuration
Logging configuration
  • The msgDomain attribute specifies the Blue Harmony domain in which the interface is running. For example, Order to Cash (O2C), Opportunity to Order (O2O), Master Data Management (MDM), Finance, and so on.
  • The sender and receiver attributes specify the source and target systems in the interface.
  • The documentType attribute identifies a set of interface transaction messages with a name.
  • The log-url element specifies the IEB queue used to send the messages to ARMS for logging.
  • The emit-point elements define the logging events produced, such as the inbound request message (IEB_Req_MsgIn), outbound request message after any IEB request transformation (IEB_Req_MsgOut), inbound response message (IEB_Rsp_MsgIn), outbound response message after any IEB response transformation (IEB_Rsp_MsgOut), and the error message (IEB_Error). The emit points specify attributes, such as message type, message format, message compression, payload codepage, email address, to be notified when the event occurs.

The IEB sets a unique message reference number (MsgRefNum) in each ARMS XML message sent to the ARMS Logger for an interface transaction. This value allows ARMS to associate all the log messages together in a full end-to-end view for a transaction in the ARMS GUI. The IEB uses either a concatenated value of the logPrefix attribute and the current timestamp or an auto generated UUID to create the MsgRefNum. The logPrefix attribute is set in the route entry of the endpoints XML file as shown in Figure 12.

Figure 12. Example of logPrefix
Example of logPrefix

When using the above logPrefix value, a sample MsgRefNum value has a timestamp of OM00032013-07-01T21:16:10.305Z.

If the logPrefix value is not used, then the UUID value looks similar to the following: 6802875a064b90d-1919-4bd9-853c-6043be81b50c.

The MsgRefNum values provided by the IEB are coordinated and shared with other middleware systems, such as Message Broker, to provide a better association of log messages in ARMS. The system that processes the message first to produce the first ARMS XML log message can share the MsgRefNum that was set. The second system can reuse that shared MsgRefNum value instead of generating a new one. This helps the ARMS GUI to associate all logged messages for an interface together into one complete end-to-end view.

For MQ messages that are exchanged between the IEB and Message Broker (either datagram or request/reply), the MsgRefNum value is populated in the usr folder of the MQRFH2 header by the system that processes the message first, as shown in Figure 13.

Figure 13. MQ message headers
MQ message headers

The second system that processes this message reads this value from the MQRFH2.usr.TxnUUID property and uses that value as the message reference number for ARMS.

For HTTP messages that are exchanged between the IEB and Message Broker (request/reply), the MsgRefNum value is populated in the TxnUUID attribute of the HTTP header by the system that processes the message first, as shown in Figure 14.

Figure 14. HTTP message headers
HTTP message headers

The second system that processes this message reads this value from the HTTP header TxnUUID property and uses that value as the message reference number for ARMS.

Reviewing the ARMS GUI

You can use the ARMS GUI to view the transaction messages logged by the IEB (or Message Broker) for analysis, auditing, or problem determination. You can also use an ARMS search screen to enter the values for various search parameters to retrieve the filtered records for the IEB interface transactions. A sample search screen and the search results are shown in Figure 15.

Figure 15. ARMS logging
ARMS logging

The user can view the detailed records for each interface transaction in ARMS by clicking on a log record in the results displayed in Figure 15. The detailed records are grouped by a single message reference number associated with all the inbound, outbound request, response, and error messages that were logged by the IEB or Message Broker for an interface transaction, as shown in Figure 16.

Figure 16. ARMS logging results
ARMS logging results

The user can further click on each log message to see the request, response, and error messages, business or technical attributes, reporting information, and so on.


Implementing the IEB security credentials

With distributed computing comes distributed security. Clients need to be assured that data is safe during message transit. Fortunately, the IEB offers all of the security features that are available to the WebSphere DataPower XI50. Configurations of the following objects remain the same as defined in IBM Redbook: Simplifying Integration with IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise.

  • Crypto Certificate
  • Crypto Firewall Credentials
  • Crypto Identification Credentials
  • Crypto Key
  • Crypto Profile
  • Crypto Shared Secret Key
  • Crypto Validation Credentials
  • Kerberos KDC Server
  • Kerberos Keytab
  • SSH Client Profile
  • SSL Proxy Profile

Because the IEB extends the security functionality of DataPower XI50, you can leverage the value of your existing application, security, and networking infrastructure investments. Security (encryption and signing), reliable delivery, and transaction management are just a few of the benefits offered by the IEB. The real benefit of routing messages through the IEB is that the client applications are completely decoupled from the target applications. This decoupling allows clients connecting to the IEB the freedom of not being concerned with any of the security issues related to target applications. The key is that clients need only create the connection to the IEB. The IEB is responsible for connections to the target application and any security that the connection requires.

The IEB provides a powerful security footprint that enables customers to confidently hand over security tasks and allows them to focus on more important things. This is accomplished in the following ways:

  • The IEB interconnects WWW (and W3) clients with web service applications running in thousands of application server instances in multiple security zones and IBM service delivery centers around the world.
  • The IEB enhances responsiveness to changing business security requirements (for example, ITCS104) by implementing requirements once only in the IEB transport layer. It enables transparent integration of applications and improves operations through management of service access at a single point.
  • The IEB implements opportunistic application security policy enforcement, at the point where the connection enters the IEB, and a "change once" security policy architecture.
  • Finally, the IEB implements security through various frontside and backside configurations called crypto validation credentials, which are detailed in IBM Redbook: Simplifying Integration with IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise.

Conclusion

As mentioned earlier, Blue Harmony is a key element to IBM's strategic transformation to the globally integrated enterprise. The fact that Blue Harmony, a project that produces many thousands of request every minute, has decided to use the IEB as part of its topology speaks volumes about its reliability. Reliability is important, but the benefits of using the IEB have impacted service providers and consumers. This allows external applications to focus on the business and off loading many of the chores inherent to enterprise messaging to the IEB.

The IEB is much more than just a DataPower appliance. The IEB configuration makes implementing, modifying, and maintaining interfaces easier and less time consuming. It is a pure DataPower solution. Interfaces routed through the IEB gain logging, auditing, transformation, mediation, and security to name a few of the benefits. Adding these functionalities to any system would be costly in terms of economics and man hours, but an application routing its messages through the IEB gets it all and without the overhead of added upfront development cost.

Resources

Learn

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=949726
ArticleTitle=Using WebSphere DataPower as the foundation of the IBM ESB Backbone
publish-date=10232013