Aggregation design patterns and performance considerations in WebSphere Enterprise Service Bus V7.5
Aggregation is an important enterprise service bus (ESB) pattern. It enables a single inbound request to map into multiple outbound service invocations, the responses from which can be aggregated into a single response to the original request. But before you develop mediations using aggregation design patterns with IBM® WebSphere® ESB and IBM Integration Developer, you need to take several performance considerations into account. This article describes those performance considerations so that you can create an effective and efficient aggregation solution.
This section describes the core mediation primitive components that are used in the development of aggregation design patterns.
Fan Out mediation primitives
The Fan Out mediation primitive is used at the start of an aggregation block. Two modes are currently supported:
- Once mode -- Provides the ability to branch the mediation flow
- Iterate mode -- Provides the ability to iterate over a repeating structure within the service message object (SMO)
You should use Once mode to branch the mediation flow for aggregation design patterns where, for example, and inbound message contains a structure that needs to be forwarded on to two services. Use the Iterate mode for aggregation design patterns where a service needs to be invoked multiple times based upon a repeating structure within the SMO.
Figure 1. Fan Out mediation primitive
The primitive is configured for Once mode by default. When developing an aggregation pattern using the Iterate mode, select For each element in XPath expression and specify the location of the repeating structure within the SMO. The following XPath defines the location of the Fan Out context within the SMO:
The Fan Out context is used in Iterate mode to store the current occurrence of the repeating structure and the index of the occurrence within the SMO.
Fan In mediation primitives
The Fan In mediation primitive is used at the end of an aggregation block, and partners with a corresponding Fan Out primitive. Fan In supports three types of decision points that control when the output terminal of the Fan In primitive is fired. When the specified decision point has been met, the last message that was received is propagated to the output terminal. Here are the three types of decision points:
- Simple count -- Defines the number of messages to be received at the input terminal before firing the output terminal.
- XPath decision -- Output terminal is fired when the XPath expression evaluates to true.
- Iterate -- Output terminal is not fired until the input terminal has received all messages produced by the corresponding Fan Out mediation primitive.
The XPath decision and Simple count decision points may be met more than once in an aggregation solution, resulting in the output terminal being fired multiple times. The Iterate decision point causes the output terminal to be fired only once.
Figure 2. Fan In mediation primitive
Service Invoke mediation primitives
The Service Invoke mediation primitive is used between the Fan Out and Fan In mediation primitives to invoke required services required. Additional functionality was introduced in WebSphere ESB V7.5 and IBM Integration Developer V7.5. You can select the new Message Enrichment Mode when you add the Service Invoke mediation primitive to the canvas in IBM Integration Developer, as shown below:
Figure 3. Service Invoke mediation primitive -- selecting Message Enrichment Mode
When Message Enrichment Mode is selected, the additional properties shown below are made available to the Service Invoke mediation primitive within IBM Integration Developer:
Figure 4. Service Invoke mediation primitive -- Message Enrichment Mode properties
An XPath statement is defined for the Input parameter to identify the location of the structure within the input SMO to be used to invoke the defined service. A second XPath statement is defined for the Output parameter to identify the location of the structure within the output SMO to be used to store the response. This functionality is applicable only if the structures required for the request and response for the given service are available within the inbound and outbound messages and are addressable by single XPath statements.
This section describes two design patterns that correlate to the two modes of operation supported by the Fan Out mediation primitive:
- Branching design pattern
- Iterate design pattern
Branching design pattern
This section describes the differences and advantages of the new Message Enrichment Mode of the Service Invoke mediation primitive.
The following example of a Branching aggregation design pattern describes a banking scenario where two services are invoked for a single inbound request. The inbound request contains information pertaining to a given account, and a Message Element Setter mediation primitive stores a RequestID field in the transient context of the SMO prior to the request entering the aggregation block. The Fan Out mediation primitive is configured in Once mode and the mediation flow is split into two branches: the first branch invokes a service to retrieve a summary of the account information, and the second branch invokes a service to retrieve the transaction history of the account.
Prior to the invocation of each service, a BOMapper mediation primitive converts the message to the correct format, and a BOMapper mediation primitive immediately following each Service Invoke mediation primitive then stores the given responses in the shared context of the SMO. The output terminal of the Fan In mediation primitive is fired when the two responses are received on the input terminals, and a final BOMapper mediation primitive aggregates the information stored within the shared context and transient context to create a response to the initial request:
Figure 5. Branching aggregation design pattern
Service Invoke Message Enrichment Mode
New functionality in WebSphere ESB V7.5 and IBM Integration Developer V7.5 enables you to simplify the Branching design pattern. Using the new Service Invoke functionality in the Branching aggregation design pattern removes the need for the BOMapper mediation primitives on either side of the Service Invoke mediation primitives, as shown below:
Figure 6. Branching aggregation design pattern -- Service Invoke Message Enrichment Mode
Integrating this functionality within the Service Invoke mediation primitive has the added benefit of a slight improvement in runtime performance by removing the overhead from the additional BOMapper mediation primitives otherwise required.
Iterate design pattern
The following example of an Iterate aggregation design pattern describes a banking scenario where a service is invoked multiple times for a single inbound request. The inbound request contains a repeating structure, each occurrence of which contains information pertaining to a particular account for a customer, and a Message Element Setter mediation primitive stores a RequestID field in the transient context of the SMO prior to the request message entering the aggregation block. The Fan Out mediation primitive is configured in Iterate mode and therefore a message is produced for each occurrence of the AccountInfo element within the inbound request message. A service is invoked for each occurrence to retrieve a summary of the account information.
Prior to the invocation of the service, a BOMapper mediation primitive replaces the multiple occurrences of the element within the message body with the single occurrence for the current iteration stored within the FanOutContext of the SMO. A Message Element Setter mediation primitive immediately following the Service Invoke mediation primitive then appends information from each response to an array of structures defined within the shared context of the SMO. The output terminal of the Fan In mediation primitive is fired after the Fan Out mediation primitive has iterated over all occurrences of the repeating structure and the Fan In mediation primitive has received all corresponding messages on the input terminal. A final BOMapper mediation primitive aggregates the information stored within the shared context and transient context to create a response to the initial request:
Figure 7. Iterate aggregation design pattern
This section describes performance considerations that you need to understand when creating aggregation solutions.
XSL transformation vs. BOMapper mediation primitives
Aggregation patterns often require multiple requests to be built from a single input request and multiple responses to be aggregated into a single response. In the examples above, the BOMapper mediation primitive is used to perform the transformations in the mediation flow. If the functionality supported by the BOMapper mediation primitive is sufficient for the data transformations, then runtime performance will be improved by using the BOMapper mediation primitive in place of the XSL Transformation (XSLT) mediation primitive, with the performance improvement proportional to the size of the message being processed. The XSLT mediation primitive provides additional functionality not supported by the BOMapper mediation primitive, so there may be circumstances when using BOMapper is not feasible.
The following chart illustrates the runtime performance for three variations of the Branching aggregation design pattern scenario illustrated in Figure 5:
- Branch-XSLT -- XSLT mediation primitive used for all transformations
- Branch-BOMapper -- BOMapper mediation primitive used for all transformations
- Branch-Enrichment -- Using the new Message Enrichment Mode of the Service Invoke mediation primitive and BOMapper mediation primitive for final aggregation (shown above in Figure 6)
Figure 8. Branching aggregation design pattern -- runtime performance
The results show that significant runtime performance improvements can be achieved by using the BOMapper mediation primitive instead of the XSLT mediation primitive in aggregation solutions, when feasible. The results also show the additional improvements in runtime performance that you can achieve with the new Message Enrichment Mode functionality in WebSphere ESB V7.5 and IBM Integration Developer V7.5 for the Service Invoke mediation primitive.
Iterate design pattern -- Batch processing considerations
When using the iterate design pattern, consider the impact that processing a large batched request may have on other applications hosted on the same server. Processing requests with a large number of repeating elements, such as sales orders, can use lots of system memory, which will remain in use until all processing is complete. Depending on the response times of the services invoked within the aggregation, block processing may tie up system resources for long periods. If the same server used for processing batch requests is also being used to host applications that need to respond quickly to requests, there may be service degradation while the batched request is being processed. To avoid this problem, process large batched requests on a separate server or during periods of low system usage.
For the reasons explained above, it is important to minimize the amount of memory used when processing large batched request messages. When using the batched processing pattern, the body of the original request message is rarely required inside the aggregation block, because the Fan Out mediation primitive stores the current occurrence of a repeating element in the Fan Out context of the SMO. At the end of the aggregation block, the response data is held in the shared context and is used to build a new response message body. If the request is large, then you can significantly reduce memory usage and processing time by deleting the message body immediately after the Fan Out mediation primitive by using the Message Element Setter mediation primitive, or by using a BOMapper mediation primitive and not propagating the body of the SMO.
Figure 9. BOMapper mediation primitive -- not propagating /body
Additional memory considerations
After an aggregation block is completed, the responses that have been stored in the shared context are combined into a new response message body, which is performed immediately after the Fan In mediation primitive. At this point, the contents of the shared context are not typically required, and performance is improved if the shared context is not propagated through the remainder of the flow. If you do not require any of the fields in the context, then leave the context unmapped. Otherwise, expand the context element and map only those parts that are required in the rest of the mediation flow.
Figure 10. BOMapper mediation primitive -- not propagating /context
Aggregation is an important and powerful ESB pattern, but you need to understand and evaluate several performance considerations before developing and deploying WebSphere ESB mediation flows that use it, including:
- Runtime performance of the BOMapper and XSLT mediation primitives
- The effect of large batch processing scenarios on requests to co-located services
- Message data propagation through the aggregation block and mediation flow
- WebSphere ESB resources
- WebSphere ESB information center
A single Web portal to all WebSphere ESB documentation, with conceptual, task, and reference information on installing, configuring, and using WebSphere ESB.
- WebSphere ESB developer resources page
Technical resources to help you use WebSphere ESB as a flexible connectivity infrastructure for integrating applications and services to support an SOA.
- WebSphere ESB Development Guide
The Development Guide manual in PDF format.
ESB product page
Product descriptions, product news, training information, support information, and more.
- WebSphere ESB documentation library
WebSphere ESB product manuals.
- WebSphere ESB FAQs
Common questions about WebSphere ESB and its relationship to other ESB products.
- WebSphere ESB support
A searchable database of support problems and their solutions, plus downloads, fixes, and problem tracking.
- Tuning the IBM virtual machine for Java
This topic from the WebSphere Application Server information center provides detailed instructions and information on tuning the IBM virtual machine for Java.
- WebSphere ESB information center
- WebSphere resources
- developerWorks WebSphere developer resources
Technical information and resources for developers who use WebSphere products. developerWorks WebSphere provides product downloads, how-to information, support resources, and a free technical library of more than 2000 technical articles, tutorials, best practices, IBM Redbooks, and online product manuals.
- developerWorks WebSphere application integration developer
How-to articles, downloads, tutorials, education, product info, and other resources to help you build WebSphere application integration and business integration solutions.
- developerWorks WebSphere business process management
WebSphere BPM how-to articles, downloads, tutorials, education, product info, and other resources to help you model, assemble, deploy, and manage business processes.
- Most popular WebSphere trial downloads
No-charge trial downloads for key WebSphere products.
- WebSphere forums
Product-specific forums where you can get answers to your technical questions and share your expertise with other WebSphere users.
- WebSphere on-demand demos
Download and watch these self-running demos, and learn how WebSphere products and technologies can help your company respond to the rapidly changing and increasingly complex business environment.
- developerWorks WebSphere weekly newsletter
The developerWorks newsletter gives you the latest articles and information only on those topics that interest you. In addition to WebSphere, you can select from Java, Linux, Open source, Rational, SOA, Web services, and other topics. Subscribe now and design your custom mailing.
- WebSphere-related books from IBM Press
Convenient online ordering through Barnes & Noble.
- WebSphere-related events
Conferences, trade shows, Webcasts, and other events around the world of interest to WebSphere developers.
- developerWorks WebSphere developer resources
- developerWorks resources
downloads for IBM software products
No-charge trial downloads for selected IBM® DB2®, Lotus®, Rational®, Tivoli®, and WebSphere® products.
Join a conversation with developerWorks users and authors, and IBM editors and developers.
cloud computing resources
Access the IBM or Amazon EC2 cloud, test an IBM cloud computing product in a sandbox, see demos of cloud computing products and services, read cloud articles, and access other cloud resources.
- developerWorks tech briefings
Free technical sessions by IBM experts to accelerate your learning curve and help you succeed in your most challenging software projects. Sessions range from one-hour virtual briefings to half-day and full-day live sessions in cities worldwide.
- developerWorks podcasts
Listen to interesting and offbeat interviews and discussions with software innovators.
- developerWorks on
Check out recent Twitter messages and URLs.
- IBM Education Assistant
A collection of multimedia educational modules that will help you better understand IBM software products and use them more effectively to meet your business requirements.
- Trial downloads for IBM software products