Level: Intermediate Callum Jackson (callumj@uk.ibm.com), Software Engineer, IBM
Brian Hulse (brian_hulse@uk.ibm.com), Software Engineer, IBM
24 Apr 2008
Part
1 and Part
2 of this three-part series introduced you to the new
aggregation capabilities in IBM® WebSphere® Enterprise Service Bus V6.1. Now learn the
best practices to follow when using the new aggregation capabilities. This article,
the third and final installment in the series, describes four core patterns that you can apply to different business scenarios to design the majority of aggregation mediation applications.
Introduction
Each of the four patterns covered in this article has been
applied to a real-life business scenario with the implementations supplied as
project interchange files to help in the discussion (see the Downloads section to access these
files). Please note
that you should take these as blueprints on which to base your own
business scenarios, because they guide you on the key points to consider.
General concepts
WebSphere Enterprise Service Bus V6.1 supports a flexible mechanism to construct aggregation scenarios,
allowing the mediation developer to combine primitives in a number of different
ways to produce a solution. However, when designing flows that use aggregation,
you must pay attention to the:
- Usage of primitives in the correct pattern.
- Creation and usage of shared context.
- Correct usage of transformation support.
Each of these is highlighted in the four basic patterns described in this
article, which are:
-
Aggregation of data from multiple sources: This pattern is useful if
the results from multiple service invocations, database lookups, and so on need to be
combined into a single message.
-
Batch processing with message enrichment: This pattern is useful if the
message to be processed contains a repeating element, and each element needs to
be enhanced by acting on each one separately.
-
Batch processing requiring embedded aggregation: This pattern is useful
if the scenario is the basic batch processing pattern described in the second
pattern (above), but the
enhancement requires information from multiple sources. In this way, it's a
combination of the first and second patterns.
-
Nested aggregation: This pattern is useful if the scenario is the basic
batch processing pattern described in the second pattern, but the repeating elements themselves
contain repeating elements.
Scenario 1: Aggregation
of data from multiple sources
This scenario is based on an insurance quote comparison Web site that involves a
Web interface that allows users to specify details of their quote. A business
processing engine controls the process and submits a single request to an
aggregation mediation, which submits this to multiple insurance firms. The
responses are then aggregated into a single response for the business process to
manipulate before displaying to the user. This scenario was built step by step in
the first
article in this series; this section discusses the key design
considerations for the scenario. (Download the Scenario 1 files.)
Design the
aggregation
The input into the flow is the InsuranceQuote
interface, which specifies a Quote
business object with the relevant data. The first step in planning the aggregation
architecture is to determine if all data required to invoke the subsequent
services are available in the inbound message or if message enrichment is
required. In this scenario, the Quote object contains all the information for both
services. If this wasn't the case, then you'd use a combination of this and the
batch processing with message enrichment patterns. In fact,
for this particular scenario, the interface for the insurance back-end service takes
the same business object as does the mediation interface. This means that the
transformation prior to the Service Invoke primitives are trivial; however, in more
realistic situations, a more complex transformation is normally required.
The mediation flow includes a Fan Out primitive (operating in once mode)
receiving the Quote message as input and branching the logic to each insurance
back-end service (InsuranceCompanyA and InsuranceCompanyB). Before each
Service Invoke primitive, an XSLT converts the body of the message to the
correct format, and immediately after the Service Invoke, a Message Element Setter
stores the response within the Shared Context for later retrieval when building
the aggregated response. These two branches are then collected by a Fan In
primitive (operating in count mode). When the second message arrives at the Fan In
primitive, its output terminal fires with this message, whereupon a final XSLT
transforms the values within the Shared Context into the aggregated response
message. This is illustrated in Figure 1.
Figure 1. Insurance quote flow
In this case, the Shared Context is straightforward, containing the parts of the
response messages (from the two back-end insurance services) that need to be stored
to generate the aggregated response (see Figure 2).
Figure 2. Shared Context for
insurance quote
The final XSLT then builds the response message that's sent directly back; this
scenario doesn't include a response flow, because all outbound requests are completed
within the request flow of the mediation flow. This design was chosen because there was
no obvious single call out; therefore, it made sense to combine the logic within
one flow.
This is a very simple usage pattern for aggregation, but its simplicity belies
its power, as you can see later in this document when it's combined into a more
complex scenario.
Scenario 2: Batch
processing with message enrichment
This scenario is based on a store ordering system, where a salesperson can submit a
request for a number of orders and have them dispatched. The original message
coming into the mediation flow identifies the customer for each order via a
customer ID. Thus, the order request message needs to be enriched with each customer's
shipping details so the dispatch step has all the information it needs to
succeed. This enrichment is done within the request flow by calling out to a
customer information system, implemented using a Service Invoke primitive. The
dispatch system is implemented here as a callout node. (Download the Scenario 2 files.)
Design the
aggregation
The key here is that the ordering and dispatch systems are implemented to be
batch systems, where multiple records arrive within the same request message.
However, the customer information system is implemented to receive a single
record—thus introducing the need for aggregation—processing each of the
occurrences of the repeating element in the request message individually and then
building a batch message appropriate to the dispatch system.
To explore how this basic scenario is built, you must first examine the
business objects used (see Figures 3 and 4).
Figure 3. The basic business
objects
Figure 4. The batch business
objects
The sales request is a batched array of SalesOrder
objects tagged by a batchID.
Figure 5. Request objects
Figure 6. Response objects
The dispatch system takes a batch of DispatchOrder objects and returns the
associated DispatchStatus objects.
The flow overview
You'll look at each part of the flow, as depicted in Figure 7, including the use of the
Fan In and Fan Out primitives along with any mappings needed within the overall
flow.
Figure 7. Batch-processing flow
However, before doing that, you must first understand how to design a useful
Shared Context for this scenario.
Shared Context
The design of the Shared Context is pivotal to a successful aggregation design, and
this flow is no different. The goal is to build an array of DispatchOrder objects so that
a message can be created to send to the dispatch system. Each DispatchOrder is
created from data in the original SalesOrder and enhanced with further data from
the customer information system. After this is done, you need to build up an aggregated
picture of all of these individual orders by moving them into an array of
DispatchOrder objects. This leads to an obvious pattern
for the Shared Context (see Figure 8).
Figure 8. Shared Context for batch
processing
The current field is used in each iteration of the Fan Out to create the
DispatchOrder for the customer in question. This is then appended to the
aggregationStore, which is effectively a running total
of the aggregation to date.
Note: This is a standard pattern for the design of Shared Context
usage in aggregation and isn't again throughout this article.
The flow detail
Now take a look at the flow in detail, starting with the aggregation primitives,
Fan Out and Fan In. The Fan Out is set to iterate over the repeating elements in the
request message, which in this case are the SalesOrders. Equally, the Fan In is set
to complete when all of the repeating elements have been handled (see Figure 9).
Figure 9. Fan In details for batch
processing
Following the Fan Out is a Message Element Setter called setBatchID, which is used to
copy the batchID from the request message into the
Shared Context (see Figure 10).
Figure 10. Message Element Setter
to set the batchID
This simply seeds the context ready for the creation of the dispatch message at
the end of the flow.
Before the customer information system can be invoked, you have to convert the
message body into the correct format; you do this using the XSLT primitive called
convertToCustomer. This is a simple transform that copies the customerID for the
current iteration into the service invocation message. This is achieved by
creating inline maps for the context and headers and a simple move of the
customerID from the FanOutContext occurrence field into the customerID field in
the body of the output message.
Figure 11. Transformation for
customer information system
The customer information system is invoked using a Service Invoke primitive called
customer and returns a message containing the relevant customer details in the
body. These need to be copied into the Shared Context, which is done in the XSLT
primitive called storeResult.
Figure 12. Transformation to
store response from customer information system
Notice that in this particular example, the current iteration's output
is built from the customer details returned from the service invocation in
addition to information copied from the current iteration's occurrence, namely the
itemID and the quantity. Initially, this transform was built by creating inline
maps for all matching fields; however, since you're copying specifically into the
current object of the Shared Context, you must remember to remove the inline map for
this particular object, as shown in Figure 13.
Figure 13. Inline map
transformation for Shared Context
Now you have the complete results from the current iteration in the current object
of Shared Context, but you have to make a copy of these before you start the next
iteration. Consequently, the next primitive in the flow is a Message Element Setter,
called aggregateResult, which appends the current object into the aggregationStore
in Shared Context.
Figure 14. Append response to
aggregationStore
The aggregation flow used for each element is now complete. All you need to
do is convert the aggregationStore into the correct message for the
dispatch Callout node. This defines what occurs when all of the iterations are complete
and the Fan In fires its out terminal. You use the XSLT primitive called convertToDispatch
to do this.
Figure 15. Transformation of
aggregationStore into request
As you can see, the key to the whole aggregation flow is one of building up the
required information in each iteration in a current object followed by
appending current into the running total aggregationStore in the Shared Context.
Scenario 3: Batch
processing requiring embedded aggregation
This scenario involves a combination of the two previously described examples,
having an aggregation of data from multiple sources embedded within a batch
processing aggregation. The scenario is a simple personnel-based application that
requests that the payslips for a number of employees be sent out. The back-end
service is represented by a Callout node and requires full employee records, rather
than the employee IDs, which arrive at the mediation flow. These employee records
are created by the mediation from two sources, namely an HR system and a payroll
system. The former stores information concerning the employee's name, address, and
leave entitlement, while the latter contains information regarding salary and
department. (Download the Scenario 3 files.)
Design the
aggregation
The scenario makes use of embedded aggregation, because it maps well to the data
expected by the two services to be invoked in the mediation flow. In other words,
they both
expect a single data record on each invocation, but it also allows the full
mediation flow to batch requests; this makes use of repeating elements. This is
best described by looking at the data objects used by the mediation flow (see
Figure 16).
Figure 16. Request and response
business objects
The input to the flow is an array of employee identifiers, while the output is an
array of status objects, indicating whether the payslip processing succeeded or
not. However, each of the services invoked from within the mediation flow can only
handle a single record, as shown in Figure 17.
Figure 17. Business objects for
back-end system interaction
So the outer aggregation iterates through all of the repeating elements, while
the inner aggregation builds a response aggregated from the two service
invocations. As with the previous example, when a response has to be created from
repeating elements, the specification of the Shared Context is key to a successful
flow design. In this case, on completion of successful calls to the two services,
you need to combine the HRRecord and PayrollRecord results into a single
EmployeeRecord. Moreover, this single response needs to be aggregated into an
array of records to satisfy the interface for the Payslip service, invoked via the
Callout node. Consequently, the same model for Shared Context as previously described is
adopted (see Figure 18).
Figure 18. Business objects for
back-end system interaction
The current field is used to create a single record for each instance of the
repeating element, which is then appended to the aggregationStore, a running total
of the overall flow.
After the data has been aggregated and batched, the Payslip service is called,
which takes an array of EmployeeRecord objects and returns an array of
PayslipStatus objects (see Figure 19).
Figure 19. Payslip service
business objects
Now, considering the middle part of the flow, let's look at Figure 20.
Figure 20. Embedded aggregation
flow
This embedded aggregation is similar to one previously described and deals with
a single record being sent to two systems via service invocations. This is
achieved by the XSLT transforms, convertToHR and convertToPayroll, taking the
single repeating element occurrence in the FanOutContext and mapping the relevant
field into the message most appropriate for the Service Invoke call. The parts of
the flow that bracket this are shown in Figure 21.
Figure 21. Outer aggregation
splitting
What you see above fires repeatedly until all of the repeating elements are
exhausted. The result is shown in Figure 22.
Figure 22. Outer aggregation
Fan In
This then converts the aggregated results in Shared Context into an appropriate
request message for the Callout node.
All of the strategies used in XSLT mapping described in the previous scenarios
also apply here.
Scenario 4: Nested
aggregation
This scenario is based on a supermarket company, where each outlet sends
multiple restocking orders to a central company-wide system that batches the
requests into a single message. This single message is submitted to the mediation
flow where, for each order and for each branch, a separate invocation to the order
system should occur. (Download the Scenario 4 files.)
Design the
aggregation
The key here is that the ordering system is implemented to accept a single
record, whereas the input into the mediation module is a nested batch message.
This introduces the need for aggregation, and in this case nested aggregation,
because the batch request message is repeated on an outlet basis and then repeated
on an order basis. Orders are sent individually, the response from each order
being stored for later aggregation into a single response message, which shows the
status of all the elements in the batch request.
To explore how this basic scenario is built, you must first examine the
business objects used (see Figure 23).
Figure 23. The basic business
objects
The GroupOrders request is a batched array of Outlets tagged by an ID, which
itself is a batched array of Orders. This nested structure allows for a nice fit
to the nested aggregation structure, which is explained later in this article.
Figure 24. Request business
object structure
The GroupOrderResult has a similar nested structure to the request. The
GroupOrderResult is a batched array of OutletResults tagged by an ID, which itself
is a batched array of OrderResults.
The flow overview
In this section, you'll look at each part of the flow, including the use of each
Fan In and Fan Out
primitive along with any mappings needed within the overall flow (see Figure 25).
Figure 25. Flow overview
However, before doing that, you must first understand how to design a useful
Shared Context for this scenario.
Shared Context
As stated earlier, the design of the Shared Context is essential to a successful
aggregation design, and within a nested aggregation it's even more critical. The
goal is to store the response from each Order into a structure that can then be
converted into a single response message. The nested aggregation structure can be
considered analogous to a Russian doll, where the initial request is at a
company-wide scope, the first aggregation is at the Outlet scope, and the second
aggregation is at the Order scope (see Figure 26).
Figure 26. Aggregation structure
Understanding the aggregation structure helps in defining the correct
Shared Context, as normally there's a mapping. The logic pattern within the
mediation flow is:
- Aggregate on an
Outlet basis.
- Aggregate on an
Order basis.
- Invoke the
Order service.
- Store the current
Order service within the current Outlet
response.
- Repeat for each
Order associated with the Outlet.
- Store the current
Outlet in the company-wide aggregation store.
- Repeat for each
Outlet.
- Convert all results into a response message.
This provides the requirement for the Shared Context to be able to store the
overall company-wide, the current Outlet, and the current Order response. Figure
27 illustrates
what this looks like.
Figure 27. Shared Context business
objects
The flow detail
Now looking at the flow in detail, it's separated into three sections:
- Company-wide scope
-
Outlet scope
-
Order scope
Figure 28. Company-wide section
of flow
At the company-wide scope (see Figure 28) the request message is passed unmodified into the
Outlet aggregation, so there's nothing interesting at this stage. On exiting the
Outlet aggregation, the response message needs to be generated from the
Shared Context. You can accomplish this by using either an XSLT or a BOMapper
primitive. In this case an XSLT primitive has been used to map from the
aggregationStore within the Shared Context to the response body, and is illustrated
in Figures 29 through 31.
Figure 29. Transformation to
response message
Figure 30. Response message:
OutletResults submap
Figure 31. Response message:
orderResults submap
The important aspect to highlight at this stage is the simple transformation
required. Careful consideration of the Shared Context structure has allowed for a
straightforward XSLT without the requirement of any custom XSLT.
Figure 32. Outlet
The outlet scope provides the looping capabilities over the array of Outlets
provided within the request message. The OutletIterator (Fan Out primitive) is set
to iterate over the repeating Outlet elements in the request message. Equally, the
Fan In is set to complete when all of the repeating elements have been handled
(see Figure 33).
Figure 33. Fan In for outlet
aggregation
Following the Fan Out is a Message Element Setter called StoreOutletID, which is used
to copy the ID from the request message into the Shared Context (see Figure 34).
Figure 34. Store the outlet ID
This simply seeds the context ready for appending the current Order to the outlet
during the Order aggregation.
Following the Fan In is a Message Element Setter called StoreOutletResult, which
appends all of the results for a single outlet into the aggregationStore by using
the append option (see Figure 35). In addition, the current Outlet is removed from the
Shared Context so that it's ready for the next iteration.
Figure 35. Append the outlet to
the aggregationStore
The Order scope provides the looping capabilities over the array of Orders for
each Outlet within the request message. The OrderIterator (Fan Out primitive) is
set to iterate over the repeating Orders (contained within the FanOutContext of
the first Fan Out primitive).
Figure 36. Order aggregation
Equally, the Fan In is set to complete when all of the
repeating elements have been handled (see Figure 37).
Figure 37. Fan In for order
aggregation
Following the Fan Out is an XSLT called TransformToOrder, which converts the new
FanOutContext (containing the current Order element) into the request message for
the Order service (see Figure 38).
Figure 38. Transformation to
order request format
The Service Invoke then invokes the Order service and, following this, a
Message Element Setter called StoreResult stores the result into the currentOrder
element within the Shared Context.
Figure 39. Store the current
order into the Shared Context
The Order result that has been stored within the Shared Context needs to be stored
within the Outlet result Order array. Therefore, another Message Element Setter
called AppendStore copies the currentOrder into the Outlet
Order result array.
Figure 40. Append the current
order to the Shared Context
There's no need to remove the currentOrder object from the Shared Context at this
aggregation level, as the values are overwritten during the next iteration.
As you can see, the aggregation functionality directly maps to the structure of
the inbound data, as does Shared Context. In this case, you have repeating elements
within repeating elements, so the aggregation is two-fold and nested.
Other things to consider
Having looked at the useful patterns in aggregation, there are now other points
to consider when designing a mediation flow using aggregation in WebSphere
Enterprise Service Bus V6.1.
-
Timeout property for Fan In: The Fan In primitive has an associated
timeout property that requires a little explanation so you can use it
successfully. The timeout period starts when the associated Fan Out fires an
output terminal for the first time. If a message arrives at the Fan In in
terminal after this timeout period, it's considered as being late, and the Fan In
fires its incomplete terminal. If a mediation primitive is badly behaved, for
example, if a DB Lookup primitive is deadlocked, the execution of the flow is
also deadlocked; this situation is not mitigated by the Fan In timeout.
Consequently, the timeout function should be considered as a way of trapping
late messages rather than a way of terminating the aggregation after a specified
period.
-
SMO body considerations: The mediation flow programming model
guarantees that, with the explicit exception of the Shared Context, separate flow
branches are isolated and have their own SMO. Therefore, the body of a message
should not be used to store any data that may be needed outside of the
Fan Out/Fan In boundary; it can't be relied on. Any data to be used in this way
should be stored within the Shared Context, which is supplied for this
purpose.
-
Use of the response flow within aggregation: Even with the enhanced
intra-flow service invocation support provided by the Service Invoke primitive,
response flows play a key role within a mediation. It's important to design the
mediation to be easy to maintain and provide a sensible flow of data across the
whole mediation flow. If there's a logical single invocation within the
mediation flow, it's sensible to use the callout node within the request flow,
and then use the response flow for any additional mediation processing. The
usage of response flows is demonstrated in the previously described
scenarios.
 |
Conclusion
This article introduced you to some common aggregation usage patterns and
explored how these patterns can be realised using WebSphere Enterprise Service
Bus V6.1. This the final in a series
of three articles, and, upon completion, you should understand
the basic building blocks that make up aggregation scenarios as well as how you
can use these
to create realistic business flows.
Downloads | Description | Name | Size | Download method |
|---|
| Scenario 1 sample files | InsuranceQuoteScenario1.zip | 28KB | HTTP |
|---|
| Scenario 2 sample files | StoreMediationScenario2.zip | 60KB | HTTP |
|---|
| Scenario 3 sample files | EmployeeMediationScenario3.zip | 72KB | HTTP |
|---|
| Scenario 4 sample files | NestedAggregationScenario4.zip | 37KB | HTTP |
|---|
Resources Learn
Get products and technologies
- Innovate your next development project with
IBM trial software, available for download or on DVD.
Discuss
About the authors  | 
| | Callum Jackson is a software engineer for WebSphere Enterprise Service Bus based in Hursley, United Kingdom, and has worked in the product area since 2005. Prior to this he worked in ISSW on SOA applications for the telecommunication industry. |
| 
| | Brian Hulse is a senior software engineer for WebSphere Enterprise Service Bus based in Hursley, United Kingdom. He has worked in software development at Hursley for 20 years, the last five of which have been in the SOA arena. |
Rate this page
| 
