Aggregation

Aggregation is a powerful and important enterprise service bus (ESB) pattern. An inbound request might map into several individual outbound requests. The responses from these requests can be used to enrich a final request to a service or can be aggregated into a single response for the original request.

Common scenarios are:

A car comparison insurance quote system where a single request is submitted to an ESB that causes multiple insurance service providers to be queried before the best quote is returned.
A holiday booking request arrives at the ESB and needs to be forwarded to separate hotel, flight and car hire booking systems, before a combined response can be returned.
A store ordering system, where a salesperson can submit a request for a number of orders and have them dispatched. The original message coming into the ESB identifies the customer for each order via a customer ID. So, the order request message needs to be enriched with each customer's shipping details so that the dispatch step has all the information it needs to succeed.

When creating aggregation solutions using WebSphere ESB there are several core parts of the programming model that you must understand before going into the details of the patterns:

The Fan Out mediation primitive
The Fan Out context
The Fan In mediation primitive
The Service Invoke mediation primitive
Aggregation block
Shared context
Special considerations

Fan Out mediation primitive

The Fan Out mediation primitive is normally used at the start of an aggregation. It provides the ability to either branch (Once mode) the mediation flow, or fire its output terminals multiple times (Iterate mode), based on the size of a repeating structure within the Service Message Object (SMO). For example, if the inbound message contains a structure that needs to be forwarded onto two services, you can branch the flow at the Fan Out mediation primitive. To call a service multiple times, each with individual requests from a repeating structure, you can use the Iterate mode of the Fan Out mediation primitive. You can configure the two modes from the Fan Out mediation primitive properties view, as shown in Figure 1.

Figure 1. The Fan Out mediation primitive properties view

The mediation primitive is configured for Once mode by default, but this can be changed by selecting the for each element in XPath expression button and specifying the location of the repeating structure within the SMO.

Fan Out context

The Fan Out context is used to store the current item of a repeating element when Fan Out is used in Iterate mode. The Fan Out context contains an integer field containing the index of the occurrence of the repeating element along with an element containing the occurrence itself. For instance, if there was a repeating element of an element called order, the Fan Out context would look like this:

<FanOutContext>
	<iteration>0</iteration>
		<occurrence>
			<order>
				<id>1234567890</id>
			</order>
		</occurrence>
</FanOutContext>

Fan In mediation primitive

The Fan In mediation primitive is a partner to the Fan Out mediation primitive. The Fan Out mediation primitive is used to decide how many messages to generate, whereas the Fan In mediation primitive is used to decide when the aggregation can be considered complete. The Fan In mediation primitive is always partnered with a Fan Out mediation primitive in the same flow (for example, a Fan Out/Fan In mediation primitive pairing can not expand across a request/response flow boundary), and the Fan In mediation primitive acts as a decision point for when to leave the aggregation block, explained later in this section. The Fan In mediation primitive receives a number of messages until a decision point is reached, at which point the last message that was received is propagated to the output terminal. The Fan In mediation primitive can only be used in combination with the Fan Out mediation primitive.

Three types of decision points for the Fan In mediation primitive are supported:

Simple count: the output terminal is fired when a set number of messages is received at the input terminal. However, more messages might be sent out by the Fan Out mediation primitive, because the Fan In mediation primitive count decision point can be reached more than once, resulting in multiple firings of the output terminal.
XPath decision: the output terminal is fired if an XPath expression evaluation of the incoming message evaluates to true. It is possible for the output terminal to fire multiple times.
Iterate: the Fan In mediation primitive waits to receive all messages produced by the corresponding Fan Out mediation primitive. All of the occurrences in the repeating element have been processed.

Figure 2 shows these three options, which are available in the Fan In mediation primitive properties view.

Figure 2. The Fan In mediation primitive properties view

The Fan In mediation primitive has two input terminals (in and stop), two output terminals (out and incomplete) and a fail terminal (fail). The input terminals are wired to accept a message and the other terminals are wired to propagate a message.

The in terminal accepts input messages until a decision point is reached. When a decision point is reached the out terminal is fired. Although the out terminal is fired when a decision point is reached, the in terminal can still accept input messages, under some circumstances. For example, if the count value is reached, the out terminal is fired. However, if the Fan Out mediation primitive is still sending messages, the Fan In mediation primitive can still receive messages.

You can stop a Fan In operation by wiring the preceding mediation primitive to the Fan In stop terminal. The stop terminal causes the incomplete output terminal to be fired, and this stops the associated Fan Out mediation primitive from sending any more messages. The incomplete output terminal is also fired if a timeout occurs. Table 1 describes terminal behaviour further.

Service Invoke mediation primitive

The Service Invoke mediation primitive is used between the Fan Out and Fan In mediation primitives to call the services required. The Service Invoke mediation primitive is discussed in the Invocation of Services chapter.

Aggregation block

When you create an aggregation solution, the Fan Out mediation primitive declares the start of the aggregation and a Fan In mediation primitive symbolizes the end of the aggregation. The logic between these two mediation primitives can be considered a repeating block if the Fan Out is configured in Iterate mode, or branching if configured in Once mode, as shown in Figure 3 Figure 4 Figure 5 Figure 6 Figure 7.

Figure 3. Example of an aggregation block

Figure 4. Example of an aggregation block

Figure 5. Example of an aggregation block

Figure 6. Example of an aggregation block

Figure 7. Example of an aggregation block

Shared context

WebSphere ESB has three user-defined data store contexts within the SMO:

Data stored in the Transient context is available within either a request or response flow.
Data stored in the Correlation context is available in both the request and response flows.
Data stored in the Shared context is available in all branches and executions of an aggregation block.

To highlight the difference Figure 8 shows an example.

Figure 8. Shared context example

In the case of this example, the SMOs passing through branch 1 and 2, and therefore the Transient and Correlation contexts, are different copies. This means that any changes made to the SMO in one branch are not made to the other branch. The Shared context area of the SMO has been designed for sharing aggregation data between the different branches of the flow inside an aggregation block. Access to the Shared context is only available inside the flow which contains the aggregation block that uses it.

Aggregation mediation programming model special considerations

When using aggregation capabilities consider these points:

The Shared context is part of the SMO, and is designed for use inside aggregation blocks because the data put into it is available in all the branches of a flow. The design of the Shared context structure is essential to a successful aggregation design. The goal is to store the response from each service invocation into a structure that can then be converted into a single response message after the aggregation block.
The Shared context is shared across SMOs and should be considered read-only outside the aggregation block.
At development time it is difficult to determine which SMO body type will be emitted from the Fan In mediation primitive output terminal. Mediation primitives after the Fan In mediation primitive should be used to generate a new message body using the information in the Shared context.
If a message arrives at a Fan Out mediation primitive, one or more output terminals of the Fan In mediation primitive will fire (assuming no ServiceRuntimeExceptions occur within the aggregation block, for example, as a result of running a Fail mediation primitive), even if a message is not received by the Fan In mediation primitive. Table 1 highlights this behaviour.
Note: In some circumstances it is possible that none of the messages sent by the Fan Out mediation primitive arrive at the corresponding Fan In mediation primitive. Using Figure 8 to illustrate, consider this example: if the second output terminal on primitives branch 1 and branch 2 are fired, this does not cause the messages to arrive at the Fan In mediation primitive, because there is no wired path. This is represented in Table 1 by the message result "No".

Table 1. Which output terminal is fired from the Fan In mediation primitive
Fan In mediation primitive mode	Message arrives at Fan In mediation primitive	Output terminal fired	SMO of Fan In mediation primitive
Iterate	No	Complete	Message body empty; SMO headers and context are the same as the message received at the Fan Out
Iterate	Yes (in terminal)	Complete	Last message received at the Fan In mediation primitive input terminal
Iterate	Yes (stop terminal)	Incomplete	Last message received at the Fan In mediation primitive input terminal
Simple count/XPath	No	Incomplete	Message body empty; SMO headers and context are the same as the message received at the Fan Out
Simple count/XPath	Yes (in terminal)	Incomplete/Complete *	Last message received at the Fan In mediation primitive input terminal
Simple count/XPath	Yes (stop terminal)	Incomplete	Last message received at the Fan In mediation primitive input terminal
Iterate/Simple count/XPath	Yes, timeout value expired and decision type not complete	Incomplete	Last message received at the Fan In mediation primitive input terminal
Iterate/Simple count/XPath	Yes, timeout value expired and decision type complete	Complete	Last message received at the Fan In mediation primitive input terminal

* Depends on if the condition (Simple count or XPath) evaluated to true for the last message that was received.