Aggregation
Aggregation is a powerful and important enterprise service bus (ESB) pattern. An inbound request might map into several individual outbound requests. The responses from these requests can be used to enrich a final request to a service or can be aggregated into a single response for the original request.
Common scenarios are:
- A car comparison insurance quote system where a single request is submitted to an ESB that causes multiple insurance service providers to be queried before the best quote is returned.
- A holiday booking request arrives at the ESB and needs to be forwarded to separate hotel, flight and car hire booking systems, before a combined response can be returned.
- A store ordering system, where a salesperson can submit a request for a number of orders and have them dispatched. The original message coming into the ESB identifies the customer for each order via a customer ID. So, the order request message needs to be enriched with each customer's shipping details so that the dispatch step has all the information it needs to succeed.
When creating aggregation solutions, there are several core parts of the programming model that
you must understand before going into the details of the patterns:
- The Fan Out mediation primitive
- The Fan Out context
- The Fan In mediation primitive
- The Service Invoke mediation primitive
- Aggregation block
- Shared context
- Special considerations
Fan Out mediation primitive
The Fan Out mediation primitive is normally used at the start of an aggregation. It provides the ability to either branch (Once mode) the mediation flow, or fire its output terminals multiple times (Iterate mode), based on the size of a repeating structure within the Service Message Object (SMO). For example, if the inbound message contains a structure that needs to be forwarded onto two services, you can branch the flow at the Fan Out mediation primitive. To call a service multiple times, each with individual requests from a repeating structure, you can use the Iterate mode of the Fan Out mediation primitive. You can configure the two modes from the Fan Out mediation primitive properties view, as shown in Figure 1.Figure 1. The Fan Out mediation primitive properties view
The mediation primitive is configured for Once mode by default, but this can be changed by
selecting the for each element in XPath expression button and specifying the
location of the repeating structure within the SMO. Fan Out context
The Fan Out context is used to store the current item of a repeating element when Fan Out is used
in Iterate mode. The Fan Out context contains an integer field containing the index of the
occurrence of the repeating element along with an element containing the occurrence itself. For
example, if there was a repeating element of an element called order, the Fan Out
context would look like this:
<FanOutContext>
<iteration>0</iteration>
<occurrence>
<order>
<id>1234567890</id>
</order>
</occurrence>
</FanOutContext>Fan In mediation primitive
The Fan In mediation primitive is a partner to the Fan Out mediation primitive. The Fan Out mediation primitive is used to decide how many messages to generate, whereas the Fan In mediation primitive is used to decide when the aggregation can be considered complete. The Fan In mediation primitive is always paired with a Fan Out mediation primitive in the same flow (for example, a Fan Out/Fan In mediation primitive pairing cannot expand across a request/response flow boundary), and the Fan In mediation primitive acts as a decision point for when to leave the aggregation block. The Fan In mediation primitive receives a number of messages until a decision point is reached, at which point the last message that was received is propagated to the output terminal. The Fan In mediation primitive can only be used in combination with the Fan Out mediation primitive.Three types of decision points for the Fan In mediation
primitive are supported:
- Simple count: the output terminal is fired when a set number of messages is received at the input terminal. However, more messages might be sent out by the Fan Out mediation primitive, because the Fan In mediation primitive count decision point can be reached more than once, resulting in multiple firings of the output terminal.
- XPath decision: the output terminal is fired if an XPath expression evaluation of the incoming message evaluates to true. It is possible for the output terminal to fire multiple times.
- Iterate: the Fan In mediation primitive waits to receive all messages produced by the corresponding Fan Out mediation primitive. All of the occurrences in the repeating element have been processed.
Figure 2. The Fan In mediation primitive properties view
The Fan In mediation primitive has two input terminals (in and stop), two output terminals (out and incomplete) and a fail terminal (fail). The input terminals are wired to accept a message and the other terminals are wired to propagate a message.
The in terminal accepts input messages until a decision point is reached. When a decision point is reached the out terminal is fired. Although the out terminal is fired when a decision point is reached, the in terminal can still accept input messages, under some circumstances. For example, if the count value is reached, the out terminal is fired. However, if the Fan Out mediation primitive is still sending messages, the Fan In mediation primitive can still receive messages.
You can stop a Fan In operation by wiring the preceding mediation primitive to the Fan In stop terminal. The stop terminal causes the incomplete output terminal to be fired, and this stops the associated Fan Out mediation primitive from sending any more messages. The incomplete output terminal is also fired if a timeout occurs. Table 1 describes terminal behavior further.
Service Invoke mediation primitive
The Service Invoke mediation primitive is used between the Fan Out and Fan In mediation primitives to call the services required. The Service Invoke mediation primitive is discussed in the Invocation of Services chapter.Aggregation block
When you create an aggregation solution, the Fan Out mediation primitive declares the start of
the aggregation and a Fan In mediation primitive symbolizes the end of the aggregation. The logic
between these two mediation primitives can be considered a repeating block if the Fan Out is
configured in Iterate mode, or branching if configured in Once mode, as shown in Figure 3
.
Figure 3. Example of an aggregation block
Shared context
There are three user-defined data store contexts within the SMO:- Data stored in the Transient context is available within either a request or response flow.
- Data stored in the Correlation context is available in both the request and response flows.
- Data stored in the Shared context is available in all branches and executions of an aggregation block.
Figure 4. Shared context example
In the case of this example, the SMOs passing through branch 1 and 2, and therefore the
Transient and Correlation contexts, are different copies. This means that any changes made to the
SMO in one branch are not made to the other branch. The Shared context area of the SMO has been
designed for sharing aggregation data between the different branches of the flow inside an
aggregation block. Access to the Shared context is only available inside the flow which contains the
aggregation block that uses it.Aggregation mediation programming model special considerations
When using aggregation capabilities consider these points:- The Shared context is part of the SMO, and is designed for use inside aggregation blocks because the data put into it is available in all the branches of a flow. The design of the Shared context structure is essential to a successful aggregation design. The goal is to store the response from each service invocation into a structure that can then be converted into a single response message after the aggregation block.
- The Shared context is shared across SMOs and should be considered read-only outside the aggregation block.
- At development time it is difficult to determine which SMO body type will be emitted from the Fan In mediation primitive output terminal. Mediation primitives after the Fan In mediation primitive should be used to generate a new message body using the information in the Shared context.
- If a message arrives at a Fan Out mediation primitive, one or more output terminals of the Fan
In mediation primitive will fire (assuming no ServiceRuntimeExceptions occur within the aggregation
block, for example, as a result of running a Fail mediation primitive), even if a message is not
received by the Fan In mediation primitive. Table 1 highlights this behavior.Note: In some circumstances it is possible that none of the messages sent by the Fan Out mediation primitive arrive at the corresponding Fan In mediation primitive. Using Figure 4 to illustrate, consider this example: if the second output terminal on primitives branch 1 and branch 2 are fired, this does not cause the messages to arrive at the Fan In mediation primitive, because there is no wired path. This is represented in Table 1 by the message result "No".
| Fan In mediation primitive mode | Message arrives at Fan In mediation primitive | Output terminal fired | SMO of Fan In mediation primitive |
|---|---|---|---|
| Iterate | No | Complete | Message body empty; SMO headers and context are the same as the message received at the Fan Out |
| Iterate | Yes (in terminal) | Complete | Last message received at the Fan In mediation primitive input terminal |
| Iterate | Yes (stop terminal) | Incomplete | Last message received at the Fan In mediation primitive input terminal |
| Simple count/XPath | No | Incomplete | Message body empty; SMO headers and context are the same as the message received at the Fan Out |
| Simple count/XPath | Yes (in terminal) | Incomplete/Complete * | Last message received at the Fan In mediation primitive input terminal |
| Simple count/XPath | Yes (stop terminal) | Incomplete | Last message received at the Fan In mediation primitive input terminal |
| Iterate/Simple count/XPath | Yes, timeout value expired and decision type not complete | Incomplete | Last message received at the Fan In mediation primitive input terminal |
| Iterate/Simple count/XPath | Yes, timeout value expired and decision type complete | Complete | Last message received at the Fan In mediation primitive input terminal |