About the Large Messaging sample

The Large Messaging sample is a message flow sample application based on the scenario of end-of-day processing of sales data. Messages recording the details of sales though the day are batched together in the store for transmission to the IT center. On receipt at the IT center the batched messages are split back out into their constituent parts for subsequent processing. This splitting is achieved using an IBM Integration Bus message flow. Each of the individual messages representing a sale has the same structure.

The input and output messages in this sample are implemented as self-defining XML messages for simplicity. Other message formats could easily be used.

Each input message consists of three parts:

A header containing a count of the number of repetitions of the repeating SaleList structure that follows:
The body that contains the repetitions of the repeating SaleList structure.
The trailer that contains the time the message was processed.

The SaleList structure is a complex one. It contains:

Multiple repetitions of the Invoice structure. The Invoice structure contains details of the individual items that were purchased.
The total cost of the items that were purchased.
The currency in which they were bought.

The aim of the processing in this sample is to write each of the instances of the SaleList structure as a separate WebSphere MQ message while minimizing overall memory requirements. Virtual memory use is minimized by deleting those portions of the message tree that have already been processed. Without this the message tree would continue to grow larger and larger as each instance of the SaleList structure was processed.

If the message tree continued to grow without any housekeeping it could lead to very large virtual memory requirements for the execution group that the message flow is running in. This can become a significant performance issue for input messages that are many megabytes in size, and can lead to a requirement for hundreds of megabytes and even gigabytes of virtual memory dependent on the size of the input message.

The memory saving technique used in the sample requires the use of a mutable message tree. That is one where parts of the message tree can be deleted. The message tree for InputRoot is immutable and as such it is not possible to delete items from the tree as needed. Accordingly another tree is created within the message flow. It is established by using an ESQL ROW variable and an ESQL CREATE function call.

The processing in the sample consists of one message flow. The processing it performs is described below.

Large messaging message flow

The large messaging message flow performs the following processing:

Reads a WebSphere MQ message containing an XML payload under transactional control.
Formats a WebSphere MQ message for each instance of the SaleList structure.
Writes the WebSphere MQ messages to the output queue.
Produces a WebSphere MQ message to signal completion of the processing when the final element has been processed

The Large Messaging message flow consists of the following nodes:

The MQInput node MessageWithRepeatingElements reads the WebSphere MQ message containing an XML payload. As the incoming message is in a self-defining XML format there is no need to specify a message set or format for it to be parsed successfully.

The Compute node ProduceMessageSlicesFromRepeatingElements contains the majority of the processing for the message flow. It performs the following processing:

Creates a mutable tree that will become a copy of InputRoot.XMLNSC (the message tree for InputRoot is immutable). This is done in order to reduce the overall memory requirements of the message flow
Creates an XMLNSC parser for the new message tree.
Copies InputRoot.XMLNSC to the new message tree.
Obtains the value in the SaleListCount field that is in the header of the input message. This value specifies how many repeating SaleList items are within the message. It is checked against the number processed.
Formats a WebSphere MQ message for each instance of the SaleList structure.
Sends the formatted message along the remainder of the message flow so that it can be written as a WebSphere MQ message. This is achieved through the use of the PROPAGATE verb.
Formats and writes a WebSphere MQ message to mark confirmation of the processing of the SaleEnvelope message.

The filter node IdentifyWhenSlicingIsComplete applies a filter expression to the message that was propagated in the node ProduceMessageSlicesFromRepeatingElements. The aim of the test is to identify whether the message entering the node is one that should be written to the queue of processed SaleList messages or the batch confirmation message. If it is a message containing a SaleList item, processing passes to the MQOutput node RepeatedElementSlices that writes the message to a WebSphere MQ queue. If the message received in the filter IdentifyWhenSlicingIsComplete is the batch confirmation this is written as a WebSphere MQ message to a different queue from the SaleList items using the MQOutput node MessageSlicingComplete.

There are additional nodes in the message flow to provide processing for different error conditions. The MQOutput node General Failure is attached to the failure terminal of the MQInput node. The node writes the message to the queue named in the MQOutput node if an error occurs in the processing within the MQInput node. The catch node of the MQInput node is connected to a filter node called CatchProcessing. A message is propagated along this route if an exception is thrown further along the message flow and caught by this node. The filter expression within CatchProcessing determines whether the message being processed has the expected format. If it does not, processing passes to the node Throw Error where a user error is thrown. If the message is malformed in some way, for example the last field is not called SlicingReport, processing passes to the MQOutput node Malformed Messages where it is is written to an error queue.

Test message

The test message used to drive the message routing sample is a self-defining XML message that contains invoice details for a customer. You can extend the message in size by repeating the SaleList item. The message for the sample contains ten repetitions for the SaleList item. The message layout is shown below. In this case only a single instance of SaleList is shown.

<SaleEnvelope>
 <Header>
  <SaleListCount>1</SaleListCount>
 </Header>
 <SaleList>
  <Invoice>
   <Initial>K</Initial>
   <Initial>A</Initial>
   <Surname>Braithwaite</Surname>
   <Item>
    <Code>00</Code>
    <Code>01</Code>
    <Code>02</Code>
    <Description>Twister</Description>
    <Category>Games</Category>
    <Price>00.30</Price>
    <Quantity>01</Quantity>
   </Item>
   <Item>
    <Code>02</Code>
    <Code>03</Code>
    <Code>01</Code>
    <Description>The Times Newspaper</Description>
    <Category>Books and Media</Category>
    <Price>00.20</Price>
    <Quantity>01</Quantity>
   </Item>
   <Balance>00.50</Balance>
   <Currency>Sterling</Currency>
  </Invoice>
  <Invoice>
   <Initial>T</Initial>
   <Initial>J</Initial>
   <Surname>Dunnwin</Surname>
   <Item>
    <Code>04</Code>
    <Code>05</Code>
    <Code>01</Code>
    <Description>The Origin of Species</Description>
    <Category>Books and Media</Category>
    <Price>22.34</Price>
    <Quantity>02</Quantity>
   </Item>
   <Item>
    <Code>06</Code>
    <Code>07</Code>
    <Code>01</Code>
    <Description>Microscope</Description>
    <Category>Miscellaneous</Category>
    <Price>36.20</Price>
    <Quantity>01</Quantity>
   </Item>
   <Balance>81.84</Balance>
   <Currency>Euros</Currency>
  </Invoice>
 </SaleList>
 <Trailer>
  <CompletionTime>12.00.00</CompletionTime>
 </Trailer>
</SaleEnvelope>

Back to sample home