Split and join overview

The split and join feature in the ebMS 3.0 (advanced) specification is useful for transferring large messages. The split and join feature is applicable to one-way push and one-way pull message exchange patterns only.

With the split and join feature the payload and attachments that are submitted by the business application can be split into fragments. The fragments can then be transferred to the trading partner. Similarly, the fragments that are received from the trading partner can be joined to form a complete message.

Transferring large messages on a network might consume the network resources or, if the transfer fails, you might have to restart the message transmission. The split and join feature addresses the disadvantages of transferring large messages on a network by breaking a large message into fragments.

You must configure split and join in the File Size Optimization section of a conformance policy. For more information about configuring split and join, see Creating a custom AS4 conformance policy topic.

The following sections provide details of split and join flow in AS4 Microservice.

Splitting an outbound message

The following list provides an overview of the tasks that are involved for splitting an outbound message when split and join is configured in AS4 Microservice:
  • The business application submits a business document object (BDO) and message (payload and attachments) to the configured messaging receiver.
  • The AS4 service in AS4 Microservice picks the BDO and message, and looks up for the exchange profile. The AS4 service learns about the split and join configuration after the exchange profile lookup.
  • The total size of the message (payload and attachments) is calculated. If the message size is lesser than the fragment size that is configured in the conformance policy, the message is not split. If the message size is greater than the fragment size, then the split task is initiated for the message.
  • MIME and ebMS packaging is initiated for the message. Concurrently, if Apply security to the source message is selected, security configuration of the conformance policy is applied to the source message.The MIME packed message with the SOAP parts and attachments is called the source message. If SPLITJOIN_WRITE_SOURCE_MESSAGE_TO_STORAGE system property is enabled in the system properties file, the complete source message is written to storage.
  • While MIME packaging is in progress, chunks of messages for which MIME packaging is completed is passed through the splitter. The message is split to fragments based on the fragment size configuration in the conformance policy. The fragments are written to storage as blobs.
  • A fragment BDO is created for each fragment (with empty payload and each fragment as an attachment). The fragment BDO consists of the following information about the fragments:
    • Group ID (generated per source message). Group ID is a unique ID for the fragment group, and not the MessageId of the source message.
    • Fragment number (sequence number of the fragment).
    • Total number of fragments in the group.
  • The store and forward service is called using the fragment BDO to start the AS4 outbound service.
  • The AS4 outbound service, starts ebMS packaging and security handlers (based on the conformance policy configuration).
  • During ebMS packaging, fragment header that contains information about the fragment and group ID is inserted. The information is retrieved from the fragment BDO.
  • During the security phase, each fragment is signed based on the conformance policy configuration.
  • Finally the fragments are sent to the trading partner destination that is configured in the exchange profile.

Splitting messages submitted for pull

When the business application submits a large message to be pulled by a trading partner, and split and join is enabled in the conformance policy, the flow is as follows:

  • The message that is submitted for pull is split (based on the conformance policy and exchange profile configuration) as described in the preceding section.
  • When a pull request is received by AS4 Microservice, and if the pull criteria of the request matches the configuration of the pull message that was submitted, the first message (part) stored for the pull destination is MIME packed along with a MessageFragment header resulting in a message fragment. The message fragment is returned as a response to the pull request.
  • The trading partner system decides the number of pull requests that must be sent based on the number of fragments that are specified in the fragment header.
  • Subsequent pull requests would be selective pulling based on the information in the fragment headers. This is to ensure that fragments of the particular group are pulled.
  • Pull requests for the particular selection (group) are sent until the empty message partition channel message is received.
  • Pull is considered as completed after the trading partner pulls all the fragments and joins them. The pull message can be processed only after joining the fragments.
  • If notification is enabled, a notification is sent to the business application after all fragments of a particular group ID are pulled from the pull destination.

Purging fragments after split

To avoid retaining large and unnecessary data, temporary messages that are generated during split are purged. Fragment data is deleted after a fragment is sent. The blog that was sent as an attachment is also deleted. A purge schedule is run on the split and join related database tables to clear stale fragments that are created when AS4 Microservice goes down when splitting is in progress.

Receipts and retries

Receipts are processed for the complete message or the source message. When all the fragments are received and joined, a receipt is sent to the trading partner. For non-repudiation information of the complete message in the receipt, the digest of the source MIME message is calculated and stored in the non-repudiation bucket in storage.

If a receipt is not received for the source message, the split and join process is repeated. The previously formed fragments are not used. If an invalid receipt is received, the transaction is failed. Store and forward retry for individual fragments and the final receipt only.
Note: Store and forward retry only when the trading partner system is not available or during connection down situations.

When split and join is enabled, you cannot enable synchronous receipts.

Joining the inbound fragments

The following list provides an overview of the tasks that are involved when the inbound fragments are joined:
  • The AS4 receiver receives a message (might be a fragment, but the receiver does not yet know).
  • During the ebMS processing phase, the receiver identifies that the message is a fragment and the flow is marked as a fragment flow.
  • All the received fragments are stored in storage and not sent to the messaging destination (business application).
  • The exchange profile lookup is completed and the security settings are determined.
  • The fragments go through security verification just like a normal message.
  • After a successful security verification, fragment-related information is extracted from the fragment header:
    • Group ID - The group ID is used to identify all the fragments in a particular group
    • Fragment count - The total number of fragments in the group. The fragment count might be available in the last fragment of the group
    • Fragment number - The sequence number of the fragment
    If security verification fails, appropriate error messages and notifications are sent to the trading partner and business application.
  • If all the fragments are received within the join timeout (specified in the conformance policy) and successfully inserted in the appropriate database tables without any duplicate exceptions, security errors, or other errors, the process of joining the fragments is started based on the group ID of the fragments. If split and join is not configured in the conformance policy, but a fragment is received from a trading partner, the default join timeout is 60 minutes.
  • A scheduled service, called the Joiner service is started to join the fragments. The Joiner service joins the fragments based on the count in the group_state and the fragment count tables. The joined message is called a source message.
  • The joined message is unpacked, security processed, and decompressed (if AS4 compression is enabled) and sent to the messaging destination that is specified in the exchange profile. The message is later picked and processed by the business application.
  • If receipts are enabled in the conformance policy, then a receipt is sent to the trading partner for the joined source message.
  • If notifications are enabled in the conformance policy, then a notification for the receipt is sent to the business application.
Note: If any error occurs during the processing of the message fragments, the incoming message fragments are rejected and appropriate notifications are sent to the business application. If a fragment are received after the joining interval has passed, the fragments are rejected and the already received and stored fragments are purged. Visibility events are emitted during the whole process and appropriate notifications are sent to the business application. For a list of errors related to split and join feature, see AS4 error handling overview.