Event Sequencing Qualifier
Event Sequencing is a Quality of Service (QoS) qualifier available within WebSphere Process Server (WPS) and WebSphere Enterprise Bus (WESB). This qualifier is often misunderstood and improperly used leading to frustration and inherent application design issues. This article will try and remove the confusion as to what functionality Event Sequencing was designed to provide and how to troubleshoot this qualifier.
Take, for example, the following application (Figure 1), which receives a batch of order records that need to populate a database. An order record is received that will cause an insert into the database table. Subsequent records will update these records. It is important that:
- The insert record is processed first
- The updates are processed in order so that the order record has the correct information at the end of processing
- If update 2 changes the quantity from 20 to 11
- If update 1 changes the quantity from 5 to 20
- If update 2 is processed before update 1, the order quantity will no longer be correct
Event Sequencing can help in this situation as the sequence of the received orders is maintained. It will not allow the 'grouped' messages to process concurrently, leaving the updates to complete in an unknown order.
Figure 1. Batched orders application
As can be seen, Event Sequencing (ES) is a Quality of Service (QoS) qualifier that will maintain the order of service requests, where requests with the same 'key' are executed serially based on the order that they are received. The 'key' of the requests can be defined by users when they specify the QoS and that ‘key’ usually consists of attributes of the payload in the service request.
Here are some simple rules regarding Event Sequencing:
- Must be used on Asynchronous calls only. (see Figure 10)
- Does maintain order of messages received
- Does not reorder messages based on a value(s) in a message
- Can only be applied to Interface operations
- Should be used on interfaces with a request-response method signature
- If this is not applicable, Event Sequencing should be used on downstream components with the same key for all messages
The most popular misconception with regards to Event Sequencing is that the qualifier will store and reorder messages based up on the 'key' value in the message. However, Event Sequencing will only maintain the order of the received messages based upon the 'key' value(s). An example of this mechanism will be seen in the section Event Sequencing Keys.
Using Event Sequencing
The application of the Event Sequencing (ES) QoS qualifier to an operation is a simple task. As can be seen in Figure 1, the qualifier is added via the Assembly Diagram screen. Select the module that requires the ES Qualifier applied and in the Details screen select the requisite Interface and Operation.
Figure 2. Selecting the Interface to add the Qualifier
Subsequently, a series of tabs will appear from where you can select the Qualifiers tab, from where the Add button will allow the inclusion of the Event Sequencing qualifier.
Figure 3. Applying the Event sequencing qualifier
Event Sequencing Keys
In order for ES to actually lock incoming events, a key needs to be created. This key will be used to group messages into sequences, allowing for effective locking of subsequent messages while an existing message with the same key value is processed.
To create a key, click on the Event Sequencing qualifier and a sub-Properties window will be shown.
Figure 4. Creating a key for an ES qualifier
A default key entry will be created, and buttons will be available to Add, Delete and Edit an entry. When using Add or Edit, a dialog will appear allowing selection of a parameter name, and a valid XPath expression. This expression can be manually entered, or using the Edit button, built using a simple tree view.
Figure 5. Selecting Business Object aspects for ES key
Multiple keys (composite keys) can be used to group messages together, and these groups can be named, via the GroupName, and reused. The group name to use is selected from from the sub-Properties window as seen in Figure 3.
In order to show Event Sequencing in action, a simple application is created to retrieve messages from a JMS queue, and load them into a database. There will be three messages per ‘recordNumber’
- Update 1
- Update 2
The Business Object (BO) is shown below:
Figure 6. Definition of the input record Business Object
The lock key is specified as the ‘recordNumber’, which will maintain the grouping for all messages with the same ‘recordNumber’, but allowing messages with different recordNumbers to be processed concurrently.
Figure 7. Flow of an Event Sequenced message
The flow of message is as follows:
- A message is delivered to a component queue via an asynchronous invocation.
- A lock entry is written to the PERSISTENTLOCK database table by the persistentLkMgr application and messages with the same recordNumber are grouped.
- If no message is currently locked (i.e. being processed), the new message is delivered to the target component.
- In an async call to a BPEL process, the endWork callback is invoked to allow the message to be unlocked.
- For non-endWork enabled components, the unlock occurs when a reply is completed, or after the completion of the async call.
- For synchronous calls, the unlock will occur after the completion of the called component.
- The message is then deleted from the lock table.
- The relevant record in the lock table is removed by the persistentLkMgr application and the next message in the groups sequence is processed.
Cross Component Event Sequencing
In some cases, having the ES qualifier on one component will not be enough to guarantee the order of the processed messages. Consider the following application, as shown in figure 7.
Figure 8. Asynchronous Component calling Asynchronous Component
The application has an asynchronous two-way component, AsynchronousCompA, and the operation has the ES qualifier applied. This component also has an asynchronous one-way invocation partner for AsynchronousBPELB.
This construct does not guarantee that the ordered messages are processed in order downstream as AsynchronousCompA will commit its message to the input queue of AsynchronousBPELB and not at the end of that components execution as the message has no reply (see Figure 11).
The one-way request to the AsynchronousBPELB component will return immediately, as it does not have to wait for a reply, and release the lock on the message in AsynchronousCompA, before processing the next grouped message.
The initial message and the subsequent messages may be executing the downstream component at the same time or later. As this behavior is not guaranteed, the sequencing of events downstream cannot be guaranteed.
In order to circumvent this problem, the ES QoS qualifier should be placed on the interface of AsynchronousBPELB with the same key – or same aspect of the BO that represents the same key – as AsynchronousCompA.
A deadlock can occur when a message cannot be unlocked until it receives a response back from an asynchronous request/reply operation, or the 'endWork' callback is executed from an existing sequenced process.
For example, a deadlock can occur when a looping construct in the assembly diagram, whereby 'Component A' calls 'Component B', which in turn calls 'Component A'.
In Figure 8, the 'processOrders' BPEL calls 'eachOrderLine', and this module calls 'processOrder' again. If the call from 'eachOrderLine' uses the same key, the second request message is placed into the lock table, waiting for the currently locked message to finish. This initial message cannot be completed, as it waits for a reply from the second request.
The number of messages that can be locked at any one time is controlled by the maxActiveMessages parameter in the eventsequencing.properties file located in the <WAS_Install>/properties directory.
The initial value is 100, meaning 100 messages can be locked and held by the system. If this number is exceeded, the component queue used to store messages will be held until such time as the number of messages is below the set value.
Setting this value too low can cause deadlocks and stop an application from functioning, while setting it too high could cause a flood of messages to be delivered and processed and thereby overwhelming the system.
Care should be taken when adjusting this parameter.
The Event Sequencing (wm/ESWorkManager) work manager is used to control the number of threads used to process messages at any given time.
Figure 9. Event Sequencing Work Manager
The minimum number of threads is the initial number created and used by the event sequencing components, whilst the maximum number is the greatest number of threads that can be used.
The work request queue size is the number of objects that can be queued before being processed by an available thread. An empty value, which is the default, indicates that the size is automatically managed. A large size will pre-allocate memory to hold the objects and may take up valuable system resources.
Similarly, setting the maxProperties to a large size could cause a high number of messages to be held on the component destination which could place a strain on the messaging engine hosting this destination.
Event Sequencing Components
The component in WPS that performs the locking and releases messages to be processed is the Persistent Lock Manager (persistentLkMgr) application. This application is used to manage:
- the received messages
- the persistent storage (PERSISTENTLOCK Db table)
- the locking and unlocking of messages
In order for the Event Sequencing to control the flow of messages it must maintain a list of the messages currently locked relative to the destination and keys. The mechanism provided is the creation of the PERSISTENTLOCK (LOCK) table located in the WPS Common Database.
When the ES qualifier is set, and messages arrive onto the appropriate destination, a record is written to this table. The LOCKID is the unique value, given to a message when it is stored into the LOCK table, and the SEQUENCEID is the lock position for the message relative to the key. These values are then persisted in the message metadata.
When a reply is received back to the destination, the LOCKID and SEQUENCEID are retrieved from the message metadata and the appropriate entry is deleted from the LOCK table, and the next SEQUENCEID is released for processing.
Figure 10. Message being locked and stored in the ES database
As can be seen in Figure 10, a message has been received but could not get the 'lock, which would allow it to be processed, as a message with the same key is being actively processed. The message information – messageID, sequenceID, etc - stored into the PERSISTENTLOCK database table. The message itself still resides on the component destination.
Locking and UnLocking
Received messages are locked until such time as they can be processed, i.e. when no other message has the 'lock'. These locks are released when the message is 'completed' which can occur at different times depending on the invocation style and the interface type.
- An asynchronous component returns a response on request-response operations
- An asynchronous component that supports the Work Completion Contract interface will have
its 'endWork' callback executed
- Business State Machines (BSM)
- For a synchronous component, the lock is released with it returns from the operation invocation.
Figure 11. Log showing the endWork callback of an async module
Invocation Style Impact on Event Sequencing
Figure 12 shows the impact on the ES qualifier based on the type of operation and the invocation style. As can be seen, synchronous invocations of modules, in most cases, will cause the ES qualifier to be ignored.
Figure 12. Downstream ES qualifiers
The esAdmin tool enables an administrator to maintain the current locks that are active within the system. The tool allows the user to list locks by module, by component or can list all locks in the system.
The tool will also allow users to unlock a message, or delete messages from the lock queue.
Great care should be taken to when deleting or unlocking messages
esAdmin [-h hostname] [-p Port Number] [-username username] [-password password] method paramaeters.
Table 1. Locks Method
|listAll||List all locks regardless of the module or component|
|listLocks <module>||List all locks for a specific module, regardless of component or method|
|listLocks <module> <component>||List all locks for a specific component in a specific module, regardless of the method|
|listLocks <module> <component> <method>||List all locks for a specific method, in a specific component in a specific module.|
|deleteLocks <module>||Deletes all locks for a specific module. The module must be stopped for this command to execute successfully|
|unlock <lockid>||Manually release a held lock, based on a specific lockid|
Adapters and Event Sequencing
MQ, MQ/JMS and JMS Export Settings
The MQ and MQ/JMS exports must be configured to force Event Sequencing to be enabled:
Figure 13. MQ/JMS and JMS export binding
Figure 14. MQ Export binding
MQ Listener Ports
Applications that utilize an MQ Export will, upon deployment, create Listeners which will be used to monitor the queues and trigger the module upon receipt of an appropriate message.
In a cluster, one Listener will be created per Application target. Each pulling messages from the queue in order to allow for applications on each Application Target to consume messages.
With two Listeners, the order in which the messages are being processed is no longer guaranteed as each Application Target may process messages at a different rate.
Figure 15. Listener Port configuration example
In order to resolve this issue, one Listener Port is required to be stopped in order to guarantee, from a WebSphere Process Server perspective, the order of the message. The maximum sessions and maximum messages settings can be optimized without impacting Event Sequencing.
Other MQ factors
There is, however, another aspect in receiving messages from MQ, and this with the MQ server itself. MQ places messages onto a queue in an 'uncommitted' state. These messages will not be seen by the Listener until they are 'committed', but can be seen during a browse of the queue and in the queue depth.
These messages could be committed in any order, and would therefore appear to the Listener port. These messages may appear to have been retrieved 'out of sequence' when browsing the queue, but are in fact in the order in which they were committed.
Another factor in the retrieval of messages is the mode of the Listener itself. Messages are only guaranteed to be in order if the Listener is set-up to retrieve messages in non-ASF (Application Server Facilities) mode.
To set the ASF mode of the Listener, a custom property must be set named 'NON.ASF.RECEIVE.TIMEOUT'. Setting this variable will disable the ASF mode for that Listener. The value of this parameter should be set to less than the transaction timeout (set to a default of 120 seconds) and the difference between the two should be greater than the amount of time to process a message.
EIS/JDBC adapters and Event Sequencing
EIS adapters require design time and runtime configuration to enforce Event Sequencing. In WID, one must change recommended interaction style to 'async'.
Figure 16. EIS adapter preferred interaction style
Once deployed, the adapter requires the 'deliveryType' custom property to be set to 'ORDERED'.
Figure 17. Flat File adapter with deliveryType custom property
Logs and Traces
Logging of Event Sequencing is accomplished by adding the following to the Runtime Diagnostic Trace Service:
This will output to the trace.log the work done by the Event Sequencing handlers and PersistentLkMgr application and will show work being executed by these components and any failures that result.
In the test, two messages are loaded onto a queue with the same record number, which will be used as the key. A snippet of the logs is shown and described below:
Figure 18. Event Sequence lock aquisition for a message
The message shows the initial lock of a message by the PersistentLockMgr application for a message with the id 058636516E9BEA4_4500079 using the module 'JMS_ES_ModuleApp', A call to esAdmin would show this message for this application and an entry will be placed into the PERSISTENTLOCK database table.
Figure 19. A message is retrieved for processing and acquires the lock message
In Figure 19 the handleMessage function is called as a message is received and the lock is obtained in order to process the message. The last entry in the snippet shows that the SIBus is now engaged to process that message.
Figure 20. More messages are received
As more messages are being received, as showing in Figure 11, the Event Sequencing module attempts to obtain the lock for this message, so that it can be processed. However, the lock cannot be obtained as the previous message has not completed yet.
This message is subsequently saved for processing, in the order is received, after the current message is processed.
Figure 21. Message is unlocked and the next sequential message is found
Figure 21 shows 'sequenceId' is unlocked and deleted from the lock queue, and that the event sequencing module found the next message in sequence, i.e. sequenceId.
This sequence of events will repeat for each message in the sequence, and for each key defined within the qualifier and for each module defined with an Event Sequencing qualifier.
End of Work Callback
From the logs, it is possible to detect the work completion of a message being processed.
- For asynchronous components that support WCC (Work Completion Contract) interface there will be a series of messages indicating the calling of the interface, and the successful completion.
Figure 22. endWork interface called
- For asynchronous components that do not implement the WCC interface, the work is assumed when the component returns.
- For synchronous components, work completion is signaled at the return from the operation invocation.
I would like to thank Chendong Zou, Event Sequencing Architect at IBM for his time and input.
- Download IBM product evaluation versions and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.