IBM Business Automation Workflow Event Manager - Common symptoms and how to resolve them

Part II - Resolving common event manager symptoms

A - Event manager is not processing any work

If the event manager does not process any work as mentioned previously in the symptoms section, this situation might be caused by the event manager not being active or being active, but blocked for some reason.

The following sections, A.1 and A.2, describe how to analyze and resolve these situations.

To monitor and diagnose the status of the event manager, the following pieces of information are helpful:

• DB table LSW_EM_INSTANCE
• DB table LSW_EM_TASK
• DB table LSW_BLACKOUT_CALENDAR
• Process Admin Console section Event Manager → Monitor as GUI to 1. and 2.
• Process Admin Console section Event Manager → Blackout Periods as GUI to 3.
• BPM server log file SystemOut.log and FFDCs
• Event manager configuration file '80EventManager.xml', and the global BPM server configuration file TeamWorksConfiguration.running.xml, which contains all of the parameters at run time. The following IBM Knowledge Base page shows where to find these files: Event Manager configuration settings

A.1 - Event manager not active

The easiest way to check the status of the event manager is by using the Process Admin Console, which shows the event manager status and lists all event manager jobs. It also lists the expiration time stamp, which is regularly updated every 15s by an internal heartbeat thread and set to the current time + 60s. Keep in mind that this time stamp is created based on the current time used on the database system that is hosting the BPM tables.

The Process Admin Console panel named 'Event Manager → Monitor' is a graphical user interface for the contents of the database tables:

• BPM DB table LSW_EM_INSTANCE containing the event manager status and heartbeat timestamp
• BPM DB table LSW_EM_TASK containing all event manager jobs

The following screen capture shows the event manager status in the Process Admin Console:

Status:

• Green - running
• Red - stopped, not running

Each cluster member has an event manager. If you have a multi-cluster topology, it is only present in the AppTarget servers. Therefore, in this example, it is a single node cluster and only one event manager instance is listed. In a clustered environment, there is an entry for every cluster member as a new row in the table. To process work, the event manager needs to be in the running state (green) and the time stamp listed in the Connect expiration field needs to be ahead of the current time.

A successful event manager startup is reflected in the server's SystemOut.log file with messages that show the start of the heartbeat thread and acquisition of the synchronous queues. You can grep the log for messages of the wle_scheduler group as shown here:

wle_scheduler I CWLLG0561I: Heartbeat thread starting…
wle_scheduler I CWLLG0615I: Heartbeat resumed.
wle_scheduler I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_1.
wle_scheduler I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_1.
wle_scheduler I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_2.
wle_scheduler I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_2.
wle_scheduler I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_3.
wle_scheduler I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_3.

In the LSW_EM_INSTANCE table, the STATUS column shows a value of '1' for an active event manager and '2' if it is paused.

With the Process Admin Console, the event manager can be paused and resumed by clicking the applicable button. Make sure to click Refresh to update the panel so that the status is read from the LSW_EM_INSTANCE table.

Reasons why the event manager might be inactive:

a) The event manager configuration is corrupted by an incorrect attribute in the custom property file (for example: 100Custom.xml)

If you tried to overwrite event manager parameters by using a 100Custom.xml file and accidentally used the 'replace' attribute for merge instead of 'mergeChildren' as show in the next example, the required parameters from 80Eventmanager.xml are not honored:

Incorrect definition sample:

<properties>
   <event‑manager>
      <scheduler merge="replace">
        <start‑paused merge="replace">true</start‑paused>
      </scheduler>
   </event‑manager>
</properties>

Correct definition sample:

<properties>
   <event‑manager>
      <scheduler merge="mergeChildren">
        <start‑paused merge="replace">true</start‑paused>
      </scheduler>
   </event‑manager>
</properties>

With the incorrect definition sample, the Process Admin Console shows a NullPointerException when you click Event Manager > Monitor as shown in the following screen capture:

To solve that problem, correct the 100Custom.xml file and restart the server. After the restart, make sure that the TeamWorksConfiguration.running.xml file contains the complete <event‑manager> section and the event manager is shown as active in the Process Admin Console.

Note: There is a list of sample configuration files to adapt the IBM Business Process Manager configuration, including some samples for the event manager. You can access these files here.

b) The event manager might have been paused manually by using the Process Admin Console

In this case, you can resume its activity as mentioned previously. Even

When it is paused, the connect expiration time stamp is renewed every 15s (default).

c) The event manager is configured to start as "paused"

The 80EventManager.xml BPM server configuration file contains a parameter called <start‑paused>, which is set to false, by default. If it is configured to true by overwriting the parameter with a 100Custom.xml file, then the heartbeat thread to set the event manager 'connect expiration' is active, but the event manager doesn't do any work.

To check for that situation, look at your TeamWorksConfiguration.running.xml BPM server configuration file. Search in that file for the <start‑paused> string in the <event‑manager> section. If that parameter is set to true, this setting explains why the event manager did not become active after server startup.

In case the event manager is configured to be started as "paused," the SystemOut.log file contains only the following messages during startup. They show that the heartbeat thread started and continuously update the connect expiration time stamp, but the event manager did not acquire the synchronous queues.

wle_scheduler I CWLLG0570I: Heartbeat paused.
wle_scheduler I CWLLG0561I: Heartbeat thread starting…
wle_scheduler I CWLLG0615I: Heartbeat resumed.

To resume the event manager, use the Process Admin Console as shown previously. A successful resume action results in the following messages in the SystemOut.log file:

wle_scheduler I CWLLG0615I: Heartbeat resumed.
wle_scheduler I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_1.
wle_scheduler I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_1.
wle_scheduler I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_2.
wle_scheduler I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_2.
wle_scheduler I CWLLG0597I: Trying to acquire synchronous queue SYNC_QUEUE_3.
wle_scheduler I CWLLG0581I: Acquired synchronous queue SYNC_QUEUE_3.

Keep in mind that the <start‑paused> parameter in the event manager configuration needs to be set back to false. Otherwise, it will still be inactive after the next server restart.
 

d) Event manager is not enabled in the configuration

The 80EventManager.xml BPM server configuration file contains a parameter called enabled, which is set to true, by default.

To check for that situation, look at your TeamWorksConfiguration.running.xml BPM server configuration file. Search in that file for the <enabled> parameter in the <event‑manager> section. If that parameter is set to false, then the event manager will not be active after the server startup. In contrast to being started as "paused", it shows none of the previous messages in the SystemOut.log file and the connect expiration time stamp is not updated!

You cannot resume the event manager through the Process Admin Console in such a case. However, you need to change your configuration to set the enabled parameter back to true and restart your server.
 

e) Blackout period is active

Administrators establish blackout periods to specify times when events cannot be scheduled. For example, you might schedule a blackout period due to a holiday or for regular system maintenance windows. When scheduling and queuing events, event subscriptions, and undercover agents (UCAs), the event manager takes blackout periods into account. The following screen capture shows if and which blackout periods are configured. This data is persisted in the LSW_BLACKOUT_CALENDAR DB table.

If a blackout period is active, the event manager monitor in the Process Admin Console lists a scheduled job named End blackout period where the scheduled time column shows when the blackout period ends. Event manager jobs created during the blackout period show a job status of Blacked out.

The following screen capture shows that scenario:

The SystemOut.log file does not show any applicable messages when the blackout period is entered.
 

f) Exceptions during the event manager startup

If the event manager is not running after startup, but was not configured as paused or disabled, the SystemOut.log file might show a couple of exceptions.

There could be various reasons why the event manager failed during startup or resume. Gather the documents as mentioned in the event manager MustGather technote.

The following section shows a few examples:

• Event manager configuration is broken
  
  This problem is caused by an incomplete fix pack installation where required postinstallation steps to upgrade the profile were not executed.
  
  The SystemOut.log file has exceptions that have the following signatures.
  
  CWLLG0144E: Exception in init(): schedule cannot be started. com.lombardisoftware.core.TeamWorksException: Message: SCHEDULER_CONFIG_BROKEN Arguments: loader‑acquire‑sync‑queue‑query: com.lombardisoftware.core.config.eventmanager.SchedulerConfig checkAndReplace Message: SCHEDULER_CONFIG_REPLACEMENT_PARAMETER_NOT_FOUND Arguments: %executing% loader-acquire-tasks-query UPDATE LSW_EM_TASK SET TASK_STATUS = %acquired%, TASK_OWNER = ? WHERE TASK_ID IN (%task‑ids%)
  
  To fix this problem, review the documented postinstallation (interim fix or fix pack) steps and rerun the missing steps.
• Event manager startup problem due to a problem in the BPM embedded document store (applies to IBM Business Process Manager V8.5 and later))
  
  Important note: If the embedded BPM document store cannot be started due to configuration or authorization problems, the event manager also doesn't start!
  
  The SystemOut.log file doesn't show any of the events manager-related startup messages as shown previously, but you see, for example, the following exception, which is related to the embedded document store:

  CWTDS1100E: An error occurred while validating or creating the default configuration for the IBM BPM document store.
                                   com.ibm.bpm.embeddedecm.exception.UserMissesWritePermissionException: CWTDS0022E: The configuration was changed in a way that the technical user 'deadmin' of the IBM BPM document store fails to change the object 'Domain'.
  Explanation: The technical user defined in the BPM role type 'EmbeddedECMTechnicalUser' is not permitted to perform changes on an object.
  Action: Revert the recent configuration changes. Ensure that the user defined by the BPM role type 'EmbeddedECMTechnicalUser' has access to the object. Verify this using the admin task 'getDocumentStoreStatus'.
      at com.ibm.bpm.embeddedecm.internal.DomainConfiguration$2.run(DomainConfiguration.java:264)
      at com.ibm.bpm.embeddedecm.internal.DomainConfiguration$2.run(DomainConfiguration.java:207)
      at java.security.AccessController.doPrivileged(AccessController.java:362)
  
  To fix this problem, contact IBM Support for instructions.

A.2 - Event manager is active but it is not processing any jobs

When the event manager is active (Process Admin Console shows it as active and connect expiration is not outdated) but is not processing any tasks, this issue could be caused by:

• Event manager configuration file '80EventManager.xml', and the global BPM server configuration file TeamWorksConfiguration.running.xml, which contains all of the parameters at run time. The following technote shows where to find these files and how they relate: Event Manager configuration settings
• Event manager blocked due to orphaned transactions in Microsoft® SQLServer holding locks on its tables:
  In case you use Microsoft® SQLServer as the process server database, the reason for that could be so called 'orphaned transactions' in the DB system. The following technote shows how to resolve such a problem: Orphaned XA transactions might exist when using Microsoft® SQL Server with IBM Business Process Manager (BPM)
• System time or time zone of BPM and remote DB system that is hosting the BPM DB is out of sync: To fix that, make sure that the system time on the BPM and the DB node are in sync. It is a best practice to have both on the same network time protocol server (NTP).

B - Event manager shows jobs with a scheduled date of 2099

If the execution of an event manager job fails, it is retried a couple of times as defined by the re‑execute‑limit configuration parameter (default = 5) in the 80EventManager.xml file. The behavior in such a case is part of the new function introduced by APAR JR47860:

• Pre JR47860 behavior: when the re‑execute‑limit is reached, the corresponding event manager job is discarded! There is no way to reexecute this job.
• Post JR47860 behavior: when the re‑execute‑limit is reached, the event manager job is rescheduled for 2099.

The interim fix for the APAR also provides a new administrative command called BPMReplayOnHoldEMTasks, which was introduced to resubmit this failed job. Check the APAR description For more information, see Managing on-hold Event Manager tasks section in the IBM Knowledge Center.

Important note: It is important to eliminate the root cause before resubmitting an event manager job! Otherwise, you might run into the same problem again. To find the root cause, check your SystemOut.log file for message CWLLG0197W. This message indicates, that the event manager tried to execute a task for 5 times but it failed. Note the thread ID and walk back in the thread history within the SystemOut.log file. The same thread, most probably, returned an exception during the execution of this event manager task.

Example for an event manager task to execute an UCA:
1. Search the SystemOut.log file for CWLLG0197W shows the following line - note thread ID 00011779.
[2/4/14 5:54:18:395 GMT] 00011779 wle_ucaexcept E CWLLG0197W: Task Notify BPD 202738 of notification failed 5 times. The task will not be re-executed.

The previous messages for thread 00011779 shows this error message:
[2/4/14 5:54:18:337 GMT] 00011779 wle_ucaexcept E   CWLLG0181E: An exception occurred during execution of task 4,425,203.  Error: PreparedStatementCallback; SQL [update LSW_BPD_INSTANCE_DATA set DATA = ? where BPD_INSTANCE_ID = ?]; Error for batch element #1: DB2 SQL Error: SQLCODE=-1476, SQLSTATE=40506, SQLERRMC=-968, DRIVER=3.61.65; nested exception is com.ibm.db2.jcc.am.SqlTransactionRollbackException: Error for batch element #1: DB2 SQL Error: SQLCODE=-1476, SQLSTATE=40506, SQLERRMC=-968, DRIVER=3.61.65
com.lombardisoftware.core.TeamWorksException: PreparedStatementCallback; SQL [update LSW_BPD_INSTANCE_DATA set DATA = ? where BPD_INSTANCE_ID = ?]; Error for batch         element #1: DB2 SQL Error: SQLCODE=-1476, SQLSTATE=40506, SQLERRMC=-968, DRIVER=3.61.65; nested exception is com.ibm.db2.jcc.am.SqlTransactionRollbackException: Error for batch element #1: DB2 SQL Error: SQLCODE=-1476, SQLSTATE=40506, SQLERRMC=-968, DRIVER=3.61.65
    at com.lombardisoftware.core.TeamWorksException.asTeamWorksException(TeamWorksException.java:130 …

In this special case, the execution of the event manager task failed due to an SQL exception with sqlcode -968, which means that the database file system is out of space.

2. Fix the problem that caused the exception. In the previous example, resolve the out-of-space condition in the database file system.

3. Resubmit the applicable event manager task by using the BPMReplayOnHoldEMTasks command.

C,D,E,F - Event manager is active, but throughput problems exist

Throughput problems might be caused by a wide range of reasons. In terms of the event manager, the potential throughput is limited by the capacity of its queues.

For a comprehensive summary of all event manager-related configuration parameters including the different queues, check the Event Manager configuration settings in the IBM Knowledge Center.

To analyze and fix this problem, you need to understand the involved configuration parameters and how to monitor and adapt them.

a) Find out the event manager queue capacities

The event manager maintains a number of internal queues. The capacity of each queue is limited by a configuration parameter that is specified in the 80EventManager.xml configuration file and limits the number of jobs that can be in the execution state simultaneously. The following table shows the different queues, the applicable configuration parameter, and the default capacity (as of IBM Business Process Manager 8.5.5):

The default values could have been overwritten by using a 100Custom.xml file. Then, find out which values are currently being used and have a look into TeamWorksConfiguration.running.xml file.

b) Determine the event manager queue usage and adapt the event manager queue sizes

To monitor the number of executing jobs on each event manager queue, use the Process Admin Console event manager monitor and count the number of rows for each 'Job Queue' with job status 'Executing'. Alternatively you could use this SQL statement:

SELECT COUNT(*) as EXECUTION_COUNT,
case QUEUE_ID
when '-100' then 'UCA Async Queue'
when '-101' then 'BPD Async Queue'
when '-102' then 'EM System Queue'
else 'UCA Sync Queue' END as QUEUE
from LSW_EM_TASK where TASK_STATUS = 3 group by QUEUE_ID WITH UR;

If the number of executing event manager tasks for a queue has reached the capacity limit and there are more tasks on that queue waiting to be executed (time to be scheduled has already passed), then there might be a performance problem or the queue capacity is too low for the workload and needs to be increased.

The BPD async queue is of special interest because its capacity is shared between the execution of system lane tasks, timer executions, and BPD notifications. If the complete capacity is already occupied by currently executing, long-running system lane tasks, no other job can be executed on that queue. The screen capture shown previously for Symptom C is an example from a system with bpd‑queue‑capacity set to 5 and the complete capacity is occupied by five executing system tasks. To eliminate a problem related to long running system tasks:

1. Find out why the system lane tasks have such a long execution time and try to fix that. There might be various reasons like back-end response time, excessive JVM garbage collection, CPU and memory constraints, network delays, etc.
2. If the system lane tasks are expected to be long-running, think about splitting them into smaller pieces or increase the capacity of the BPD async queue as shown in the next paragraph.

c) Increase the event manager queue capacities

To increase the event manager queue sizes, specify the applicable parameter as shown in the previous table in a 100Custom.xml file. For example:

<properties>
    <event‑manager merge="mergeChildren">
     <scheduler merge="mergeChildren">
      <bpd‑queue‑capacity merge="replace">40</bpd‑queue‑capacity>
      <async‑queue‑capacity merge="replace">10</bpd‑queue‑capacity>
      <sync‑queue‑capacity merge="replace">10</bpd‑queue‑capacity>
      <system‑queue‑capacity merge="replace">10</bpd‑queue‑capacity>
     </scheduler>
    </event‑manager>
</properties>


Important: While increasing the capacity of event manager queues, keep in mind that using more threads in your JVM, also requires more JDBC connections. Thus, the JDBC data source (jdbc/TeamworksDB) connection pool also needs to be increased.
Generally, increase the number of database connections by two times the value by which you increased the queue capacity. Apart from database connections, also more JVM heap size is needed.

If there is a mismatch between the queue capacities and the number of available connections for the data source, IBM Business Process Manager tries to scale down the queue size. That issue is indicated in the log by the warning messages shown under Symptom F.

d) Understand the event manager queue capacity and the related thread pool size

The event manager configuration also shows a parameter named max‑thread‑pool‑size. By default, the value for the max‑thread‑pool‑size parameter is the sum of the individual queue capacities (70). It is important to understand that its size does not limit the overall number of event manager tasks that can be executed simultaneously. So even if you set max‑thread‑pool‑size to 5 and bpd‑queue‑capacity to 10, you are able to execute 10 system lane tasks simultaneously. It is possible because the thread pool is defined as 'growable', which means it temporarily allows the number of threads to exceed the defined limit, but such a thread would be discarded directly after it finished and not be returned to the pool. Therefore, these threads are a bit more expensive.

Starting in IBM Business Process Manager 8.5.5.0, the event manager no longer uses its own internal thread pool. Instead, it uses a WebSphere Application Server work manager thread pool. This function is configured by these two event manager parameters:

• <use‑was‑work‑manager>true</use‑was‑work‑manager>
• <was‑work‑manager>wm/BPMEventManagerWorkManager</was‑work‑manager>

When you use the WebSphere Application Server work manager thread pool, the maximum pool size is configured in the WebSphere Application Server administrative console as shown in the following screen capture:

In a default configuration, the work manager thread pool for the event manager is defined with a maximum of 70 threads, but also as 'growable'. While sizing the work manager thread pool for the event manager, make sure that its size is at least equal to the sum of queue capacities.

In case you modified the thread pool properties and removed the checkbox for "Growable", then the maximum number of threads implicitly also limits the number of event manager jobs that can be executed simultaneously! See this screen capture.

One of the advantages of using a WebSphere Application Server work manager thread pool for the event manager is that you can use the Tivoli Performance Viewer. It is available from the WebSphere Application Server administrative console to monitor the thread pool activity. See this screen capture:

G - Event manager tasks fail when the Lombardi Event Emitter InputQueue reaches the maximum threshold

Check the current queue depth and the threshold. You can use the service integration bus browser, which is integrated into the WebSphere Application Server administrative console, to easily check the queue depth and the high message threshold.

Make sure that there is a message consumer active to read from that queue, which would typically be the Business Monitor infrastructure. If it is not started, start the message consumer and the queue depth decrease.

If the Business Monitor environment is started and consuming messages, but the queue depth is still at the limit, then perform tuning actions for the Business Monitor server or increase the high message threshold for the involved queues.

In case the Business Monitor environment is no longer existent, but you did not revert the BPM server configuration, there are still messages generated and put to the queue, but not consumed!

To solve it, you need to:

1. Pause the event manager so that no new messages are created.
2. Manually delete the message on the queue destination as shown in the following procedure.
3. Disable the event emission on the IBM Business Process Manager Server according to the following instructions.
4. Restart your IBM Business Process Manager Server

To delete the message from the queue destination by using the WebSphere Application Server administrative console, complete these steps:

1. In the SIBUS section, navigate to the queue point for the Lombardi Event Emitter Input Queue.
2. Select the runtime tab.
3. Click messages, which display DeleteAll option to delete all of the messages on that queue.

The event emission for Business Monitor is explicitly enabled by the following entry. To disable it, set the value for parameter 'enabled' to false as shown here:

<monitor‑event‑emission>
  <enabled>false</enabled>
  <j2c‑authentication‑alias>MonitorBusAuth</j2c‑authentication‑alias>
<monitor‑event‑emission>