Maximo - Only 50% of messages in the continuous queue are processed, and the other 50% are missing and no errors recorded

Body

I have run into a couple of situations as of late that presented strange symptoms when loading data into the continuous queue.  In these scenarios, clients were loading records into the continuous queue using various methods, however only 50% of the messages in the queue were being processed successfully into Maximo, and the other half of the messages were simply missing and no longer in the queue.

After digging around in the client's systems, I found that the JMS email listener MDB had been enabled in the following files:

maximo/applications/maximo/mboejb/ejbmodule/meta-inf/ejb-jar.xml

and

maximo/applications/maximo/mboejb/ejbmodule/meta-inf/ibm-ejb-jar-bnd.xmi

This can be seen in the following two sections of the ejb-jar.xml that are uncommented in order to enable them:

<!-- Email Listener JMS is not deployed by default -->
<message-driven id="MessageDriven_LSNRMessageBean">
<ejb-class>psdi.common.emailstner.LSNRMessageBEAN</ejb-class>
<transaction-type>Container</transaction-type>
<message-destination-type>javax.jms.Queue</message-destination-type>
</message-driven>

<!-- Email Listener JMS is not deployed by default -->
<container-transaction>
<method>
<ejb-name>LSNRMessageBean</ejb-name>
<method-name>*</method-name>
</method>
<trans-attribute>Required</trans-attribute>
</container-transaction>

This can also be seen in the following section of the ibm-ejb-jar.xml that is uncommented in order to enable it, showing the activation specification name it should be using:

<!– Email Listener JMS is not deployed by default -->
<ejbBindings xmi:type="ejbbnd:MessageDrivenBeanBinding"
xmi:id="MessageDrivenBeanBinding_2"
activationSpecJndiName="lsnrjmsact">
<enterpriseBean xmi:type="ejb:MessageDriven"
href="META-INF/ejbjar.xml#MessageDriven_LSNRMessageBean"/>
</ejbBindings>

Reviewing these files showed that all resources were uncommented correctly, and referred to the correct JMS activation specification name, lsnrjmsact.

The resources needed in WebSphere for the JMS email listener MDB are as follows:

Bus name:  lsnrjmsbus

Bus destination name: qin

Activation specification name: lsnrjmsact

Queue connection factory name:  lsnrconfact

Queue name: lsnrqin

In each case, the clients were not using the JMS email listener MDB for their email listeners even though they were enabled. 

Knowing this was enabled gave me a clue that perhaps this MDB had not been deployed correctly, and could be pointing to the wrong activation specification name.

Further checking in the WebSpher console, I found that the email listener JMS resources had not been configured as per the above requirements, is where the problem began.  This is because it will  be impossible to deploy the JMS email listener MDB correctly.

For example, during deployment of the maximo.ear, if the JMS resources (activation specifications and queue etc) have not been created in the WebSphere console specifically for the JMS email listener MDB, you will be presented during deployment with a screen asking you to choose which activation specification to deploy the JMS email listener MDB to. 

You can check this in the WebSphere console after deployment.  I was able to see which activation specification was bound by each MDB by navigating to the following location in the WebSphere console:

All Applications > MAXIMO > Message Driven Bean listener bindings

It was here that I was able to see in the client's environment that the JMS email listener MDB was bound to the Integration Framework's activation specification named intjmsact, as well as the MIF MDB used by Integration Framework.  See below:

Since the intjmsact activation specification is configured in WebSphere to listen to the continuous inbound queue used by the Integration Framework, the MIF messages sent inbound to the continuous queue can be retrieved from the queue by either the JMS email listener MDB, or the MIF MDB.  The result of this is that there will be two MDBs picking up messages from the same queue.  Half of those will be picked up by the JMS email listener MDB, and the other half will be picked up by the MIF MDB. 

The problem with this scenario is that the JMS email listener MDB cannot process messages that are formatted for the MIF MDB.  Therefore, every message the JMS email listener MDB picks up (generally every second message) from the continuous queue is silently thrown away, and no errors are produced or recorded.

If you find this scenario in your enviornment,  you can simply pause the message consumption from the JMS email listener MDB from within the WebSphere console, then test again to ensure that all Integration Framework messages are processed from the continuous queue successfully into Maximo.  If you now see that all messages are processing successfully from the queue into Maximo once the JMS email listener MDB is paused, you can proceed to fix the JMS email listener resources in WebSphere to ensure they exist, and that they are mapping correctly during the next deployment of the maximo.ear.

To pause message consumption, see my previous blog post found here:

/support/pages/node/1113999

I hope you do not run into this problem as it can be quite perplexing to lose messages without any errors, but if you do, I hope the above helps you to resolve it quickly.  If you have any questions or concerns, please contact Maximo Support.