PH59305: WebSphere Application Server v9.0.5.x JMS application fails with CWSIT0088E Exception due to delay in messaging engine

APAR status

• Closed as program error.

Error description

• DB2 database failover was performed from the primary database
  node to the standby node. After the database failover,
  the JMS application running in WebSphere Application Server
  v9.0.5.13 fails to connect to the messaging engine and the
  following errors are logged in the SystemOut.log:
  
  CWSIT0009W: A client connect to bus XXXBus failed in the
  bootstrap server with endpoint p15w:7264:BootstrapBasicMessaging
  with reason:
  CWSIT0088E: There are currently no messaging engines in bus
  XXXBus running. Additional failure information: CWSIT0103E: No
  messaging engine was found that
  matched the following parameters: bus=XXXBus, targetGroup=null,
  targetType=BusMember, targetSignificance=Preferred,
  transportChain=InboundBasicMessaging, proximity=Bus..
  
  javax.jms.JMSException: CWSIA0241E: An exception was received
  during the call to the method
  JmsManagedConnectionFactoryImpl.createConnection:
  com.ibm.websphere.sib.exception.SIResourceException: CWSIT0008E:
  A successful connection was made to the bootstrap server at
  xxxhost:7264:BootstrapBasicMessaging but the server returned an
  error condition: CWSIT0088E: There are currently no messaging
  engines in bus XXXBus running. Additional failure information:
  CWSIT0103E: No messaging engine was found that matched the
  following parameters: bus=XXXBus, targetGroup=null,
  targetType=BusMember, targetSignificance=Preferred,
  transportChain=InboundBasicMessaging, proximity=Bus..
   at
  com.ibm.ws.sib.api.jms.impl.JmsManagedConnectionFactoryImpl.crea
  teConnection(JmsManagedConnectionFactoryImpl.java:204)
   at
  com.ibm.ws.sib.api.jms.impl.JmsManagedQueueConnectionFactoryImpl
  .createQueueConnection(JmsManagedQueueConnectionFactoryImpl.java
  :84)
   at
  com.ibm.ws.sib.api.jms.impl.JmsManagedQueueConnectionFactoryImpl
  .createQueueConnection(JmsManagedQueueConnectionFactoryImpl.java
  :73)
   ....
  
  Caused by: com.ibm.websphere.sib.exception.SIResourceException:
  CWSIT0009W: A client connect to bus XXXBus failed in the
  bootstrap server with endpoint
  xxxhost:7264:BootstrapBasicMessaging with reason: CWSIT0088E:
  There are currently no messaging engines in bus XXXBus running.
  Additional failure information: CWSIT0103E: No messaging engine
  was found that matched the following parameters: bus=XXXBus,
  targetGroup=null, targetType=BusMember,
  targetSignificance=Preferred,
  transportChain=InboundBasicMessaging, proximity=Bus..
   at
  com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.generat
  eException(TrmSICoreConnectionFactoryImpl.java:951)
   at
  com.ibm.ws.sib.trm.client.TrmSICoreConnectionFactoryImpl.contact
  BootstrapService(TrmSICoreConnectionFactoryImpl.java:745)
  
  The above exceptions were caused due to delay in the startup
  of the messaging engine.
  

Local fix

• Drain all the SIB queues so that no message data is retained
  in the database. Then, stop the messaging engine and drop the
  data store tables.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:  Users of IBM WebSphere Application Server   *
  *                  using an external database as an SI Bus     *
  *                  data store                                  *
  ****************************************************************
  * PROBLEM DESCRIPTION: Messaging engine startup takes an       *
  *                      increasingly long time, and the size    *
  *                      of the SIB tables in the data store     *
  *                      increase.                               *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  Trace collected during data store initialization shows trace
  similar to:
  11/8/23 10:27:11:841
  IST	000000b2	?	ItemTable
  (com.ibm.ws.sib.msgstore.persistence.im
  pl.ItemTable) [:/bb524d3b] Item filtered out:
  4318005276	ItemTable
  indicating rows in the data store table that refer to
  logically deleted messages that haven't been removed from the
  database, but for which the XID cannot be reconciled with the
  transaction manager. These rows are ignored and do not impact
  ME function, but remain in the database.
  A preferred solution is to empty the message store so there is
  no required data left in the database, drop the database tables
  and let them be recreated on ME restart.
  

Problem conclusion

• The code was changed to save the details of the database
  rows to be output to a file so that they can be removed
  manually.
  
  
  The fix for this APAR is targeted for inclusion in fix pack
  9.0.5.22.
  For more information, see 'Recommended Updates for WebSphere
  Application Server':
  https://www.ibm.com/support/pages/node/715553
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBS APP SERV N

• Fixed component ID

  5724H8800

Applicable component levels