PH28719: JMS message may remain stuck in Committing/Removing state when using the Default JMS Provider

APAR status

• Closed as program error.

Error description

• When using the Default JMS Provider, JMS messages may remain
  stuck on the queues in either committing or removing state. The
  messages remain in these states even after expiry of TTL.
  The SystemOut.log of the JVM hosting the messaging engine shows
  errors like the following:
  
  
  CWSIC2009E: There was a failure to send a message: exception
  com.ibm.ws.sib.processor.exceptions.SIMPErrorException:
  com.ibm.websphere.sib.exception.SIErrorException:
  CWSIS1002E: An unexpected exception was caught during
  transaction completion.
  Exception: java.lang.RuntimeException: Found invalid balance
  factor: -1.
  
  CWSIP0002E: An internal messaging error occurred in
  com.ibm.ws.sib.processor.impl.ConsumerDispatcher,
  1:786:1.280.7.4,
  com.ibm.ws.sib.msgstore.TransactionException:
  CWSIS1002E: An unexpected exception was caught during
  transaction completion.
  Exception: java.lang.RuntimeException: Found invalid balance
  factor: -1
  
  CWSIP0002E: An internal messaging error occurred in
  com.ibm.ws.sib.processor.impl.JSLockedMessageEnumeration,
  1:272:1.8.1.10,
  com.ibm.ws.sib.msgstore.TransactionException:
  CWSIS1002E: An unexpected exception was caught during
  transaction completion.
  Exception: java.lang.NullPointerException
  
  CWSIP0003E: An internal messaging error occurred
  JMS/OperationFeedbackCmOpenApi in
  com.ibm.ws.sib.processor.impl.destination.PubSubRealization,
  1:2646:1.35.2.5,
  com.ibm.ws.sib.msgstore.TransactionException: CWSIS1002E: An
  unexpected exception was caught during transaction completion.
  Exception:
  com.ibm.ws.sib.msgstore.gbs.GBSTreeException:
  optimisticDelete, v1 = 5384170
  
  FFDC is produced with the following stack:
  
  Exception:com.ibm.ws.sib.msgstore.MessageStoreRuntimeException
  SourceId:com.ibm.ejs.util.am._Alarm.run ProbeId:95
  Reporter:com.ibm.ejs.util.am._Alarm@2620b3b3
  
  com.ibm.ws.sib.msgstore.MessageStoreRuntimeException:
  CWSIS2001E: The expiry daemon ended abnormally due to an
  unexpected exception:
  com.ibm.ws.sib.msgstore.gbs.OptimisticDepthException: maxDepth
  (47) exceeded. Depth is 48.
  
  at
  com.ibm.ws.sib.msgstore.expiry.Expirer.alarm(Expirer.java:598)
  at com.ibm.ejs.util.am._Alarm.runImpl(_Alarm.java:151)
  at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:136)
  at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)
  Caused by: com.ibm.ws.sib.msgstore.gbs.OptimisticDepthException:
  maxDepth (47) exceeded. Depth is 48.
  at
  com.ibm.ws.sib.msgstore.gbs.NodeStack.push(NodeStack.java:155)
  at
  com.ibm.ws.sib.msgstore.gbs.NodeStack.push(NodeStack.java:143)
  at
  com.ibm.ws.sib.msgstore.gbs.GBSIterator.nextNode(GBSIterator.jav
  a:704)
  at
  com.ibm.ws.sib.msgstore.gbs.GBSIterator.internalGetNext(GBSItera
  tor.java:656)
  at
  com.ibm.ws.sib.msgstore.gbs.GBSIterator.pessimisticGetNext(GBSIt
  erator.java:613)
  at
  com.ibm.ws.sib.msgstore.gbs.GBSIterator.findNextBeforeEof(GBSIte
  rator.java:411)
  at
  com.ibm.ws.sib.msgstore.gbs.GBSIterator.findNext(GBSIterator.jav
  a:381)
  at
  com.ibm.ws.sib.msgstore.gbs.GBSIterator.next(GBSIterator.java:75
  1)
  at
  com.ibm.ws.sib.msgstore.expiry.ExpiryIndex.next(ExpiryIndex.java
  :151)
  at
  com.ibm.ws.sib.msgstore.expiry.Expirer.alarm(Expirer.java:526)
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  WebSphere Application Server users of the   *
  *                  Default JMS Provider (SIBus)                *
  ****************************************************************
  * PROBLEM DESCRIPTION: For systems using message expiry under  *
  *                      heavy loads, processing of expired      *
  *                      messages may fail and leave messages    *
  *                      locked.                                 *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  Messages seen stuck in committing or removing state.
  

Problem conclusion

• The SIBus message store component was modified to fix an
  internal concurrency error.
  
  The fix for this APAR is targeted for inclusion in fix packs
  8.5.5.19 and 9.0.5.7. For more information, see 'Recommended
  Updates for WebSphere Application Server':
  https://www.ibm.com/support/pages/node/715553
  

Temporary fix

• Restarting the messaging engine will restore normal operation.
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBS APP SERV N

• Fixed component ID

  5724H8800

Applicable component levels

• R850 PSY

     UP

• R900 PSY

     UP