JR47874: RESILIENT CONNECTION POOL FOR ES

7.5.1.1-WS-BPMADVWESB-IFJR47874
Version 8.0.1 Fix Pack 2 for the IBM Business Process Manager products
Version 7.5.1 Fix Pack 2 for the IBM Business Process Manager products
Version 8.5 Refresh Pack 5 for the IBM Business Process Manager products

APAR status

• Closed as program error.

Error description

• ES enabled component can stop processing messages when a
  connection from the ES pool receives a
  SIConnectionDroppedException. To overcome the situation an
  application restart is required. This APAR enhances error
  resiliency by explicitly checking if a
  SequencingRuntimeException was caused by an
  SIConnectionDroppedException (similar to below exception
  stacktrace). If this happens the ES connection pool for this
  particular application will be refreshed.
  
  [9/2/13 16:01:58:547 CST] 000000b5 LocalExceptio E   CNTR0020E:
  EJB threw an unexpected (non-declared) exception during
  invocation of method "processMessageInBifurcatedSession" on bean
  "BeanId(persistentLkMgr#persistentLkMgrEJB.jar#ESWork, null)".
  Exception data:
  com.ibm.wbiserver.sequencing.exceptions.SequencingRuntimeExcepti
  on: CWRES0015E: The message with msgId=7E3935DE7A2DC62D_2500245
  failed during processing. The error is CWSIV0155E: An internal
  error occurred. The exception
  javax.resource.spi.ResourceAllocationException: CWSIV0353E: An
  internal error occurred. The exception
  com.ibm.wsspi.sib.core.exception.SIConnectionDroppedException:
  CWSIJ0044E: An operation was attempted on a connection that is
  already closed. was thrown while attempting to clone a
  connection to the messaging engine. was thrown while attempting
  to re-associate with a managed connection..
   at
  com.ibm.wbiserver.sequencing.work.ESWorkBean.processMessage(ESWo
  rkBean.java:177)
   at
  com.ibm.wbiserver.sequencing.work.ESWorkBean.processMessageInBif
  urcatedSession(ESWorkBean.java:101)
   at
  com.ibm.wbiserver.sequencing.work.EJSLocalStatelessESWork_ddc8f6
  c2.processMessageInBifurcatedSession(Unknown Source)
   at
  com.ibm.wbiserver.sequencing.work.EsWorkImpl.run(EsWorkImpl.java
  :251)
   at
  com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java
  :268)
   at
  java.security.AccessController.doPrivileged(AccessController.jav
  a:250)
   at javax.security.auth.Subject.doAs(Subject.java:495)
   at
  com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:13
  1)
   at
  com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:89
  )
   at
  com.ibm.ws.asynchbeans.J2EEContext$DoAsProxy.run(J2EEContext.jav
  a:335)
   at
  java.security.AccessController.doPrivileged(AccessController.jav
  a:277)
   at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:777)
   at
  com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithE
  xecutionContextImpl.java:222)
   at
  com.ibm.ws.asynchbeans.ABWorkItemImpl.run(ABWorkItemImpl.java:15
  9)
   at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656)
  Caused by: com.ibm.websphere.sib.exception.SIResourceException:
  CWSIV0155E: An internal error occurred. The exception
  javax.resource.spi.ResourceAllocationException: CWSIV0353E: An
  internal error occurred. The exception
  com.ibm.wsspi.sib.core.exception.SIConnectionDroppedException:
  CWSIJ0044E: An operation was attempted on a connection that is
  already closed. was thrown while attempting to clone a
  connection to the messaging engine. was thrown while attempting
  to re-associate with a managed connection.
   at
  com.ibm.ws.sib.ra.impl.SibRaConnection.getAssociatedManagedConne
  ction(SibRaConnection.java:1846)
   at
  com.ibm.ws.sib.ra.impl.SibRaConnection.getContainerTransaction(S
  ibRaConnection.java:1939)
   at
  com.ibm.ws.sib.ra.impl.SibRaConnection.mapTransaction(SibRaConne
  ction.java:2046)
   at
  com.ibm.ws.sib.ra.impl.SibRaBifurcatedConsumerSession.deleteSet(
  SibRaBifurcatedConsumerSession.java:172)
   at
  com.ibm.wbiserver.sequencing.work.ESWorkBean.processMessage(ESWo
  rkBean.java:141)
   ... 14 more
  Caused by: javax.resource.spi.ResourceAllocationException:
  CWSIV0353E: An internal error occurred. The exception
  com.ibm.wsspi.sib.core.exception.SIConnectionDroppedException:
  CWSIJ0044E: An operation was attempted on a connection that is
  already closed. was thrown while attempting to clone a
  connection to the messaging engine.
   at
  com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(Fr
  eePool.java:2413)
   at
  com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java
  :1759)
   at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2631)
   at
  com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionMa
  nager.java:1064)
   at
  com.ibm.ejs.j2c.ConnectionManager.associateConnection(Connection
  Manager.java:2382)
   at
  com.ibm.ejs.j2c.ConnectionManager.associateConnection(Connection
  Manager.java:2315)
   at
  com.ibm.ws.sib.ra.impl.SibRaConnection.getAssociatedManagedConne
  ction(SibRaConnection.java:1813)
   ... 18 more
  Caused by:
  com.ibm.wsspi.sib.core.exception.SIConnectionDroppedException:
  CWSIJ0044E: An operation was attempted on a connection that is
  already closed.
   at
  com.ibm.ws.sib.jfapchannel.impl.Connection.preSendProcessing(Con
  nection.java:529)
   at
  com.ibm.ws.sib.jfapchannel.impl.Connection.send(Connection.java:
  672)
   at
  com.ibm.ws.sib.jfapchannel.impl.ConversationImpl.sendInternal(Co
  nversationImpl.java:1157)
   at
  com.ibm.ws.sib.jfapchannel.impl.ConversationImpl.exchange(Conver
  sationImpl.java:978)
   at
  com.ibm.ws.sib.comms.common.JFAPCommunicator.jfapExchange(JFAPCo
  mmunicator.java:386)
   at
  com.ibm.ws.sib.comms.client.ConnectionProxy.cloneConnection(Conn
  ectionProxy.java:2997)
   at
  com.ibm.ws.sib.ra.impl.SibRaManagedConnectionFactory.createManag
  edConnection(SibRaManagedConnectionFactory.java:248)
   at
  com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(Fr
  eePool.java:2082)
   ... 24 more
  

Local fix

• App restart
  

Problem summary

• This fix improves the resiliency of the connection pool of event
  
  sequencing. I.e. if certain exception happens in the messaging
  layer, the connection pool will be refreshed.
  
  PROBLEM DETAILED DESCRIPTION:
  Event sequencing enabled components can stop processing messages
  when a connection from the event sequencing pool receives a
  SIConnectionDroppedException, as listed below. Normally, to
  overcome the situation an application restart is required. Which
  is not always acceptable for customers.
  
  CNTR0020E: EJB threw an unexpected (non-declared) exception
  during invocation of method "processMessageInBifurcatedSession"
  on bean "BeanId(persistentLkMgr#persistentLkMgrEJB.jar#ESWork,
  null)".
  
  Exception data:
   com.ibm.wbiserver.sequencing.exceptions.
    SequencingRuntimeException: CWRES0015E: The message with
    msgId=7E3935DE7A2DC62D_2500245 failed during processing. The
    error is CWSIV0155E: An internal error occurred. The exception
    javax.resource.spi.ResourceAllocationException: CWSIV0353E: An
    internal error occurred. The exception
    com.ibm.wsspi.sib.core.exception.SIConnectionDroppedException:
    CWSIJ0044E: An operation was attempted on a connection that is
    already closed. was thrown while attempting to clone a
    connection to the messaging engine. was thrown while
    attempting to re-associate with a managed connection..
  

Problem conclusion

• This fix enhances error resiliency by explicitly checking if a
  SequencingRuntimeException was caused by an
  SIConnectionDroppedException.  If this happens the event
  sequencing connection pool for this particular application will
  be refreshed.
  
  
  FIX AVAILABILITY:
  Fix is targetted for inclusion in next fixpack for BPM V7.5.1,
  BPM 8.0.1, BPM 8.5.0
  
  When obtaining any of the above fixes, be sure to download the
  accompanying readme, for itself, and any prerequisite fixes, and
  review them thorougly.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  JR47613

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  BPM ADVANCED

• Fixed component ID

  5725C9400

Applicable component levels

• R750 PSN

     UP