IJ37364: TWS MDM DB POOL EXHAUSTION DUE WRONG FORMAT OF AGENT ID ON THE DA DEFINITION

10.1.0.1-2022.11-IWS-DOCKER-SERVER-LNX_Z
10.1.0.1-2022.11-IWS-DOCKER-SERVER-LNX_X86_64
10.1.0.1-2022.11-IWS-DOCKER-ZAGENT-LNX_Z
10.1.0.1-2022.11-IWS-DOCKER-ZAGENT-LNX_X86_64
10.1.0.1-2022.11-IWS-DOCKER-AGENT-LNX_Z
10.1.0.1-2022.11-IWS-DOCKER-AGENT-LNX_X86_64
10.1.0.1-2022.11-IWS-SERVER-WINDOWS
10.1.0.1-2022.11-IWS-SERVER-LNX_Z
10.1.0.1-2022.11-IWS-SERVER-LNX_X86_64
10.1.0.1-2022.11-IWS-SERVER-AIX
10.1.0.1-2022.11-IWS-ZAGENT-WINDOWS
10.1.0.1-2022.11-IWS-ZAGENT-SOLARIS
10.1.0.1-2022.11-IWS-ZAGENT-LNX_Z
10.1.0.1-2022.11-IWS-ZAGENT-LNX_X86_64
10.1.0.1-2022.11-IWS-ZAGENT-LNX_PPC_LE
10.1.0.1-2022.11-IWS-ZAGENT-IBMi
10.1.0.1-2022.11-IWS-ZAGENT-HPIA64
10.1.0.1-2022.11-IWS-ZAGENT-AIX
10.1.0.1-2022.11-IWS-AGENT-WINDOWS
10.1.0.1-2022.11-IWS-AGENT-SOLARIS
10.1.0.1-2022.11-IWS-AGENT-LNX_Z
10.1.0.1-2022.11-IWS-AGENT-LNX_X86_64
10.1.0.1-2022.11-IWS-AGENT-LNX_PPC_LE
10.1.0.1-2022.11-IWS-AGENT-IBMi
10.1.0.1-2022.11-IWS-AGENT-HPIA64
10.1.0.1-2022.11-IWS-AGENT-AIX

APAR status

• Closed as program error.

Error description

• The following can be observed from the MDM Liberty logs:
  
  WLTC0016E: Resource
  dataSource[oracle]/connectionManager[default-0] failed to
  complete. Exception stack trace follows:
  javax.transaction.xa.XAException
  at com.ibm.ejs.j2c.LocalTransactionWrapper.rollback(LocalTransa
  ctionWrapper.java:234)
  at com.ibm.tx.ltc.impl.LocalTranCoordImpl.cleanup(LocalTranCoor
  dImpl.java:1000)
  at com.ibm.tx.ltc.impl.LocalTranCoordImpl.end(LocalTranCoordImp
  l.java:1179)
  at com.ibm.tx.ltc.impl.LocalTranCurrentImpl.end(LocalTranCurren
  tImpl.java:425)
  at com.ibm.tx.ltc.impl.LocalTranCurrentSet.end(LocalTranCurrent
  Set.java:278)
  at com.ibm.ws.transaction.context.internal.TransactionContextIm
  pl.taskStopping(TransactionContextImpl.java:113)
  at com.ibm.ws.context.service.serializable.ThreadContextDescrip
  torImpl.taskStopping(ThreadContextDescriptorImpl.java:450)
  at com.ibm.ws.concurrent.internal.ManagedThreadImpl.run(Managed
  ThreadImpl.java:96)
  Caused by:
  com.ibm.ws.rsadapter.exceptions.DataStoreAdapterException:
  DSRA0080E: An exception was received by the Data Store Adapter.
  See original exception message: Closed Connection. with SQL
  State : 08003 SQL Code : 17008
  at com.ibm.ws.rsadapter.AdapterUtil.translateSQLException(Adapt
  erUtil.java:774)
  at com.ibm.ws.rsadapter.impl.WSRdbSpiLocalTransactionImpl.rollb
  ack(WSRdbSpiLocalTransactionImpl.java:406)
  at com.ibm.ejs.j2c.LocalTransactionWrapper.rollback(LocalTransa
  ctionWrapper.java:225)
  ... 7 more
  Caused by: java.sql.SQLRecoverableException: Closed Connection
  DSRA0010E: SQL State = 08003, Error Code = 17,008
  at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConne
  ction.java:2222)
  at oracle.jdbc.OracleConnectionWrapper.rollback(OracleConnectio
  nWrapper.java:154)
  at com.ibm.ws.rsadapter.impl.WSRdbSpiLocalTransactionImpl.rollb
  ack(WSRdbSpiLocalTransactionImpl.java:393)
  ... 8 more
  
  J2CA0046E: Method reserve caught an exception during creation of
  the ManagedConnection for resource jdbc/twsdb, throwing
  ResourceAllocationException. Original exception: Pool requests
  blocked, connection pool is being shut down.
  
  SRVE0315E: An exception occurred: java.lang.Throwable:
  java.lang.NullPointerException
  
  WLTC0033W: Resource
  dataSource[oracle]/connectionManager[default-0] rolled back in
  cleanup of LocalTransactionContainment.
  WLTC0032W: One or more local transaction resources were rolled
  back during the cleanup of a LocalTransactionContainment.
  
  com.ibm.ejs.j2c.SharedPool I J2CA0086I: Multiple open connection
  handles cannot share a physical connection to {1} within a local
  transaction containment boundary, so a different physical
  connection will be used for each connection handle. In this
  situation, it is important to ensure that no transactional work
  remains in progress after connection handle close because if
  additional sharable connection handles are later requested
  within the local transaction containment boundary, the physical
  connection to which they associate will be unpredictable.
  
  MCWrapper id ee5c0040 Managed connection
  WSRdbManagedConnectionImpl@6ae9742f
  State:STATE_TRAN_WRAPPER_INUSE
  dataSource[oracle]/connectionManager[default-0]
  
  DB is not responding anymore due to number or connections
  exceeded 500. After DB restart, we still keep getting the same
  errors in messages.log and the number of simultaneous DB
  connections seem to be increasing again.
  
  
  They're sharing the connection which is then not being
  properly closed/committed which causes the multiple shared
  connections to be rolled back. As this behavior continues the
  connections slowly build up until they hit the max connection
  limit. The WLTC0033W and WLTC0032W warnings occur typically
  because there are unresolved local transactions and the
  webcontainer tries to cleanup those outstanding transactions.
  
  The following errors can be found on the DB side:
  
  ORA-00020: maximum number of processes (500) exceeded
  
  Upon investigation, it was found out that during resource update
  phase there was a java exception that was not handled because of
  wrong format provided on the agent ID on a dynamic agent.
  

Local fix

• TS007668377
  

Problem summary

•  See apar description
  

Problem conclusion

•  This apar will be fixed into 94 FP8
  95. FP7 AND 10.1 fp1
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TIV WKLD SCHDL

• Fixed component ID

  5698WKB95

Applicable component levels