PH54677: WEBSPHERE APPLICATION SERVER SHUTDOWN MAY HANG WHEN CONNECTION VALIDATION RETRIES SPECIFIED

APAR status

• Closed as program error.

Error description

• The application server appears to hang when attempting to
  shutdown gracefully.  A javacore shows a long running thread
  executing inside the RecoveryManager.performResync method,
  waiting to obtain a connection.
  
  An example callstack from a javacore is shown below:
  
  3XMTHREADINFO      "Default : DMN0"
  J9VMThread:0x00000000019F4700, omrthread_t:0x00007EFFB04D2F80,
  java/lang/Thread:0x00000007025A9808, state:R, prio=5
  3XMJAVALTHREAD            (java/lang/Thread getId:0x69,
  isDaemon:true)
  3XMJAVALTHRCCL
  com/ibm/ws/bootstrap/ExtClassLoader(0x00000007021AF560)
  3XMTHREADINFO1            (native thread ID:0x6D92, native
  priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread
  flags:0x000000a1)
  3XMTHREADINFO2            (native stack address range
  from:0x00007EFF65519000, to:0x00007EFF65559000, size:0x40000)
  3XMCPUTIME               CPU usage total: 0.998471044 secs,
  current category="Application"
  3XMHEAPALLOC             Heap bytes allocated since last GC
  cycle=941240 (0xE5CB8)
  3XMTHREADINFO3           Java callstack:
  4XESTACKTRACE                at sun/nio/ch/Net.connect0(Native
  Method)
  4XESTACKTRACE                at
  sun/nio/ch/Net.connect(Net.java:481)
  4XESTACKTRACE                at
  sun/nio/ch/Net.connect(Net.java:473)
  4XESTACKTRACE                at
  sun/nio/ch/SocketChannelImpl.connect(SocketChannelImpl.java:662)
  5XESTACKTRACE                   (entered lock:
  java/lang/Object@0x00000007F9F2DB58, entry count: 1)
  5XESTACKTRACE                   (entered lock:
  java/lang/Object@0x00000007F9F2DB78, entry count: 1)
  5XESTACKTRACE                   (entered lock:
  java/lang/Object@0x00000007F9F2DB68, entry count: 1)
  4XESTACKTRACE                at
  java/nio/channels/SocketChannel.open(SocketChannel.java:200)
  4XESTACKTRACE                at
  oracle/net/nt/TimeoutSocketChannel.(TimeoutSocketChannel.java:81
  )
  ...
  4XESTACKTRACE                at
  oracle/jdbc/pool/OracleDataSource.getPhysicalConnection(OracleDa
  taSource.java:384)
  4XESTACKTRACE                at
  oracle/jdbc/xa/client/OracleXADataSource.getPooledConnection(Ora
  cleXADataSource.java:507)
  4XESTACKTRACE                at
  oracle/jdbc/xa/client/OracleXADataSource.getXAConnection(OracleX
  ADataSource.java:146)
  4XESTACKTRACE                at
  oracle/jdbc/xa/client/OracleXADataSource.getXAConnection(OracleX
  ADataSource.java:126)
  4XESTACKTRACE                at
  com/ibm/ws/rsadapter/spi/InternalGenericDataStoreHelper$1.run(In
  ternalGenericDataStoreHelper.java:1370)
  4XESTACKTRACE                at
  com/ibm/ws/security/util/AccessController.doPrivileged(AccessCon
  troller.java:118(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/ws/rsadapter/spi/InternalGenericDataStoreHelper.getPoole
  dConnection(InternalGenericDataStoreHelper.java:1389)
  4XESTACKTRACE                at
  com/ibm/ws/rsadapter/spi/WSRdbDataSource.getPooledConnection(WSR
  dbDataSource.java:2154)
  4XESTACKTRACE                at
  com/ibm/ws/rsadapter/spi/WSManagedConnectionFactoryImpl.getGoodC
  onnection(WSManagedConnectionFactoryImpl.java:3680)
  4XESTACKTRACE                at
  com/ibm/ws/rsadapter/spi/WSManagedConnectionFactoryImpl.createMa
  nagedConnection(WSManagedConnectionFactoryImpl.java:1699)
  4XESTACKTRACE                at
  com/ibm/ws/rsadapter/spi/WSManagedConnectionFactoryImpl.createMa
  nagedConnection(WSManagedConnectionFactoryImpl.java:1137)
  4XESTACKTRACE                at
  com/ibm/ejs/j2c/J2CXAResourceFactory$1.run(J2CXAResourceFactory.
  java:441)
  4XESTACKTRACE                at
  com/ibm/ws/security/util/AccessController.doPrivileged(AccessCon
  troller.java:118(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/ejs/j2c/J2CXAResourceFactory.getXAResource(J2CXAResource
  Factory.java:232)
  4XESTACKTRACE                at
  com/ibm/ws/Transaction/JTA/XARecoveryData.getXARminst(XARecovery
  Data.java:517)
  4XESTACKTRACE                at
  com/ibm/ws/Transaction/JTA/XARecoveryData.recover(XARecoveryData
  .java:718)
  4XESTACKTRACE                at
  com/ibm/tx/jta/impl/PartnerLogTable.recover(PartnerLogTable.java
  :433)
  4XESTACKTRACE                at
  com/ibm/tx/jta/impl/RecoveryManager.resync(RecoveryManager.java:
  1586)
  4XESTACKTRACE                at
  com/ibm/tx/jta/impl/RecoveryManager.performResync(RecoveryManage
  r.java:2348)
  4XESTACKTRACE                at
  com/ibm/ws/tx/jta/RecoveryManager.performResync(RecoveryManager.
  java:119)
  4XESTACKTRACE                at
  com/ibm/tx/jta/impl/RecoveryManager.run(RecoveryManager.java:230
  1)
  4XESTACKTRACE                at
  java/lang/Thread.run(Thread.java:825)
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server                                      *
  ****************************************************************
  * PROBLEM DESCRIPTION: Connection validation properties        *
  *                      specifying large numbers of retries or  *
  *                      retry intervals prolong application     *
  *                      server shutdown                         *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  Datasources may be configured with connection validation
  properties that specify large numbers of retries and long
  retry intervals.  If such a datasource is a partner in the
  transaction partner log at application restart, then the
  transaction service will try to recover that partner and the
  connection validation properties are applied.  Should that
  partner be non-contactable, either because it is currently
  unavailable or the data is for an old, no-longer existent
  partner, then the reconnections may take a considerable amount
  of time.
  When server shutdown is initiated, the shutdown thread must
  wait for the transaction service recovery thread to complete
  before progressing the shutdown and it indicates this to the
  transaction service, however the transaction service recovery
  thread cannot process this event until the connection retry
  loop has returned.
  

Problem conclusion

• Transaction service recovery of partners read from the partner
  log at restart is retried by the transaction service itself in
  the event that a connection cannot be made to a recovered
  partner.  There is no need for the datasource's connection
  validation properties to be used to apply retries.  The J2C
  component was modified so that when the transaction service
  attempts to recover an XAResource, the datasource's connection
  validation properties relating to retries are not applied,
  allowing the transaction service itself to perform retries
  when necessary and allowing it to terminate in a timely
  fashion when an application server shutdown is initiated.
  
  The fix for this APAR is targeted for inclusion in fix pack
  9.0.5.17. For more information, see 'Recommended Updates for
  WebSphere Application Server':
  https://www.ibm.com/support/pages/node/715553
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBSPHERE APP S

• Fixed component ID

  5724J0800

Applicable component levels