PH04833: JAVA BATCH SCHEDULER IN WEBSPHERE V8.5.5 RUNNING OUTOFMEMORY

9.0.5.0: WebSphere Application Server traditional Version 9.0.5 Refresh Pack
9.0.5.1: WebSphere Application Server traditional Version 9.0.5 Fix Pack 1
9.0.5.2: WebSphere Application Server traditional Version 9.0.5 Fix Pack 2
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
9.0.5.3: WebSphere Application Server traditional Version 9.0.5 Fix Pack 3
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21

APAR status

• Closed as program error.

Error description

• In a multi-JVM  Websphere v8.5.5 environment, java batch
  scheduler runs OutOfMemory
  
  Systemout log shows the following hung threads:
  
  
  UTLS0008W: The return of alarm thread "Deferrable Alarm : 1"
  (0000009b)
  to the alarm thread pool has been delayed for 10266
  milliseconds. This
  may be preventing normal alarm function within the application
  server.
  The alarm listener stack trace is as follows:
  
  at java.lang.ClassLoader.loadClass(ClassLoader.java:749)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:731)
  
  ...
  WSVR0605W: Thread "SoapConnectorThreadPool : 23" (000000d2) has
  been
  active for 662105 milliseconds and may be hung. There is/are 1
  thread(s) in total in the server that may be hung.
  at com.ibm.db2.jcc.am.jc.l(jc.java:2007)
  at com.ibm.db2.jcc.am.jc.e(jc.java:1648)
  at com.ibm.db2.jcc.am.kc.K(kc.java:1303)
  at com.ibm.db2.jcc.am.ResultSet.getLongX(ResultSet.java:818)
  at com.ibm.db2.jcc.am.ResultSet.getLong(ResultSet.java:798)
  at com.ibm.db2.jcc.am.ResultSet.getLong(ResultSet.java:1626)
  at
  com.ibm.ws.rsadapter.jdbc.WSJdbcResultSet.getLong(WSJdbcResult
  Set.java:1594)
  at
  com.ibm.ws.batch.JobUsageStoreImpl.getPrimaryKeys(JobUsageStore
  Impl.java:434)
  at
  com.ibm.ws.batch.JobUsageStoreImpl.getAllPrimayKeys(JobUsageStor
  e
  Impl.java:620)
  ..
  DCSV9416E: An internal error occurred. Exception is
  com.ibm.ws.dcs.
  common.exception.DCSTransportLayerException: HeartbeatProcessor
  at
  com.ibm.ws.dcs.vri.transportAdapter.rmmImpl.common.LogAdapter.
     handle9416(LogAdapter.java:257)
  at
  com.ibm.ws.dcs.vri.transportAdapter.rmmImpl.common.LogAdapter.
     handleNls(LogAdapter.java:170)
  at
  com.ibm.ws.dcs.vri.transportAdapter.rmmImpl.common.LogAdapter.
     onLogEvent(LogAdapter.java:131)
  at com.ibm.rmm.util.RmmLogger.baseLog(RmmLogger.java:245)
  at com.ibm.rmm.ptl.tchan.receiver.HeartbeatProcessor.run
     (HeartbeatProcessor.java:449)
  Caused by: java.lang.OutOfMemoryError: Java heap space
  at
  com.ibm.rmm.ptl.tchan.receiver.HeartbeatProcessor.run(Heartbeat
  Processor.java:371)
  
  .
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server Java Batch                           *
  ****************************************************************
  * PROBLEM DESCRIPTION: When the option to store Job usage      *
  *                      metrics in the job scheduler database   *
  *                      is enabled, if there are already a      *
  *                      very large number of entries in the     *
  *                      job usage table, an OutOfMemory         *
  *                      exception can occur.                    *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  Currently when a job usage entry needs to be added or updated,
  the entire contents of the job usage table are loaded by the
  runtime code.  After which they are searched and updated
  accordingly.  When the job usage table has a large number of
  entries, this can result in an OutOfMemory exception.  This
  condition can occur if older jobs are not periodically purged
  and the table size grows large.
  

Problem conclusion

• A code update has been made to more efficiently process the
  required job usage updates to be made in the job scheduler
  database.
  
  The fix for this APAR is currently targeted for inclusion in
  fix pack 9.0.5.0 and 8.5.5.16.  Please refer to the Recommended
  Updates page for delivery information:
  http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBS APP SERV N

• Fixed component ID

  5724H8800

Applicable component levels

• R850 PSY

     UP

• R900 PSY

     UP