PI99507: NATIVE OUTOFMEMORY ERRORS DUE THREAD LEAK IN OTIS CONNECTION HANDLING

9.0.0.10: WebSphere Application Server traditional V9.0 Fix Pack 10
8.5.5.15: WebSphere Application Server V8.5.5 Fix Pack 15
9.0.0.11: WebSphere Application Server traditional V9.0 Fix Pack 11
9.0.5.0: WebSphere Application Server traditional Version 9.0.5 Refresh Pack
9.0.5.1: WebSphere Application Server traditional Version 9.0.5 Fix Pack 1
9.0.5.2: WebSphere Application Server traditional Version 9.0.5 Fix Pack 2
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
9.0.5.3: WebSphere Application Server traditional Version 9.0.5 Fix Pack 3
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21

APAR status

• Closed as program error.

Error description

• In rare cases the OTiS Derby Database might be locked. As a
  result, connections to the Derby DB are piling up and causing
  the native OutOfMemory Error. Deletion of the lock file solved
  the issue temporarily.
  
  JavaCores capturing the thread leak will show following
  callstack:
  
  3XMTHREADINFO "derby.rawStoreDaemon"
  ...
  3XMTHREADINFO3 Java callstack:
  4XESTACKTRACE at java/lang/Object.wait(Native Method)
  4XESTACKTRACE at java/lang/Object.wait(Object.java:201(Compiled
  Code))
  4XESTACKTRACE at
  org/apache/derby/impl/services/daemon/BasicDaemon.rest(Bytecode
  PC:22(Compiled Code))
  5XESTACKTRACE (entered lock:
  org/apache/derby/impl/services/daemon/BasicDaemon@0x0000000748AA
  CB40, entry count: 1)
  4XESTACKTRACE at
  org/apache/derby/impl/services/daemon/BasicDaemon.run(Bytecode
  PC:22(Compiled Code))
  4XESTACKTRACE at java/lang/Thread.run(Thread.java:785)
  

Local fix

• Deleting the derby lock file db.lck temporary fix the problem.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server Network Deployment version 8.5.5 or  *
  *                  later                                       *
  ****************************************************************
  * PROBLEM DESCRIPTION: OutOfMemoryError may occur on           *
  *                      deployment manager or job manager       *
  *                      server when there is an issue with      *
  *                      closing database.                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  OutOfMemoryError may occur on deployment manager
  or job manager server if there is a problem in closing the
  database that is used by the internal OTiS application.
  

Problem conclusion

• The code is updated to ensure the database is shut down if
  there is an issue in closing the database.
  
  The fix for this APAR is currently targeted for inclusion in
  fix pack 8.5.5.15 and 9.0.0.10.  Please refer to the
  Recommended
  Updates
  page for delivery information:
  http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBS APP SERV N

• Fixed component ID

  5724H8800

Applicable component levels

• R850 PSY

     UP

• R900 PSY

     UP