IT23288: JAVA/LANG/OUTOFMEMORYERROR JAVACORES IN DEVICE/LOG DIRECTORY

IBM Spectrum Control V5.2.17 (May 2018)

APAR status

• Closed as program error.

Error description

• A large number of *TestConnection directories can accumulate in
  both in device/log and device/log/epm. When Spectrum Control
  lists them out to delete them, the ExternalProcess crashes with
  OutOfMemory errors.  This results in javacores in device/log.
  
  grep "Dump Event.*Java heap space" javacore.20171*
  javacore.20171014.221245.15024.0003.txt:1TISIGINFO     Dump
  Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError"
  "Java heap space" received
  1XMCURTHDINFO  Current thread
  3XMTHREADINFO      "LogAndTraceManager"
  J9VMThread:0x0000000001903200, j9thread_t:0x0000000014A18588,
  java/lang/Thread:0x00000000E00939F8, state:R, prio=5
  3XMJAVALTHREAD            (java/lang/Thread getId:0x17,
  isDaemon:true)
  3XMTHREADINFO1            (native thread ID:0x4464, native
  priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread
  flags:0x00000020)
  3XMCPUTIME               CPU usage total: 0.670804300 secs,
  user: 0.405602600 secs, system: 0.265201700 secs, current
  category="Application"
  3XMHEAPALLOC             Heap bytes allocated since last GC
  cycle=0 (0x0)
  3XMTHREADINFO3           Java callstack:
  4XESTACKTRACE                at
  java/io/WinNTFileSystem.list(Native Method)
  4XESTACKTRACE                at
  java/io/File.list(File.java:1133)
  4XESTACKTRACE                at
  java/io/File.listFiles(File.java:1297)
  4XESTACKTRACE                at
  com/ibm/srm/utils/logging/FileUtils.deleteRecursive(FileUtils.ja
  va:208)
  4XESTACKTRACE                at
  com/ibm/srm/utils/logging/FileUtils.deleteRecursive(FileUtils.ja
  va:191)
  4XESTACKTRACE                at
  com/ibm/srm/utils/logging/LogAndTraceManager$3.run(LogAndTraceMa
  nager.java:231)
  4XESTACKTRACE                at
  java/util/TimerThread.mainLoop(Timer.java:566)
  4XESTACKTRACE                at
  java/util/TimerThread.run(Timer.java:516)
  
  javacore.20171110.150105.15804.0003.txt:1TISIGINFO     Dump
  Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError"
  "native memory exhausted" received
  3XMTHREADINFO      "LogAndTraceManager"
  J9VMThread:0x00000000017AAE00, j9thread_t:0x00000000143EF208,
  java/lang/Thread:0x00000000E024A648, state:R, prio=5
  3XMJAVALTHREAD            (java/lang/Thread getId:0x17,
  isDaemon:true)
  3XMTHREADINFO1            (native thread ID:0x43C0, native
  priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread
  flags:0x00000001)
  3XMCPUTIME               CPU usage total: 0.421202700 secs,
  user: 0.140400900 secs, system: 0.280801800 secs, current
  category="Application"
  3XMHEAPALLOC             Heap bytes allocated since last GC
  cycle=0 (0x0)
  3XMTHREADINFO3           Java callstack:
  4XESTACKTRACE                at
  java/io/WinNTFileSystem.getBooleanAttributes(Native Method)
  4XESTACKTRACE                at
  java/io/File.isDirectory(File.java:860(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/srm/utils/logging/FileUtils$3.accept(FileUtils.java:188)
  4XESTACKTRACE                at
  java/io/File.listFiles(File.java:1302(Compiled Code))
  4XESTACKTRACE                at
  com/ibm/srm/utils/logging/FileUtils.deleteRecursive(FileUtils.ja
  va:208)
  4XESTACKTRACE                at
  com/ibm/srm/utils/logging/FileUtils.deleteRecursive(FileUtils.ja
  va:191)
  4XESTACKTRACE                at
  com/ibm/srm/utils/logging/LogAndTraceManager$3.run(LogAndTraceMa
  nager.java:231)
  4XESTACKTRACE                at
  java/util/TimerThread.mainLoop(Timer.java:566)
  4XESTACKTRACE                at
  java/util/TimerThread.run(Timer.java:516)
  

Local fix

• As a temporary workaround, the *TestConnection directories from
  device/log and device/log/epm can be deleted.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Control 5.2.x users with frequent connection    *
  * failures to SVC and/or DS8K devices.                         *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * If SVC or DS8K performance monitor cannot connect to the     *
  * storage subsystem then a testConnection directory was        *
  * created under device/epm directory.                          *
  * Those directories were infrequently or not at all deleted.   *
  * The large accumulation of those directories generated out of *
  * memory errors.                                               *
  * The fix was to clean up those directories more frequently.   *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fix maintenance when available                         *
  ****************************************************************
  

Problem conclusion

• The fix for this APAR is contained in the following maintenance
  package:
  
  IBM Spectrum Control 5.2.17  |  refresh pack  |
  5.2-TIV-TPC-RP0017  |  May 2018
  
  http://www.ibm.com/support/docview.wss?&uid=swg21320822
  
  The target dates for future releases do not represent a formal
  commitment by IBM. The dates are subject to change without
  notice.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TPC ADVANCED

• Fixed component ID

  5608TPCA0

Applicable component levels