IT32270: DATA COLLECTOR SERVICE STOPS UNEXPECTEDLY DUE TO CORE DUMPS. PROBES HANG. [OUTOFMEMORY]

IBM Spectrum Control 5.3.7 (May 2020)

APAR status

• Closed as program error.

Error description

• After upgrading the Storage Insights data collector to the 1Q20
  update (5.3.6) and running for a while, the probes would not
  complete. No data collection was occurring. Attempting to
  restart the data collector from Storage Insights was not
  successful. After stopping the data collector processes and
  restarting the service, the same problem reoccurred.
  
  Javacore,
  core, heapdump, and snap files can be found under the data
  collector log directory.
  
  javacore
  1TISIGINFO Dump Event
  "systhrow" (00040000) Detail "java/lang/OutOfMemoryError"
  "native memory exhausted" received
  
  The OutOfMemory errors
  occur because the batching of the devices of same device type
  is not working and more EP JVMs than needed are created.
  
  The
  following log message for each of the device PM processes
  suggests that each Batch contains only the PM task for 1 device
  instead of a group of them.
  [pool-6-thread-1]
  [TimerQueue$StartBatch.run] [INFO] BATCH Starting batch of
  process IDs
  [1584063881]([94ebd600-b405-11e8-ae35-9f980c9bb3a5]) and batch
  process ID -1021. Batch size: 1
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Storage Insights users with over 30 devices managed by a     *
  * single Data Collector                                        *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * After upgrading the Storage Insights Data Collector to the   *
  * 1Q20                                                         *
  * update (5.3.6) and running for a while, the probes would not *
  * complete. No data collection was occurring. Attempting to    *
  * restart the data collector from Storage Insights was not     *
  * successful. After stopping the data collector processes and  *
  * restarting the service, the same problem reoccurred.         *
  *                                                              *
  * Javacore, core, heapdump, and snap files can be found under  *
  * the data collector log directory.                            *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  The Out Of Memory errors were caused by too many external
  processes launched to run performance monitors.
  The fix for this APAR allows batching of several performance
  monitors in one external process.
  

Problem conclusion

• The fix for this APAR is targeted for the following release:
  
  IBM Storage Insights 2Q20   [ 5.3.7-IBM-SC ]
  
  ( release target 2Q 2020 / May )
  
  The target dates for future releases do not represent a formal
  commitment by IBM. The dates are subject to change without
  notice.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  STORAGE INSIGHT

• Fixed component ID

  5608TPCSI

Applicable component levels

• R536 PSY

     UP