Data Streamer issues a message about Java out of memory when a target subscriber remains unresponsive for a long time

Edit online

When a target subscriber is not available for a long time, the Data Streamer issues message HBO6113W and HBO6114W, and the message JVMDUMP039I appears in the log.

Symptom

When a target subscriber is not available for a long time, the Data Streamer issues message HBO6113W:

HBO6113W The percentage of Data Streamer heap storage that is being used is 87.

This message states how much of the Java™ Heap size is used, which is 87% in the above case. The Data Streamer can even start discarding events after the 90% of Heap size is exceeded as per message HBO6114W:

HBO6114W Because the Data Streamer was using more than 90 percent of  its available memory, bytes of data intended for subscriber SPLUNK have been discarded.

The Data Streamer also gets the following message in the log:

JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2019/03/28 05:10:37 - please wait.

Cause

Data Streamer was not designed to buffer data for long periods of time while a subscriber is unavailable. Although the code changes in PTF UA96562 make it less likely for an out of memory condition, it's still possible. With PTF UA96562 applied, the data discards attempt to prevent the out of memory condition, but the longer the subscriber is down, the greater the chance that other Data Streamer processing, or a new large block of data will cause an out of memory condition.

Solution

Make the Java heap size as large as possible and ensure that the subscriber is available as much as possible. The current default size is 4 GB. In order to increase (for example to the value of 8 GB) the Default and Maximum Heap size the Data Streamer can use, specify the following lines in the JCL (under //STDENV DD *):

DEFAULT_HEAP=8GB                             
MAXIMUM_HEAP=8GB

After that, make sure when you start the Data Streamer, the following message is produced:

HBO6119I Default Heap:8GB Maximum Heap:8GB

If the subscriber will be unavailable for more than several hours, you should consider stopping Z Common Data Provider during the subscriber outage. When the subscriber becomes available, start the Log Forwarder with the warm start option and batch load any SMF data being collected via the System Data Engine batch procedure. You can use the message HBO6113W to automate all this batch load.