Data Streamer issues a message about Java out of memory when a target subscriber remains unresponsive for a long time
When a target subscriber is not available for a long time, the Data Streamer issues message HBO6113W and HBO6114W, and the message JVMDUMP039I appears in the log.
This message states how much of the Java™ Heap size is used, which is 87% in the above case. The Data Streamer can even start discarding events after the 90% of Heap size is exceeded as per message HBO6114W:
HBO6113W The percentage of Data Streamer heap storage that is being used is 87.
The Data Streamer also gets the following message in the log:
HBO6114W Because the Data Streamer was using more than 90 percent of its available memory, bytes of data intended for subscriber SPLUNK have been discarded.
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2019/03/28 05:10:37 - please wait.
Data Streamer was not designed to buffer data for long periods of time while a subscriber is unavailable. Although the code changes in PTF UA96562 make it less likely for an out of memory condition, it's still possible. With PTF UA96562 applied, the data discards attempt to prevent the out of memory condition, but the longer the subscriber is down, the greater the chance that other Data Streamer processing, or a new large block of data will cause an out of memory condition.
After that, make sure when you start the Data Streamer, the following message is produced:
HBO6119I Default Heap:8GB Maximum Heap:8GB
If the subscriber will be unavailable for more than several hours, you should consider stopping Z Common Data Provider during the subscriber outage. When the subscriber becomes available, start the Log Forwarder with the warm start option and batch load any SMF data being collected via the System Data Engine batch procedure. You can use the message HBO6113W to automate all this batch load.