Question & Answer
Question
When the QRadar Console reports an out of memory error, what information is necessary to gather?
Cause
Out of memory errors trigger when a process tries to request or allocate more memory than the defined maximum memory allocation for that process.
Symptoms
When an out of memory error triggers, the following messages are received in
GUI
Event processor
[main] com.q1labs.frameworks.core.ThreadExceptionHandler: [ERROR] [NOT:0030003100][IPADDRESS/- -] [-/- -]Out of memory discovered
Note
The bold IPADDRESS is the host that is having the out of memory error. |
In
/var/log/qradar.log
OutOfMemoryMonitor[12345]: Starting out-of-memory monitoring (enabled: yes)…
OutOfMemoryMonitor[12345]: Discovered out-of-memory error for [servicename] process.
OutOfMemoryMonitor[12345]: out-of-memory TYPE: [servicename], PID: 123456
OutOfMemoryMonitor[12345]: Discovered out-of-memory error for [servicename](type: [servicename], pid: 123456).
[servicename][123456]: JVMDUMP030E Cannot write dump to file /store/jheap/[servicename]/[servicename].*.dmp: File exists
NoteThe bold [servicename] is the process having the out of memory error. |
Answer
Review the following common causes of out of memory errors:
In case that none of the previous documentations solved the out of memory error, gather the following information before you restart the service that is affected and create the support case:
- Errors found in the
/var/log/qradar.log
- Check
/store/jheap/[servicename]/
to see whether a dump file name: "[servicename].system.dmp
" or "[servicename].javacore.dmp
" was created at the same date and time of the out of memory error reported:
# cd /store/jheap/[servicename]/
# ls -lh
- If the dump file does not match the date and time, and there are error messages like in the next example in
/var/log/qradar.log
:
[servicename][123456]: JVMDUMP030E Cannot write dump to file /store/jheap/[servicename]/[servicename].*.dmp: File exists
Erase the old dump file with the following command to allow a new dump to be created:# rm -r /store/jheap/[servicename]/[servicename].*.dmp
- If the dump file was created and it matches the date and time, create the following folder:
# mkdir /store/ibm_support/[servicename-date
]
- Compress the file by using the
gzip
tool and move the file:
# gzip [servicename].*.dmp
]
# mv /store/jhead/[servicename]/file.gz /store/ibm_support/[servicename-date
- If the dump file does not match the date and time, and there are error messages like in the next example in
- If the dump created a
core.[pid]
file in the same directory/store/jheap/[servicename]/
, note the timestamp and file size, but it is not necessary to gather.
- Gather get_logs from console and affected managed hosts. For more information, see How to collect log files for QRadar support from the user interface.
- If you have a security policy about providing logs to support teams, you can sanitize the logs before you upload them to your support case. For more information, see QRadar: Sanitizing logs before you open a support case.
- Create a case with support for the specific service that is having the out of memory error and provide the information gathered.
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
31 January 2023
UID
ibm16568037