Troubleshooting
Problem
When the /home partition in QRadar does not have enough space, it can affect the regular functioning of QRadar® SIEM. The purpose of this article is to help the administrator with the removal of files and directories when the /home partition has not enough available disk space.
Symptom
Lack of available space in the /home partition can cause the following issues:
- Alerts about "Process monitor application failed to start multiple times".
- Searches reporting I/O errors.
- Services not starting.
- Configuration deployment changes due to critical disk space.
[tomcat.tomcat] /console/JSON-RPC/QRadar.scheduleDeployment QRadar.scheduleDeployment] com.q1labs.configservices.util.ConfigServicesUtil: [INFO] [-/--] Deployment is blocked due to critical disk space issue
Cause
By default, the QRadar disk sentry check runs every 60 seconds and looks for high disk usage across the partitions. When a partition goes beyond the critical warning threshold, an alert is triggered for administrators to investigate.
Diagnosing The Problem
Administrators can identify the largest directories and files by following the steps in Troubleshooting disk space usage problems. Once identified, compare them with the following list.
drwxr-xr-x. 9 root root 119 Oct 12 18:27 .
dr-xr-xr-x. 22 root root 4.0K Oct 12 18:28 ..
drwx------ 3 customactionuser customactionuser 78 Jun 10 16:56 customactionuser
drwx------ 3 fusionvm fusionvm 78 Jun 10 17:06 fusionvm
drwx------ 4 qniconfiguser qniconfiguser 90 Jun 10 16:52 qniconfiguser
drwx------ 2 qradar qradar 62 Jun 10 16:36 qradar
drwx------ 2 qvmuser qvmuser 62 Jun 10 16:32 qvmuser
drwx------ 2 solr solr 62 Jun 10 16:40 solr
drwx------ 3 vis qradar 78 Jun 10 16:55 vis
The following example shows the /home/backup20220929/ directory using 800MB. This directory is not in the list, therefore, it's likely a directory that can be deleted.
[root@qradar ]# du -xch -d 1 /home | sort -h | tail -n 5
12K /home/vis
800M /home/backup20220929
801M /home
The following example shows the /home/backup.tar.gz file using 800MB. This file is likely a leftover that can be deleted. By default, no file exists in /home outside the directories in the previous list.
[root@qradar ]# find /home -type f -size +100M -exec ls -lah {} \;
-rw-r--r-- 1 root root 800M Oct 12 16:33 /home/backup.tar.gz
Once these large directories and files are identified, follow the instructions in Resolving the Problem to remove them.
Resolving The Problem
Use the following instructions to identify safe to remove files and regain space.
Depending on the directory reported during diagnosis, follow the suggestions provided. You might follow some or all of the suggestions, depending on your needs.
- Move or remove user leftover files.
To move the file:mkdir -pv /store/IBM_Support/ mv -v /home/<file> /store/IBM_Support/
‘/home/backup.tar.gz’ -> ‘/store/IBM_Support/backup.tar.gz’ removed ‘/home/backup.tar.gz’
To remove the file:rm -fv /home/<file>
Output Example:removed ‘/home/backup.tar.gz’
- Move or remove the conflicting directory.
To move the directory:mkdir -pv /store/IBM_Support/ mv /home/<directory> /store/IBM_Support/
‘/home/backup20220929/’ -> ‘/store/IBM_Support/backup20220929’ removed directory: ‘/home/backup20220929/’
To remove the directory:rm -rfv /home/<directory>
removed directory: ‘/home/backup20220929/’
- Verify the partition usage decreased.
df -Th /home
Output ExampleFilesystem Type Size Used Avail Use% Mounted on /dev/mapper/rootrhel-home xfs 1019M 33M 987M 4% /home
Result
The /home partition no longer has disk space constraints. If the partition reached the point of critical services stop, restart the services in the proper order and wait 5 mins with the following commands:
IMPORTANT: When the QRadar core service restart, the QRadar UI, event processing, and database are not available to all users. Administrators with strict outage policies are advised to complete the next step during a scheduled maintenance window for their organization.
systemctl stop hostcontext
systemctl stop tomcat
systemctl restart hostservices
systemctl start tomcat
systemctl start hostcontext
If the partition does not decrease its usage or the services do not start properly, contact QRadar Support for assistance.
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
19 October 2022
UID
ibm16829025