Question & Answer
Question
Cause
The /var/log partition is a storage area for QRadar and system log files. Some of the common logs on this partition are qradar.log, qradar.error, qradar-ha.log, messages, and httpd.log.
The partition size and type varies based on the appliance type (Console, Event Processor, etc.), model (newer Console model has larger storage), hardware, software installation (customer appliance) or VM, and QRadar version.
By default, the QRadar disk sentry check runs every 60 seconds and looks for high disk usage across the /var/log partition. If the /var/log partition fills up to critical levels (95%), it will not stop the QRadar critical services as some other partitions would.
To find out what files or directories are filling up the /var/log partition, see the Troubleshooting Disk Space Problems Technote:
Technote 0881013 - QRadar: Troubleshooting Disk Space Problems
Answer
Quick Links
- 1. Troubleshooting /var/log space issues
- 2. Defects around /var/log partition
- 3. General Information about the sizing of /var/log partition
1. Troubleshooting /var/log space issues
These are the most common issues that cause /var/log to fill. For specific information about troubleshooting /var/log space issues, see the following Technotes:
Technote 10719969 - QRadar: /var/log fills to capacity due to logrotate issue
Logrotate script fails to move the file needing rotated from /var/log/ to /var/log/qradar.old/ with a .1 extension. This is due to there already being an uncompressed file with a .1 extension in any of these folders.
It has been identified that changes made to logrotate in QRadar 7.3.1 Patch 6, where the logrotate file is missing from /etc/cron.hourly/, can cause the /var/log to run out of free space prematurely.
Technote 10794387 - QRadar: Deploy Changes fails with Error from Disk Space Issue
Due to a partition crossing maximum threshold, deploy changes fails to start and returns an error message popup window with message "Error performing deployment. See logs for details"
2. Defects around /var/log partition
This is a summary list of the common defects encountered on the /var/log partition:
It has been identified that /var/log/ on high availability secondary appliances can fill due to the /var/log/systemStabMon directories not being rotated.
It has been observed that the qradar.log and qradar.error can fill rapidly with test exception messages when rules containing the test "DoubleSequenceFunction_Test" are in use.
IV98932: /VAR/LOG/ PARTITION CAN BECOME FILLED DUE TO REPEATED TEST EXCEPTION MESSAGES BEING LOGGED
It has been observed in some customer environments that /var/log/qradar.log can become populated with repeated test exception messages.
It has been observed that the /var/log/ partition can run out of free space due to log files being quickly filled with messages similar to the following:
[ariel.ariel_query_server] [qw_1:8e73245a-8469-4f8e-8234-999a71c4682a] com.q1labs.frameworks.session. SessionContext: [INFO] [NOT:0000006000][127.0.0.1/- -] [-/- -]The UserSession object in SessionContext is null. Cannot get a valid security profile ID.
In some circumstances, /var/log/systemStabMon/ on a High Availability (HA) Secondary can fill up with uncompressed files and potentially cause the '/var/log/' partition of an HA Secondary to run out of free disk space. It has been determined that file deletion as part of HA Secondary disk maintenance is working as expected, but that file compression is not.
IV94515: WGET.LOG FILE CAN CONTRIBUTE TO THE /VAR/LOG PARTITION RUNNING OUT OF SUFFICIENT FREE SPACE
It has been observed that the wget.log file is not a part of the current set of rotated log files and there are instances where wget.log can grow too large in size.
It has been identified that changes made to logrotate in QRadar 7.3.1 Patch 6 can cause the /var/log and or the /opt partition to run out of free space prematurely.
3. General Information about the sizing of /var/log partition
Partition requirements and recommendations when upgrading:
During a software upgrade (for software installations only), partition requirements and recommendations are generated and stored in the /root/partition_instructions.txt file. This file is deleted during QRadar setup on the new operating system. If you choose not to use the partitions recommendations, make sure that you meet these partition requirements.
QRadar 7.3.1 Partition requirements and recommendations documentation
Note: The upgrade mentioned here is when upgrading from 7.2.8 to 7.3.x, as it also upgrades the underlying Operating System. If you are upgrading to 7.3.0, you can use the drop-down in the IBM Knowledge Center linked above to change to the 7.3.0 version.
Linux operating system partition properties for QRadar installations on your own hardware:
If you use your own appliance hardware, you can delete and re-create partitions on your Red Hat Enterprise Linux operating system rather than modify the default partitions.
Note: Once on the IBM Knowledge Center, you can change version from drop down for 7.2.8/7.3.0/7.3.1
Was this topic helpful?
Document Information
Modified date:
07 January 2021
UID
ibm10882056