IBM Support

QRadar: /var/log and /var/log/audit fills to capacity due to logrotate issue



The /var/log and /var/log/audit partition can fill to capacity due to an issue with logrotate properly rotating files, caused by a decompressed file existing.


Lack of available space in the /var/log and /var/log/audit partition can cause the following symptoms:


The logrotate script runs hourly and verifies whether the log files need compression and be moved as a compressed file in /var/log/qradar.old and /var/log/audit or be removed.

When the partition does not have enough space, the script cannot create the compressed file due to lack of space, leading to an uncontrolled growth of the log file until the partition is filled up.

Another cause for logrotate to fail is having a decompressed file in /var/log/qradar.old. This file can exist due to a previous log rotation failure that broke the logrotate sequence.

Diagnosing The Problem

To verify there is a problem, run the following steps:
  1. SSH to the Console, then if necessary, the conflicting Managed Host.
  2. Run the steps in the QRadar: Troubleshooting disk space usage problems article.
  3. Run /etc/cron.hourly/logrotate and verify it reports an error like the following:
    ​error creating output file...{file_name}.1 already exists
  4. Run drq -t logrotate and verify it reports a similar error to any of the following:
    Available Space Checks
      Checks if /var/log has enough space
             Not enough space in /var/log: Available Space: 273 MB - File:
             /var/log/qradar.log 13312 MB. This will cause logrotate to fail.
             Free up space in /var/log. You need at least 13512 MB free.
      Logrotate Checks
      Verifies logrotate is running properly
             /var/log/qradar.old/qradar.log.2 is not a valid .gz file, should not be
             in /var/log/qradar.old.
             Recompress or remove /var/log/qradar.old/qradar.log.2 to resolve.
    [SUMMARY]  2 failed checkup

Note: You might see that it fails due to a file in /var/log/qradar.old/, /var/log/httpd/, or /var/log/audit/.

Resolving The Problem

Use the following instructions to identify safe to remove files and regain space.
IMPORTANT: Administrators must be cautious to move or remove the file /var/log/si-postgres-pam.log to avoid IJ39257.
Depending on the directory reported during diagnosis, follow the suggestions provided. You might follow some or all of the suggestions, depending on your needs.
  1. Back up the current file. If any issues are encountered after attempting the following steps, you will need to send this backup to QRadar Support.
    Note: The following example uses /var/log/qradar.log as the file taking all the capacity. Change the file to suit your needs.
    mkdir -pv /store/IBM_Support/
    cp -fv /var/log/qradar.log /store/IBM_Support/
  2. Truncate the file to release the space.
    ​truncate -s0 /var/log/qradar.log
  3. Compress any decompressed file preventing log rotation.
    gzip /var/log/qradar.old/<file name>
    Output example:
    gzip /var/log/qradar.old/qradar.log.1
  4. Verify no decompressed file exists.
    ls /var/log/qradar.old/ | grep -vE "gz$"
    ls /var/log/audit/ | grep -vE "audit.log.*.gz$"
    For /var/log/qradar.old, no output is generated. For /var/log/audit, the output must look similar to the following:
  5. Restart the syslog-ng service.
    Note: Restarting this service causes a small interruption in the writing of the system logs like /var/log/messages. If required, administrators must schedule a suitable maintenance window to do perform this restart.
    systemctl restart syslog-ng
  6. Verify the partition is now under normal values.
    df -Th /var/log /var/log/audit
  7. Run again the steps in the Diagnosing the Problem section to ensure no errors are reported.

    The /var/log and /var/log/audit partition no longer has disk space constraints. If the affected log was inside /var/log/httpd, administrators must restart the Tomcat service to release the space.
    systemctl restart tomcat
    If the partition does not decrease its usage, the functionality is not restored or the problem reoccurs often, contact QRadar Support for assistance and include the backup file.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
19 October 2022