IBM Support

QRadar: /var/log and /var/log/audit fills to capacity due to logrotate issue

Troubleshooting


Problem

The /var/log and /var/log/audit partition can fill to capacity due to an issue with logrotate properly rotating files, caused by a decompressed file existing.

Symptom

Lack of available space in the /var/log and /var/log/audit partition can cause the following symptoms:
  • Alerts about "Process monitor application failed to start multiple times"
  • Searches reporting I/O errors
  • Services not starting
  • Failed disk space checks when a software update runs
  • Database backup issues: 
    Backup Failed: com.q1labs.configservices.hostcontext.exception.BackupException: unable to process backup target of type
  • PSQL authentication issues with services: 
    org.postgresql.util.PSQLException: FATAL: PAM authentication failed for user "qradar"
  • Configuration deployment changes due to critical disk space 
    [tomcat.tomcat] /console/JSON-RPC/QRadar.scheduleDeployment QRadar.scheduleDeployment] com.q1labs.configservices.util.ConfigServicesUtil: 
[INFO] [-/--] Deployment is blocked due to critical disk space issue

Cause

The logrotate script runs hourly and verifies whether the log files need compression and be moved as a compressed file in /var/log/qradar.old and /var/log/audit or be removed.

When the partition does not have enough space, the script cannot create the compressed file due to lack of space, leading to an uncontrolled growth of the log file until the partition is filled up.

Another cause for logrotate to fail is having a decompressed file in /var/log/qradar.old. This file can exist due to a previous log rotation failure that broke the logrotate sequence.

Diagnosing The Problem

To verify there is a problem, run the following steps:
  1. SSH to the Console, then if necessary, the conflicting Managed Host.
  2. Run the steps in the QRadar: Troubleshooting disk space usage problems article.
  3. Run /etc/cron.hourly/logrotate and verify it reports an error like the following:
      
    ​error creating output file...{file_name}.1 already exists
  4. Run drq -t logrotate and verify it reports a similar error to any of the following:
      
    Available Space Checks
  Checks if /var/log has enough space
     [FAILURE]
         Not enough space in /var/log: Available Space: 273 MB - File:
         /var/log/qradar.log 13312 MB. This will cause logrotate to fail.
        [REMEDIATION]
         Free up space in /var/log. You need at least 13512 MB free.

  Logrotate Checks
  Verifies logrotate is running properly
     [FAILURE]
         /var/log/qradar.old/qradar.log.2 is not a valid .gz file, should not be
         in /var/log/qradar.old.
        [REMEDIATION]
         Recompress or remove /var/log/qradar.old/qradar.log.2 to resolve.

[SUMMARY]  2 failed checkup

Note: You might see that it fails due to a file in /var/log/qradar.old/, /var/log/httpd/, or /var/log/audit/.

Resolving The Problem

Use the following instructions to identify safe to remove files and regain space.
IMPORTANT: Administrators must be cautious to move or remove the file /var/log/si-postgres-pam.log to avoid IJ39257.
 
Depending on the directory reported during diagnosis, follow the suggestions provided. You might follow some or all of the suggestions, depending on your needs.
 
  1. Back up the current file. If any issues are encountered after attempting the following steps, you will need to send this backup to QRadar Support.
    Note: The following example uses /var/log/qradar.log as the file taking all the capacity. Change the file to suit your needs.
      
    mkdir -pv /store/IBM_Support/
cp -fv /var/log/qradar.log /store/IBM_Support/
  2. Truncate the file to release the space.
      
    ​truncate -s0 /var/log/qradar.log
  3. Compress any decompressed file preventing log rotation.
      
    gzip /var/log/qradar.old/<file name>
    Output example: 
    gzip /var/log/qradar.old/qradar.log.1
  4. Verify no decompressed file exists. 
    ls /var/log/qradar.old/ | grep -vE "gz$"
ls /var/log/audit/ | grep -vE "audit.log.*.gz$"
    For /var/log/qradar.old, no output is generated. For /var/log/audit, the output must look similar to the following:
      
    auditd.log
auditd.log.1
auditd.log.2
auditd.log.3
auditd.log.4
audit.log
  5. Restart the syslog-ng service.
    Note: Restarting this service causes a small interruption in the writing of the system logs like /var/log/messages. If required, administrators must schedule a suitable maintenance window to do perform this restart.
      
    systemctl restart syslog-ng
  6. Verify the partition is now under normal values.
      
    df -Th /var/log /var/log/audit
  7. Run again the steps in the Diagnosing the Problem section to ensure no errors are reported.

    Result
    The /var/log and /var/log/audit partition no longer has disk space constraints. If the affected log was inside /var/log/httpd, administrators must restart the Tomcat service to release the space. 
    systemctl restart tomcat
    If the partition does not decrease its usage, the functionality is not restored or the problem reoccurs often, contact QRadar Support for assistance and include the backup file.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
13 June 2024

UID

ibm10719969

Page Feedback