IBM Support

QRadar: Deploy fails with error "Deployment is blocked due to critical disk space issue"

Troubleshooting


Problem

In the QRadar SIEM Admin user interface, a Deploy Changes fails to complete with the following error message: "Error performing deployment. See logs for details." A common reason for this general error message is that a service is disabled or unresponsive due to a disk space issue on the Console or All-in-One appliance.

Diagnosing The Problem

A Deploy Changes can fail to start for a number of reasons, but one of the most common issues seen in support cases is insufficient disk space. If an administrator has issues attempting to deploy changes, review the QRadar error log on the QRadar All-In-One or Console appliance in the /var/log/qradar.error directory.
Administrators can review tomcat service, specifically looking for scheduleDeployment error messages similar to the following error messages in the /var/log/qradar.error log:
Jan 1 09:00:00 ::ffff:IP-ADDRESS [tomcat.tomcat] [admin@IP-ADDRESS (4775) /console/JSON-RPC/QRadar.scheduleDeployment QRadar.scheduleDeployment] com.q1labs.configservices.util.ConfigServicesUtil: [INFO] [NOT:0000006000][IP-ADDRESS/- -] [-/--] Deployment is blocked due to critical disk space issue

May 19 07:46:53 ::ffff:IP-ADDRESS [tomcat.tomcat] [configservices@127.0.0.1 (422) /console/restapi/api/config/deploy_action] com.q1labs.configservices.util.ConfigServicesUtil: [INFO] [NOT:0000006000][IP-ADDRESS/- -] [-/- -]Deployment is blocked due to critical disk space issue

May 19 07:46:53 ::ffff:IP-ADDRESS [tomcat.tomcat] [configservices@127.0.0.1 (422) /console/restapi/api/config/deploy_action] com.ibm.si.configservices.api.v3_0.configuration.ConfigAPI: [ERROR] [NOT:0000003000][IP-ADDRESS/- -] [-/- -]Error scheduling deployment

Resolving The Problem

  1. Using SSH, log in to the QRadar Console's command-line interface (CLI) as the root user.
  2. To verify the amount of free disk space, type: df -Th
  3. Review the output to determine whether any file systems have a Use% greater than 95%.  In this example, the / file system is full:
    [root@hostname ~]# df -h
    Filesystem                        Size  Used Avail Use% Mounted on
    /dev/mapper/rootrhel-root          13G   13G   13G 100% /
    devtmpfs                           16G     0   16G   0% /dev
    tmpfs                              16G   20K   16G   1% /dev/shm
    tmpfs                              16G  325M   16G   3% /run
    tmpfs                              16G     0   16G   0% /sys/fs/cgroup
    /dev/mapper/storerhel-transient    16G   75M   16G   1% /transient
    /dev/mapper/storerhel-store        62G   31G   31G  50% /store
    /dev/mapper/rootrhel-storetmp      15G  306M   15G   2% /storetmp
    /dev/mapper/rootrhel-opt           13G  7.5G  7.5G  50% /opt
    /dev/mapper/rootrhel-tmp          3.0G  219M  2.8G   8% /tmp
    /dev/mapper/rootrhel-var          5.0G  311M  4.7G   7% /var
    /dev/mapper/rootrhel-varlog        15G  7.5G     0  50% /var/log
    /dev/mapper/rootrhel-varlogaudit  3.0G  237M  2.8G   8% /var/log/audit
    /dev/sda3                          32G  4.5G   28G  15% /recovery
    /dev/sda2                        1014M  302M  713M  30% /boot
    /dev/mapper/rootrhel-home        1014M   33M  982M   4% /home
    tmpfs                             3.2G     0  3.2G   0% /run/user/0
    
  4. Take actions to free disk space by moving old or unused files off the QRadar Appliance.  To investigate the cause of the / partition to filling up, see How to resolve disk space usage problems for / partition.
  5. Log in to the QRadar user interface.
  6. Click the Admin tab.
    Important: Deploy Changes might result in services being restarted. While services are restarting, event processing stops until services restart. Scheduled reports that are in-progress need to be manually restarted by users. Administrators with strict outage policies are advised to complete the next step during a scheduled maintenance window for their organization.
  7. Click Deploy Changes.

    Results
    The changes should deploy successfully. If you continue to have issues or need further guidance on Disk Space issues in QRadar, see the Disk Space 101 page, ask in our forums, or review the APARs 101 page for the term 'Disk Space'. The APARs 101 page can help indicate where fixes reside for reported issues in QRadar software.

[{"Business Unit":{"code":"BU008","label":"Security"},"Product":{"code":"SSBQAC","label":"IBM QRadar SIEM"},"Component":"Deploy Changes","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
07 January 2021

UID

ibm10794387