IBM Support

QRadar: /store/tmp partition can reach usage limit due to large vulnerability scans

Troubleshooting


Problem

Large vulnerability scan imports can cause /store/tmp partition to reach usage limits, which in turn can lead to services shutting down.

Symptom

By default, you receive warnings about your /store/tmp partition usage when it nears or reaches its capacity. Also, when any of the disk partitions on your QRadar appliance reach the capacity, services are shut down. Scheduled scans that trigger this issue do not complete.

Cause

When QRadar is running scans or importing scan results from a third-party scanner, the /store/tmp/vis directory is used as the default location for scanner files to be written. However, by default the /store/tmp partition has approximately 10 GB of space. When the scans cover wide ranges, resulting scanner files can cause this partition to fill.

Diagnosing The Problem

Follow the steps to verify the disk space that is used by a scan:
  1. Using SSH, log in to the QRadar Console as the root user.
  2. The following command can be used to confirm the current disk usage for /store/tmp partition: 
    df -h
    The output of this command lists your disk partitions and their usage levels. If you are experiencing this issue, the usage of the /store/tmp partition is above 85% as shown in the following example: 
    [root@qradar ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda7 20G 16G 2.7G 86% /
.
.
.
/dev/sda6 9.7G 9.2G 500M 99% /store/tmp
.
.
.

    Note: If your scanner is being managed by a host other than your Console, the above command as well as the rest of the procedure will need to be run after connecting to that host from your console using SSH. If you are unsure about which managed host is experiencing this issue, you can use the all_servers.sh command to verify the disk usage on all your appliances as discussed in Technote 1978283: QRadar: Using the all_servers.sh command.
  3. To verify the size of scan import directory itself, run the following command: 
    du -sh /store/tmp/vis
    You can use the following command to view the size and the last modification timestamp of the individual files: 
    ls -ltrh /store/tmp/vis

Resolving The Problem

Once you verified that you are experiencing this issue, you can clear disk space and identify problem scans by using the instructions in the following sections.

Clear disk space
 
The first step is to clear disk space in the /store/tmp partition so that the system can return to it normal operating state.
 
  1. Stop all active scans. To do this, verify that you do not have any active scans in the UI under Admin > Vulnerability > Schedule VA Scanners.
  2. Free up disk space in the /store/tmp partition by removing the temporary scanner files. You can accomplish this by running the following command on the target managed host: 
    rm -r /store/tmp/vis/*
  3. Verify that the disk space is now free by rerunning the commands from steps 2 and 3 in the Diagnosing the problem section.

    To view partition usage: 
    df -h
    To view scanner files: 
    du -sh /store/tmp/vis
Identifying problem scans

With the system now returned to a functional state, the next step is to identify which scanners and scans are causing the problem so that future recurrences can be avoided. You can do this by reviewing your scheduled scans from the UI under Admin > Vulnerability > Schedule VA Scanners.

Any scans that cover wide CIDR ranges (such as 0.0.0.0/0 or 10.0.0.0/8) or that are not completing successfully are the likely causes of this issue. If you are unsure about which scanner is causing the problem, you can run the suspected scan or scans and monitor the resulting files that are created as discussed in the Diagnosing the problem section.

Keeping partition usage within limits

Once you identify which scan or scans are causing the problem, use one or more of the following methods to ensure that /store/tmp partition usage remains within the limits. The first two options discuss possible configuration changes to reduce disk usage, while the third option discusses ways to use a partition other than /store/tmp for temporary scanner files.
 
  • Reducing scan size
    Using a more manageable scan size results in smaller scanner files being used. This is accomplished by breaking up large CIDR ranges into smaller, more manageable ranges.
     
  • Reducing the Cache Timeout
    The UI panel for some VA Scanner configurations, such as the Rapid7 Nexpose Scanner configuration UI, includes a configurable parameter called Cache Timeout. This property specifies the length of time in minutes that scan report data is stored in the cache. For large scans, you can try lowering this value to ensure that large files do not remain your system long enough to cause issues. The value for Cache Timeout can also be set to 0, which causes temporary scan report files to be immediately deleted once they are processed.
  • Using a different partition

    It is possible to change where the temporary scanner files are written on the system. Some scanners, such as Rapid7 Nexpose and eEye REM (SNMP), allow administrators to configure a scan directory option in the user interface. It is also possible to force all temporary scanner files, regardless of source, to be written to a different partition by using a symbolic link. The latter approach allows you to prevent future problems without making any configuration changes in the UI. While both approaches are valid, the use of a symbolic link resolves the issue for all scanners configured to use the particular managed host without any changes to your existing configuration.

    Note: You can use option 'a' or option 'b'. It is not recommended to use both. If you implement both, scanners with configuration changes use the custom directory that is specified in their configuration, while the rest of the system use the directory defined in the implementation of option 'b'. Scanners that use their own custom directory are not monitored by QRadar Disk Maintenance.
    1. Editing a scanner to use a custom scan directory
      Some scanners allow you to configure a custom directory for temporary files. Follow the configuration instructions for the scanner to accomplish this task. Note: using this option is not preferred as the QRadar Disk Maintenance is not guaranteed to monitor the custom directories that your scanners might be configured to use. If the directories are not monitored, their temporary files will not be cleaned after they are no longer needed.
       
    2. Using symbolic links to utilize a custom directory for scan files
      By creating a symbolic link, you can utilize a different disk partition to store temporary scan files. The effect of this option differs from option 'a' in two significant ways: First, it impacts all scanners that are configured to use the particular managed host unless another custom directory is also defined when using option 'a'. Second, the custom directory that is used is monitored by QRadar Disk Maintenance, which reclaims unused disk space.

      The process to implement a symbolic link is as follows:
      1. On your QRadar appliance, locate a partition that has sufficient disk space and create a custom directory there. For example, if you wanted to create the custom directory in the /store partition, you would run this command from the root command line:
        mkdir /store/largescans
      2. Configure the permissions of this new directory so that it has global read and write permissions. This allows the vis user to have access to it. Continuing with the same example, the command would be: 
        chmod 777 /store/largescans
      3. Create a symbolic link from the /store/tmp/vis directory to the target directory by using the following command: 
        ln -s <target> <source>
        In our example, the resulting command would be: 
        ln -s /store/largescans /store/tmp/vis
      4. Rerun your scans to verify that the scan files are now being created in the custom directory.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtiAAA","label":"Performance"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.5.0"}]

Document Information

Modified date:
29 June 2023

UID

swg21988472

Page Feedback