Configuring Data Explorer search result retention

IBM® Security Data Explorer search can handle large volumes of data. The default retention period for search results can be adjusted and must be considered when you are planning the size of your storage.

Before you begin

You must be an administrator for the IBM Security QRadar® Suite Software cluster on Red Hat® OpenShift® Container Platform.

  • The Red Hat OpenShift CLI client helps you develop, build, deploy, and run your applications on any Red Hat OpenShift or Kubernetes cluster. It also includes the administrative commands for managing a cluster under the adm subcommand.

    1. Download Red Hat OpenShift CLI 4.16 or 4.18 from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-4.16/. The file to download is called openshift-client-<platform>-<version>.tar.gz.
    2. Extract the binary file that you downloaded by typing the following command, where <oc_cli_archive_file> is the name of the archive file that you downloaded.
      tar -xf <oc_cli_archive_file>
    3. Modify the permissions of the binary file by typing the following command, where <oc_cli_binary> is the name of the Red Hat OpenShift binary that you extracted from the archive.
      chmod 755 <oc_cli_binary>
    4. Move the binary file to the /usr/local/bin directory by typing the following command.
      mv <oc_cli_binary> /usr/local/bin/oc
      Tip: If this command returns a No such file or directory or Not a directory error message, create the /usr/local/bin directory by typing the following command.
      sudo mkdir /usr/local/bin
    5. Ensure that the Red Hat OpenShift CLI client is working by typing the following command.
      oc version
      Tip: MacOS users might see a message that this tool cannot be opened because it is from an unidentified developer. Close this message and go to System Preferences > Security & Privacy. On the General tab, click Open Anyway or Allow Anyway. Repeat the oc version command.

About this task

Data Explorer search results retention can drive significant volumes of data. The default search retention period is 3 days. After this period, all search results are deleted.

Tip: Queries are retained so that you can easily re-create results.

QRadar Suite Software provides for a notional allocation of 0.5 terabytes for Data Explorer search results retention. This allocation is contained within the storage for Postgres, which includes up to 1 TB of data. For more information about recommended storage, see Storage requirements.

The following table shows examples of estimated search size and results retention capacity.

Table 1. Estimating search size and retention capacity
Query row size PVC size (MiB) Query schema size Number of searches (results capacity)
1k rows 6.9 6488 kB 25328
10k rows 63.8 63 MB 2739
60k 378.8 376 MB 461
100k 614.4 627 MB 284

For example, if your searches in Data Explorer result in 60k of data on average, the system can store approximately 461 search results. So, if you plan to run more than 461 of these searches over the course of 3 days, increase the storage or follow the procedure to lower retention time. For more information about resizing Postgres PVCs, see related information.

Procedure

  1. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
    • Using a username and password.
      oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>
    • Using a token.
      oc login --token=<token> --server=<openshift_url>
  2. Ensure that you have the serviceability pod running by typing the following command. 
    oc get pod -lrun=cp-serviceability -n <cp4s_namespace>
  3. Exec into the serviceability pod by typing the following command. 
    oc exec -ti deploy/cp-serviceability -- bash
  4. Type the following command. 
    /opt/bin/modify_deployment -n <cp4s_namespace> -a udiendpoints -p default_redis_expiry_in_days:<number_of_days>

Results

To incorporate any changes, updates to the configmap file trigger an automatic restart of the IBM Security QRadar Suite Software services.