Backup and restore

To recover from any data loss that might occur, regularly back up the data in your IBM Security QRadar® Suite Software and integrated databases. You can use the backup and restore process to support a disaster recovery that requires a redeployment of your environment.

The following table shows where the backup files are stored in the backup and restore pod, and the backup file name convention that is used for each data store.
Table 1. Backup file name conventions
Component Flag Location Naming convention
CouchDB couch /opt/data/backup/couchdb
  • In 1.10.12 and later couch_backup_default_YYYY_MM_DD__HH_MM_SS.tz
  • In 1.10.11 and earlier default_data_YYYY_MM_DD__HH_MM_SS.gz and default_k8s_YYYY_MM_DD__HH_MM_SS.gz
Case Management cases /opt/data/backup/cases
  • In 1.10.12 and later cases_backup_YYYY_MM_DD__HH_MM_SS.gz.aes
  • In 1.10.11 and earlier cases_backup_YYYY_MM_DD__HH_MM_SS.gz
Postgres pg /opt/data/backup/pg
  • In 1.10.12 and later pg_backup_default_YYYYMMDD_HHMMSS.tz
  • In 1.10.11 and earlier backup-YYYYMMDD_HHMMSS.sql.tgz
Entitlements entitlements /opt/data/backup/entitlements
  • In 1.10.12 and later entitlements_backup_default_YYYY_MM_DD_HH_MM_SS.tz
  • In 1.10.11 and earlier entitlements_backup_YYYY_MM_DD_HH_MM_SS.gz
New in 1.10.12QRadar Suite Software configuration cp4sconfigurations /opt/data/backup/cp4sconfigurations cp4sconfigurations_YYYY_MM_DD_HH_MM_SS.tz

Backing up QRadar Suite Software

To back up your databases for IBM Security QRadar Suite Software, you must run the backup scripts from within the Backup and Restore pod.

Before you begin

You need cluster administrator level privileges to complete the backup and restore process.

New in 1.10.12 You need to supply an AES-128 GCM key, an AES-192 GCM key, or an AES-256 GCM key to encrypt your backups.

In 1.10.11 and earlier, you need to supply a password to encrypt your backups.

To install QRadar Suite Software, you configure a suitable storage class in the cluster. You support the configuration with one or more persistent volumes of suitable size. For more information about storage, see Persistent storage requirements.

You provide secure storage for the backups that is mounted as a Persistent Volume Claim (PVC) in a pod. The backup and restore process uses a Backup and Restore pod, which contains all of the necessary utilities that are required for the backup and restore process. The Backup and Restore pod is deployed automatically as part of the installation or upgrade of QRadar Suite Software.

For the backup data, you can opt to provision your own storage instead of using the default specified for installation. For more information, see Creating the backup and restore PVC.

  • The Red Hat OpenShift CLI client helps you develop, build, deploy, and run your applications on any Red Hat OpenShift or Kubernetes cluster. It also includes the administrative commands for managing a cluster under the adm subcommand.

    1. Download Red Hat OpenShift CLI 4.12 or later from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-4.12/. The file to download is called openshift-client-<platform>-<version>.tar.gz.
    2. Extract the binary file that you downloaded by typing the following command, where <oc_cli_archive_file> is the name of the archive file that you downloaded.
      tar -xf <oc_cli_archive_file>
    3. Modify the permissions of the binary file by typing the following command, where <oc_cli_binary> is the name of the Red Hat OpenShift binary that you extracted from the archive.
      chmod 755 <oc_cli_binary>
    4. Move the binary file to the /usr/local/bin directory by typing the following command.
      mv <oc_cli_binary> /usr/local/bin/oc
      Tip: If this command returns a No such file or directory or Not a directory error message, create the /usr/local/bin directory by typing the following command.
      sudo mkdir /usr/local/bin
    5. Ensure that the Red Hat OpenShift CLI client is working by typing the following command.
      oc version
      Tip: MacOS users might see a message that this tool cannot be opened because it is from an unidentified developer. Close this message and go to System Preferences > Security & Privacy. On the General tab, click Open Anyway or Allow Anyway. Repeat the oc version command.

About this task

The backup and restore process for QRadar Suite Software covers the main data stores within the system. The following table summarizes the main data stores for persistent storage.

Table 2. Persistent storage
Application Persistent storage
Platform Services (Profile, Entitlements, Connected Assets and Risk, IBM® Threat Hunting Language, Threat Intelligence Service, Dashboards, Universal Data Insights) CouchDB, PostgreSQL
IBM Security Data Explorer CouchDB
IBM Security Case Management and Orchestration & Automation PostgreSQL
IBM Security Risk Manager CouchDB, PostgreSQL
IBM Security Threat Investigator PostgreSQL
IBM Security Threat Intelligence Insights CouchDB
IBM Detection and Response Center PostgreSQL

The secrets that are associated with the databases are backed up as part of the backup process.

The following conditions apply to the backup and restore process.

LDAP configuration
The LDAP configuration is managed through IBM Cloud Pak® foundational services and is not part of the QRadar Suite Software backup and restore process. If the LDAP configuration is lost, you must re-create it before you start the QRadar Suite Software restore process. For more information, see Configuring LDAP authentication.
Data Explorer queries
Data Explorer query results, including the results of queries that are saved in case artifacts, are not retained through the backup and restore process. After a backup and restore process is completed, if you try to open a preexisting query in Data Explorer from the case that was created in relation to it, an error is displayed.

When the backups are run, the data is stored on the PVC. Backup data is restored from the PVC that is mounted in the Backup and Restore pod. After the restore script is completed, the data is restored and the QRadar Suite Software system returns to the state at the time of the backup. Following a complete uninstallation of QRadar Suite Software, if previous backups are no longer required, the backup and restore components can be removed.

To run the backup script, you must provide an encryption password that must later be supplied to restore data from the backup.

Procedure

  1. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
    • Using a username and password.
      oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>
    • Using a token.
      oc login --token=<token> --server=<openshift_url>
  2. Switch to your QRadar Suite Software namespace by typing the following command. 
    oc project <cp4s_namespace>

    For example, if your QRadar Suite Software namespace is cp4sexample, type the following command.

    oc project cp4sexample

    The following example output shows that you have switched to the cp4sexample namespace.

    Now using project "cp4sexample" on server "https://example.com:6443".
  3. Ensure that the Backup and Restore pod is running and retrieve the pod name by typing the following command. 
    oc get pods | grep cp4s-backup-restore
    Verify that the command completes and take note in the output of the full pod name. In the following example, the name of the pod is cp4s-backup-restore-8ffb54b4f-lfblg.
    cp4s-backup-restore-8ffb54b4f-lfblg                           1/1     Running                      0          8m53s
  4. Create an environment variable for your backup and restore pod name by typing the following command, where <backup_restore_podname> is the backup and restore pod name from the previous command. 
    export BACKUP_AND_RESTORE_POD=<backup_restore_podname>
  5. Back up QRadar Suite Software data stores by typing one of the following commands.
    • In 1.10.12 and later
      Table 3. Backup arguments
      Argument Description
      -a Your AES key. This parameter is required. For more information, see Generating an AES key.
      -k The number of backups to keep. If you don't specify this parameter, the 30 most recent backups are kept.
      -b A data store to back up. You can repeat this parameter as needed. If you don't specify this parameter, all data stores are backed up. The data stores are shown in Table 1.
      For example, the following command backs up all data stores with an AES key. 
      oc exec $BACKUP_AND_RESTORE_POD -- /opt/bin/backup-cp4s -a <AES_key>
      For example, the following command backs up CouchDB and Cases data only with an AES key. 
      oc exec $BACKUP_AND_RESTORE_POD -- /opt/bin/backup-cp4s -b couch -b cases -a <AES_key>
    • In 1.10.11 and earlier
      Table 4. Backup arguments
      Argument Description
      -couch Back up CouchDB data.
      -cases Back up IBM Security Case Management data.
      -pg Back up Postgres database.
      -entitlements Back up entitlements data.
      -keepfiles Enter the number of backup files to keep.
      -pruneonly Prune back up files without running the full backup.
      For example, the following command backs up all data stores with an encryption password.
      oc exec $BACKUP_AND_RESTORE_POD -- /opt/bin/backup-cp4s.sh -p <encryption_password>
      For example, the following command backs up the CouchDB and Cases data stores with an encryption password. 
      oc exec $BACKUP_AND_RESTORE_POD -- /opt/bin/backup-cp4s.sh -couch -cases -p <password>
  6. It is good practice to maintain a copy of the backup files in a secure alternative location outside of the QRadar Suite Software cluster. After you complete the backup procedure, the backup files can be copied to your local system or to another cluster.
    • To copy the whole /opt/data/backup folder to your local system, type the following command. To remove the folder from the Persistent Volume Claim (PVC) after you copy it to your local system, add the --remove-source-files flag to the command.
      rsync --rsh='oc rsh' -av -c --inplace --partial --append --progress $BACKUP_AND_RESTORE_POD:/opt/data/backup <local_path>

      The result is a copy of the complete backup directory structure on your local system, in your current directory, in a new subfolder called backup. This folder can then be transferred to an appropriate secure storage location.

    • To copy an individual file to your local system, type the following command. To remove the file from the PVC after you copy it to your local system, add the --remove-source-files flag to the command.
      rsync --rsh='oc rsh' -av -c --inplace --partial --append --progress $BACKUP_AND_RESTORE_POD:/opt/data/backup/<path_to>/<file_name> <local_path>/<file_name>

      The result is a copy of the file on your local system, in your current directory, in a new subfolder called backup. This file can then be transferred to an appropriate secure storage location.

    Tip: If the rsync command times out, run it again to continue copying files.

Scheduling QRadar Suite Software backup

IBM Security QRadar Suite Software provides a support action to schedule QRadar Suite Software backup.

Before you begin

To access the schedule_cp4s_full_backup action, you must install the command-line interface (CLI) utility cpctl from the cp-serviceability pod. For more information, see Installing the cpctl utility to access support actions.

About this task

The schedule_cp4s_full_backup action runs a Red Hat OpenShift Container Platform cron job that creates a backup regularly, according to your schedule.

The schedule_cp4s_full_backup action uses the following parameters.
Parameter Default Required Description
--password none No In 1.10.12 and later, an AES-128 GCM key, an AES-192 GCM key, or an AES-256 GCM key that is used to encrypt the backup files. This key is required during the restore process.

In 1.10.11 and earlier, a user-defined password that is used to encrypt the backup files. This password is required during the restore process. This password cannot be recovered if it is lost.
--generate aes-256-gcm No Generate an AES-128 GCM key, an AES-192 GCM key, or an AES-256 GCM key that is used to encrypt the backup files. This key is required during the restore process.

The generated key is stored in the cp4s-backup-cron-secret secret.

When you use --generate, --password is not required.
--token none Yes A token that the administrator generates by running oc whoami -t on the local system.
--schedule ("0 0 * * */6") No The cron job schedule that is used to set the schedule for the backup.
--airgap none No The local registry URL that was used to deploy IBM Security QRadar Suite Software.
--disable false No To disable (false) or enable (true) the scheduled backup.
--keepfiles 7 No To configure the backup file maintenance process. When the number of backup files that are maintained is greater than the value of this parameter, the oldest backup files are deleted.

To set up a backup schedule, disable a scheduled backup, or configure the number backup files that are maintained, see the following procedures.

Procedure

  1. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
    • Using a username and password.
      oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>
    • Using a token.
      oc login --token=<token> --server=<openshift_url>
  2. To set up the schedule, type the following command.
    • In an online environment 
      cpctl tools schedule_cp4s_full_backup --token "$(oc whoami -t)" --password <AES_key> --schedule <your_schedule>
    • In an air-gapped environment 
      cpctl tools schedule_cp4s_full_backup --token "$(oc whoami -t)" --password <AES_key> --schedule <your_schedule> --airgap <registry_domain_name>:5000
  3. It is good practice to maintain a copy of the backup files in a secure alternative location outside of the QRadar Suite Software cluster. After you complete the backup procedure, the backup files can be copied to your local system or to another cluster.
    • To copy the whole /opt/data/backup folder to your local system, type the following command. To remove the folder from the Persistent Volume Claim (PVC) after you copy it to your local system, add the --remove-source-files flag to the command.
      rsync --rsh='oc rsh' -av -c --inplace --partial --append --progress $BACKUP_AND_RESTORE_POD:/opt/data/backup <local_path>

      The result is a copy of the complete backup directory structure on your local system, in your current directory, in a new subfolder called backup. This folder can then be transferred to an appropriate secure storage location.

    • To copy an individual file to your local system, type the following command. To remove the file from the PVC after you copy it to your local system, add the --remove-source-files flag to the command.
      rsync --rsh='oc rsh' -av -c --inplace --partial --append --progress $BACKUP_AND_RESTORE_POD:/opt/data/backup/<path_to>/<file_name> <local_path>/<file_name>

      The result is a copy of the file on your local system, in your current directory, in a new subfolder called backup. This file can then be transferred to an appropriate secure storage location.

    Tip: If the rsync command times out, run it again to continue copying files.
  4. To check the current configured backups and their respective schedule by typing the following command, where $CP4S_NAMESPACE is the namespace where you are installing QRadar Suite Software. 
    oc get cp4sbackupschedulers.isc.ibm.com cp4s-backup-scheduler -ojsonpath='{.spec.backups}' -n "$CP4S_NAMESPACE"
  5. To check the health of the configured scheduled backups, type the following command: 
    oc get cp4sbackupschedulers.isc.ibm.com cp4s-backup-scheduler -ojsonpath='{.status.conditions[0]}' -n "$CP4S_NAMESPACE"
  6. If you need to disable the scheduled backup, type the following command. 
    cpctl tools schedule_cp4s_full_backup --token "$(oc whoami -t)" --password <AES_key> --disable

Restoring QRadar Suite Software

When the restore process is completed, data is restored and the system returns to the state at the time of the backup. The corresponding databases are restored in the appropriate persistent volume of the IBM Security QRadar Suite Software instance.

Before you begin

  • The Red Hat OpenShift CLI client helps you develop, build, deploy, and run your applications on any Red Hat OpenShift or Kubernetes cluster. It also includes the administrative commands for managing a cluster under the adm subcommand.

    1. Download Red Hat OpenShift CLI 4.12 or later from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-4.12/. The file to download is called openshift-client-<platform>-<version>.tar.gz.
    2. Extract the binary file that you downloaded by typing the following command, where <oc_cli_archive_file> is the name of the archive file that you downloaded.
      tar -xf <oc_cli_archive_file>
    3. Modify the permissions of the binary file by typing the following command, where <oc_cli_binary> is the name of the Red Hat OpenShift binary that you extracted from the archive.
      chmod 755 <oc_cli_binary>
    4. Move the binary file to the /usr/local/bin directory by typing the following command.
      mv <oc_cli_binary> /usr/local/bin/oc
      Tip: If this command returns a No such file or directory or Not a directory error message, create the /usr/local/bin directory by typing the following command.
      sudo mkdir /usr/local/bin
    5. Ensure that the Red Hat OpenShift CLI client is working by typing the following command.
      oc version
      Tip: MacOS users might see a message that this tool cannot be opened because it is from an unidentified developer. Close this message and go to System Preferences > Security & Privacy. On the General tab, click Open Anyway or Allow Anyway. Repeat the oc version command.

Procedure

  1. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
    • Using a username and password.
      oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>
    • Using a token.
      oc login --token=<token> --server=<openshift_url>
  2. Switch to your QRadar Suite Software namespace by typing the following command. 
    oc project <cp4s_namespace>

    For example, if your QRadar Suite Software namespace is cp4sexample, type the following command.

    oc project cp4sexample

    The following example output shows that you have switched to the cp4sexample namespace.

    Now using project "cp4sexample" on server "https://example.com:6443".
  3. Ensure that the Backup and Restore pod is running and retrieve the pod name by typing the following command. 
    oc get pods | grep cp4s-backup-restore
    Verify that the command completes and take note in the output of the full pod name. In the following example, the name of the pod is cp4s-backup-restore-8ffb54b4f-lfblg.
    cp4s-backup-restore-8ffb54b4f-lfblg                           1/1     Running                      0          8m53s
  4. Create an environment variable for your backup and restore pod name by typing the following command, where <backup_and_restore_pod> is the backup and restore pod name from the previous command. 
    export BACKUP_AND_RESTORE_POD=<backup_and_restore_pod>
  5. If you are restoring backup data that is stored outside of the cluster, copy the backup data into the toolbox PVC from your local system by typing the following command in the QRadar Suite Software namespace. 
    rsync --rsh='oc rsh' -av -c --inplace --partial --append --progress <local_path> $BACKUP_AND_RESTORE_POD:/opt/data/backup

    <local_path> is the backup folder name and full path location on your local system.

    Tip: If the rsync command times out, run it again to continue copying files.
  6. Restore QRadar Suite Software data stores by typing one of the following commands.
    • In 1.10.12 and later
      Important: The encryption key that is used in the restore command must be the same as the one used to back up the data.
      Table 5. Restore arguments
      Argument Description
      -a The AES key that was used to encrypt the backup. This parameter is required to restore a backup file encrypted with an AES key.
      -b A data store to restore. You can repeat this parameter as needed. If you don't specify this parameter, all data stores are restored. The data stores are shown in Table 1.
      -f A specific backup file to restore. You can repeat this parameter as needed. If you don't specify this parameter, the most recent backup file is restored.
      For example, the following command restores all backup files that are encrypted with an AES key. 
      oc exec $BACKUP_AND_RESTORE_POD -- /opt/bin/restore-cp4s -a <AES_key>
      For example, the following command restores the most recent CouchDB backup file that is encrypted with an AES key.
      oc exec $BACKUP_AND_RESTORE_POD -- /opt/bin/restore-cp4s -b couch -a <AES_key>
      For example, the following command restores the most recent CouchDB backup file and a specific Cases backup file that are encrypted with an AES key.
      oc exec $BACKUP_AND_RESTORE_POD -- /opt/bin/restore-cp4s -a <AES_key> -b couch -f /opt/data/backup/cases/cases_backup_2020_10_22__10_38_17.gz
    • In 1.10.11 and earlier
      Important: The encryption password that is used in the restore command must be the same as the one used to back up the data.
      oc exec $BACKUP_AND_RESTORE_POD -- /opt/bin/restore-cp4s.sh -p <encryption_password>
      When the command is run without any scoping arguments, by default it completes a full restore of all of the data stores. The following arguments can be used in the command to restore individual components, or combinations of components.
      Table 6. Restore arguments
      Argument Description
      -couch Restore CouchDB data.
      -cases Restore IBM Security Case Management and IBM Security SOAR data.
      -pg Restore Postgres data.
      -entitlements Restore entitlements data.
      By default, the restore operation automatically restores the most recent backup file. To override the default and provide the name of a specific file to restore, use the following arguments.
      Table 7. Restore file name arguments
      Argument
      -couch-data-file <file_name>, -couch-k8s-file <file_name>
      -cases-file <file_name>
      -pg-file <file_name>
      -entitlements-file <file_name>

Results

After the restore is complete, allow up to 15 minutes for the pods to complete the restart operation. If the first attempt at restoring the system is not successful for any reason, the full restore procedure can be run again without any impact.

What to do next

  1. If users can't log in after you restore QRadar Suite Software from a backup, resynchronize LDAP directories. For more information, see Account, user, and entitlements troubleshooting.
  2. If you see a message after you restore indicating that the Threat Investigator automatic investigation user is no longer valid, see Automatic investigation user no longer valid.
  3. If you are using Threat Investigator Advanced, or any Threat Intelligence Insights external data sources, reconfigure them. For more information, see Enabling Threat Investigator and Configuring Threat Intelligence Insights external data sources.

Removing the backup and restore PVC

Following a complete uninstallation of IBM Security QRadar Suite Software, if previous backups are no longer required, the backup and restore components can be removed from the cluster by running the following command.

Before you begin

  • The Red Hat OpenShift CLI client helps you develop, build, deploy, and run your applications on any Red Hat OpenShift or Kubernetes cluster. It also includes the administrative commands for managing a cluster under the adm subcommand.

    1. Download Red Hat OpenShift CLI 4.12 or later from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-4.12/. The file to download is called openshift-client-<platform>-<version>.tar.gz.
    2. Extract the binary file that you downloaded by typing the following command, where <oc_cli_archive_file> is the name of the archive file that you downloaded.
      tar -xf <oc_cli_archive_file>
    3. Modify the permissions of the binary file by typing the following command, where <oc_cli_binary> is the name of the Red Hat OpenShift binary that you extracted from the archive.
      chmod 755 <oc_cli_binary>
    4. Move the binary file to the /usr/local/bin directory by typing the following command.
      mv <oc_cli_binary> /usr/local/bin/oc
      Tip: If this command returns a No such file or directory or Not a directory error message, create the /usr/local/bin directory by typing the following command.
      sudo mkdir /usr/local/bin
    5. Ensure that the Red Hat OpenShift CLI client is working by typing the following command.
      oc version
      Tip: MacOS users might see a message that this tool cannot be opened because it is from an unidentified developer. Close this message and go to System Preferences > Security & Privacy. On the General tab, click Open Anyway or Allow Anyway. Repeat the oc version command.

About this task

The command covers both deployment options for the Persistent Volume Claim (PVC) deployment, whether the PVC was deployed manually or automatically during installation.

Procedure

  1. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
    • Using a username and password.
      oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>
    • Using a token.
      oc login --token=<token> --server=<openshift_url>
  2. Switch to your QRadar Suite Software namespace by typing the following command. 
    oc project <cp4s_namespace>

    For example, if your QRadar Suite Software namespace is cp4sexample, type the following command.

    oc project cp4sexample

    The following example output shows that you have switched to the cp4sexample namespace.

    Now using project "cp4sexample" on server "https://example.com:6443".
  3. Delete the PVC by typing the following command. 
    oc delete pvc cp4s-backup-pv-claim
    Important: This command removes any backup data in the PVC. It is not possible to restore QRadar Suite Software data from backup after this command is run.

Configuring the number of backup files to keep

You can configure the number of backup files to keep on the cluster. When a backup is completed, any excess files over the configured number of files to keep are deleted.

Procedure

  1. Set the $KEEPFILES and $NAMESPACE variables by typing the following command. 
    export KEEPFILES=<number of files to keep>
export NAMESPACE=<CP4S_Namespace>
  2. Apply the CP4SConfiguration CR by typing the following command. 
    cat << EOF | oc apply -f - 
apiVersion: isc.ibm.com/v1
kind: CP4SConfiguration
metadata:
  name: backup-keep-files
  namespace: $NAMESPACE
spec:
  values:
  - backup.keepFiles=$KEEPFILES
EOF