Upgrading QRadar Suite Software by using CASE

The Red Hat OpenShift CLI client helps you develop, build, deploy and run your applications on any Red Hat OpenShift or Kubernetes cluster. It also includes the administrative commands for managing a cluster under the adm subcommand.

1. Download Red Hat OpenShift CLI 4.16 or 4.18 from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-4.16/ or https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-4.18/ . The file to download is called openshift-client-<platform>-<version>.tar.gz.
2. Extract the binary file that you downloaded by typing the following command, where <oc_cli_archive_file> is the name of the archive file that you downloaded.

   tar -xf <oc_cli_archive_file>

3. Modify the permissions of the binary file by typing the following command, where <oc_cli_binary> is the name of the Red Hat OpenShift binary that you extracted from the archive.
4. Move the binary file to the /usr/local/bin directory by typing the following command.

   mv <oc_cli_binary> /usr/local/bin/oc

   Tip: If this command returns a No such file or directory or Not a directory error message, create the /usr/local/bin directory by typing the following command.

   sudo mkdir /usr/local/bin

5. Make sure that the Red Hat OpenShift CLI client is working by typing the following command.

   oc version

   Tip: MacOS users might see a message that this tool cannot be opened because it is from an unidentified developer. Close this message and go to System Preferences > Security & Privacy. On the General tab, click Open Anyway or Allow Anyway. Repeat the oc version command.

Before you prepare for an upgrade, you must schedule a maintenance window to avoid disruptions, and review the documentation for any new features or known issues that might impact the upgrade process. For more information, see What's new or changed and Known issues.

1. Back up your QRadar Suite Software data.
2. Verify that you have the latest serviceability pod in the QRadar Suite Software namespace by using the following command.

   oc rollout restart deployment cp-serviceability -n $QRS_NAMESPACE
   oc rollout status deployment cp-serviceability  -n $QRS_NAMESPACE

3. When you upgrade to QRadar Suite Software 1.11, data from the Stolon PostgreSQL 12 de-udi-postgres cluster is migrated to the EnterpriseDB PostgreSQL 14 default-postgres cluster. If you previously customized your Stolon PostgreSQL 12 storage sizes, these changes must also be applied to the EDB PostgreSQL 14 storage sizes.
   1. To check whether the Stolon PostgreSQL 12 storage is modified, run the following command and compare the storage settings to the default sizes.

      oc get pvc | grep postgres

      The default storage sizes for Stolon PostgreSQL 12 are:

      • de-udi-postgres: 500 Gi
      • default-postgres: 220 Gi
      • soar-postgres: 220 Gi
   2. If the Stolon PostgreSQL storage sizes are customized, you can modify the EDB PostgreSQL 14 storage sizes by using one of the following methods.
      • Resizing the default EDB PVCs
      • Resizing the SOAR EDB PVCs

      There is no existing EDB PostgreSQL PVC currently and the default storage sizes for EDB PostgreSQL are:

      • default-postgres: 630 Gi
      • soar-postgres: 220 Gi
   Important:

   • The next step stops the Detection and Response Center application. QRadar Suite Software admins must not create new accounts until the upgrade process is complete.
   • The estimate time for data replication per GiB is 2 minutes.
   • Except for Connected Assets and Risk and Detection and Response Center applications, the QRadar Suite Software system remains available while the database replication runs online and new data that is written to the version 12 database replicates to version 14.
4. Install PostgreSQL version 14, and then start the data replication process from PostgreSQL version 12 to version 14 by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade start-replication --token=$(oc whoami -t) 

   The following example shows the output after you run the command.

   INFO: Checking storage configuration in cp4s-values and current DB cluster utilization, note that data from stolon instance de-udi will be replicated to edb instance default
   INFO: stolon default - storage setting: 220Gi, current database size: 127.5 Mi
   INFO: stolon de-udi - storage setting: 500Gi, current database size: 47.8 Mi
   INFO: stolon soar - storage setting: 220Gi, current database size: 98.5 Mi
   INFO: edb default - storage setting: 630Gi
   INFO: edb soar - storage setting: 220Gi
   INFO: installing edb postgres
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: edb clusters installed
   INFO: creating edb database soar/activemq
   INFO: creating edb database de-udi/atkhunts
   INFO: creating edb database de-udi/stixdata
   INFO: creating edb database default/apps
   INFO: creating edb database default/car
   INFO: creating edb database soar/co3
   INFO: creating edb database default/configstore
   INFO: creating edb database default/drc
   INFO: creating edb database default/edgegateway
   INFO: creating edb database default/cp4s_entitlements
   INFO: creating edb database default/cp4s_changelog
   INFO: creating edb database default/pulsebackend
   INFO: creating edb database default/advisor
   INFO: creating edb database default/cp4s_tis
   INFO: stopping drc services
   INFO: configuring replication publications and subscriptions
   INFO: replication configuration complete

5. Monitor the status of data replication by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade check-replication --token=$(oc whoami -t) 

   The following example output displays the replication status of each database. Before you proceed to the next step, make sure that the copy status is Complete and the streaming lag is less than 10 Mib.

   Instance  Database           Replication Status  Tables Copied  Copy Status  Streaming Lag  Streaming Status  WAL Held
   de-udi    atkhunts           Configured          5/5            Complete     0 bytes        active            56 bytes
   de-udi    stixdata           Configured          3/3            Complete     0 bytes        active            56 bytes
   default   advisor            Configured          169/169        Complete     0 bytes        active            60.5 Ki
   default   apps               Configured          22/22          Complete     0 bytes        active            60.5 Ki
   default   car                Configured          360/360        Complete     0 bytes        active            60.5 Ki
   default   car1               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car2               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car3               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car4               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car5               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car6               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car7               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car8               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   configstore        Configured          1/1            Complete     0 bytes        active            60.5 Ki
   default   cp4s_changelog     Configured          4/4            Complete     0 bytes        active            60.5 Ki
   default   cp4s_entitlements  Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   cp4s_tis           Configured          11/11          Complete     0 bytes        active            60.5 Ki
   default   drc                Configured          1381/1381      Complete     0 bytes        active            60.5 Ki
   default   edgegateway        Configured          3/3            Complete     0 bytes        active            60.5 Ki
   default   pulsebackend       Configured          0/0            Complete     0 bytes        active            60.5 Ki
   default   tii_threats        Configured          1/1            Complete     0 bytes        active            60.5 Ki
   soar      activemq           Configured          4/4            Complete     0 bytes        active            56 bytes
   soar      co3                Configured          565/565        Complete     0 bytes        active            56 bytes

   If the copy status of a database does not decrement or if the streaming lag is inactive, you can restart the replication for the database by using the following command.

   oc exec -it deploy/cp-serviceability -- postgres_upgrade reset-replication -t $(oc whoami -t) -d <database>

   For example, to restart the data replication for the configstore database, use the following command.

   oc exec -it deploy/cp-serviceability -- postgres_upgrade reset-replication -t $(oc whoami -t) -d configstore

6. Schedule a maintenance window for the upgrade.
   Important: All applications stop in the next step.
7. Complete the data replication process by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade complete-replication --token=$(oc whoami -t) 

   Tip:

   By default, the streaming lag must be less than 10 MiB to run the complete-replication command. On busy systems, replication streaming might not keep up with the live transactions. In this case, you can over ride the default setting by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade complete-replication --token=$(oc whoami -t) -s <MiB>

   To allow a 1024 MiB steaming lag, you can use the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade complete-replication --token=$(oc whoami -t) -s 1024

   If you increase the streaming lag, the replication process takes longer to complete.

   The following example shows the output after you run the command.

   INFO: replication lag within threshold, stopping applications
   INFO: deleting deployment isc-cases-activemq
   INFO: deleting deployment isc-cases-co3postgres-operator
   INFO: deleting deployment isc-cases-operator
   INFO: deleting deployment debackend
   INFO: deleting deployment ibm-aitk-orchestrator
   INFO: deleting deployment isc-aitk-thaddeus
   INFO: deleting deployment tis-udiworkers
   INFO: deleting deployment udi-udiedgeservice
   INFO: deleting deployment udi-udiendpoints
   INFO: deleting deployment udi-udiworkers
   INFO: deleting deployment isc-app-manager-ds
   INFO: deleting deployment car
   INFO: deleting deployment idrmingestion
   INFO: deleting deployment isc-cases-application
   INFO: deployment isc-cases-co3postgres-operator already deleted
   INFO: deleting deployment isc-cases-event-handler
   INFO: deployment isc-cases-operator already deleted
   INFO: deleting deployment inf-configstore
   INFO: deleting deployment drc
   INFO: deleting deployment drcapi
   INFO: deleting deployment drcimporter
   INFO: deployment drc already deleted
   INFO: deployment drcapi already deleted
   INFO: deployment drcimporter already deleted
   INFO: deployment drc already deleted
   INFO: deployment drcapi already deleted
   INFO: deployment drcimporter already deleted
   INFO: deleting deployment edgegateway-ui
   INFO: deleting deployment isc-entitlements
   INFO: deleting deployment pulsedashboard
   INFO: deleting deployment idrmdashboard
   INFO: deployment idrmingestion already deleted
   INFO: deleting deployment idrmintex
   INFO: deleting deployment idrmriskengine
   INFO: deleting deployment threat-inv-api
   INFO: deleting deployment tis-data-gateway
   INFO: databases are in sync
   INFO: removing replication subscriptions and publications
   INFO: syncing postgres sequences
   INFO: updating connection details in postgres secrets

8. Delete the ibm-cp-security-operator CSV by using the following command.

   OPERATOR_NS=<cp4s namespace or openshift-operators>
   oc get csv -n $OPERATOR_NS -o name | grep ibm-cp-security-operator | xargs oc delete -n $OPERATOR_NS

oc get cp4sthreatmanagement.isc.ibm.com/threatmgmt -o jsonpath='{.spec.basicDeploymentConfiguration.adminUser}' -n <cp4s_namespace>

oc get cp4sthreatmanagement.isc.ibm.com/threatmgmt -o jsonpath='{.spec.basicDeploymentConfiguration.domain}' -n <cp4s_namespace>

oc get cp4sthreatmanagement.isc.ibm.com/threatmgmt -o jsonpath='{.spec.basicDeploymentConfiguration.storageClass}' -n <cp4s_namespace>

1. Create the following environment variables with the installer image name and the image inventory on your mirroring device.

   export CASE_NAME=ibm-cp-security && export CASE_VERSION=1.1.3

2. Download the IBM Security QRadar Suite Software installer and image inventory to your mirroring device by typing the following command.

   oc ibm-pak get $CASE_NAME --version $CASE_VERSION --disable-top-level-images-mode

   The CASE is saved to the ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION directory and the log file is saved to ~/.ibm-pak/logs/oc-ibm_pak.log.

   Tip: If you want to save the CASE to a directory other than your home directory, set the $IBMPAK_HOME environment variable by typing the following command.

   export IBMPAK_HOME=<working_directory>

   When you set the $IBMPAK_HOME environment variable, the CASE is saved to <working_directory>/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION and the log is saved to <working_directory>/.ibm-pak/logs/oc-ibm_pak.log.

   Important: If you change where the CASE is saved to, you must use $IBMPAK_HOME/.ibm-pak in place of ~/.ibm-pak throughout this procedure.
3. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
   • Using a username and password.

     oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>

   • Using a token.

     oc login --token=<token> --server=<openshift_url>

4. Extract the QRadar Suite Software CASE by typing the following command.

   tar -xf \
   ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION/ibm-cp-security-$CASE_VERSION.tgz \
   -C ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION

5. Update the parameters in the ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION/ibm-cp-security/inventory/ibmSecurityOperatorSetup/files/values.conf file. The following table lists the configurable parameters for the QRadar Suite Software installation and their descriptions.

   Table 2. QRadar Suite Software installation parameters

   Parameter	Description	Do you need to update this parameter?
   adminUser	The user that is to be given administrator privileges in the QRadar Suite Software System Administration account after installation. Specify a username or an email address that exists in your identity provider.	Yes
   airgapInstall	Set to false.	No
   clusterProxy	Set to true if you are installing in a cluster that is using a cluster-wide proxy. Set to false if you are not installing in a cluster that is using a cluster-wide proxy. Cluster-wide proxy is not supported in a disconnected environment.	No, unless you are installing in a cluster that is using a cluster-wide proxy.
   domain	The fully qualified domain name (FQDN) created for QRadar Suite Software. If you don't specify an FQDN, it is generated as cp4s.<cluster_ingress_subdomain>. For more information about the FQDN requirements, see Domain name and TLS certificates.	No, unless you want to specify your own FQDN.
   domainCertificatePath	The path of the TLS certificate that is associated with the QRadar Suite Software domain. If the domain is not specified, the Red Hat OpenShift cluster certificates are used. For more information, see Domain name and TLS certificates.	No, unless you updated the domain parameter.
   domainCertificateKeyPath	The path of the TLS key that is associated with the QRadar Suite Software domain. If the domain is not specified, the Red Hat OpenShift cluster certificates are used. For more information, see Domain name and TLS certificates.	No, unless if you updated the domain parameter.
   customCaFilePath	The path of the custom TLS certificate associated with the QRadar Suite Software domain. For more information, see Domain name and TLS certificates.	No, unless you are using a custom or self-signed certificate.
   storageClass	The provisioned block or file storage class for all the PVCs required by QRadar Suite Software. When it is not specified, the default storage class for the cluster is used. For more information, see Storage requirements.	No, unless you are using a storage class other than the default storage class for the cluster.
   backupStorageClass	Storage class for the backup and restore pod. If this value is not set, QRadar Suite Software takes the value from the storageClass parameter.	No, unless you are using a different storage class for the backup and restore pod than you set for the storageClass parameter.
   backupStorageSize	The storage size for the backup and restore PVC. Must be 500Gi or higher.	No, unless you need the storage size for the backup and restore pod to be greater than 500 Gi.
   imagePullPolicy	The pull policy for the images. When Red Hat OpenShift creates containers, it uses the imagePullPolicy to determine whether to pull the container image from the registry before starting the container. Options are Always, IfNotPresent, or Never.	No
   repository	The IBM® Entitled Registry from which the images are pulled. Must be set to cp.icr.io/cp/cp4s.	No
   repositoryUsername	The username for the IBM Entitled Registry that images are pulled from. Must be set to cp.	No
   repositoryPassword	The password to access the IBM Entitled Registry.	Yes
   deployDRC	Set to false to skip deployment of Detection and Response Center.	No, unless you don't want to deploy Detection and Response Center.
   deployThreatInvestigator	Set to false to skip deployment of Threat Investigator.	No, unless you don't want to deploy Threat Investigator.

6. Install QRadar Suite Software.

   Table 3. QRadar Suite Software installation command arguments

   Argument	Description
   --namespace	The namespace where QRadar Suite Software is installed. The namespace must meet the following criteria: Contain only lowercase alphanumeric characters or - Start and end with an alphanumeric character Be a dedicated namespace for QRadar Suite Software Not be default, kube-*, or openshift-* The namespace is created automatically if it does not exist.
   --allNamespaceMode	In this mode, the QRadar Suite Software operators are installed in the openshift-operators project (namespace). The QRadar Suite Software operators are available to all namespaces in the cluster. Note: If the allNamespaceMode is not selected, the QRadar Suite Software operators are installed in ownNamespaceMode and is only available in the Operand Namespace.
   --acceptLicense	Read the QRadar Suite Software license that is in the ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION/ibm-cp-security/licenses directory. By accepting the license, you confirm that you read the license and accept the terms. For the QRadar Suite Software installation to proceed, the acceptLicense true parameter is added to the installation action. After QRadar Suite Software is installed, you can use the license and usage page to turn on and off applications to comply with your QRadar Suite Software license purchase. For more information, see Managing licensing and usage.

   1. Set the QRadar Suite Software namespace variable by typing the following command.

      export CP4S_NAMESPACE=$NAMESPACE

   2. Install QRadar Suite Software by typing the following command.

      oc ibm-pak launch -t 1 \
      $CASE_NAME \
      --version $CASE_VERSION \
      --inventory ibmSecurityOperatorSetup \
      --namespace $CP4S_NAMESPACE \
      --action install \
      --args "--acceptLicense true --inputDir ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION"

   Important: Installation takes approximately 1.5 hours. When installation is complete, the latest version of IBM Cloud Pak® foundational services, and QRadar Suite Software 1.11.1 are installed.
7. Verify QRadar Suite Software installation by typing the following command.

   oc ibm-pak launch -t 1 \
   $CASE_NAME \
   --version $CASE_VERSION \
   --inventory ibmSecurityOperatorSetup \
   --namespace $CP4S_NAMESPACE \
   --action validate

[INFO] IBM Cloud Pak for Security deployment is complete.