Upgrading QRadar Suite Software using the Red Hat OpenShift CLI

The Red Hat OpenShift CLI client helps you develop, build, deploy and run your applications on any Red Hat OpenShift or Kubernetes cluster. It also includes the administrative commands for managing a cluster under the adm subcommand.

1. Download Red Hat OpenShift CLI 4.16 or 4.18 from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-4.16/ or https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-4.18/ . The file to download is called openshift-client-<platform>-<version>.tar.gz.
2. Extract the binary file that you downloaded by typing the following command, where <oc_cli_archive_file> is the name of the archive file that you downloaded.

   tar -xf <oc_cli_archive_file>

3. Modify the permissions of the binary file by typing the following command, where <oc_cli_binary> is the name of the Red Hat OpenShift binary that you extracted from the archive.
4. Move the binary file to the /usr/local/bin directory by typing the following command.

   mv <oc_cli_binary> /usr/local/bin/oc

   Tip: If this command returns a No such file or directory or Not a directory error message, create the /usr/local/bin directory by typing the following command.

   sudo mkdir /usr/local/bin

5. Make sure that the Red Hat OpenShift CLI client is working by typing the following command.

   oc version

   Tip: MacOS users might see a message that this tool cannot be opened because it is from an unidentified developer. Close this message and go to System Preferences > Security & Privacy. On the General tab, click Open Anyway or Allow Anyway. Repeat the oc version command.

Before you prepare for an upgrade, you must schedule a maintenance window to avoid disruptions, and review the documentation for any new features or known issues that might impact the upgrade process. For more information, see What's new or changed and Known issues.

1. Back up your QRadar Suite Software data.
2. Ensure that the serviceability deployment has the icr.io/cpopen/cp4s/cp-serviceability:1.10-latest image and imagePullPolicy set to Always by running the following commands:

   oc get deployment cp-serviceability -o jsonpath='{.spec.template.spec.containers[].image}' -n $QRS_NAMESPACE
   oc get deployment cp-serviceability -o jsonpath='{.spec.template.spec.containers[].imagePullPolicy}' -n $QRS_NAMESPACE

   Restart the cp-serviceability pod to ensure the latest version is running, as follows:

   oc rollout restart deployment cp-serviceability -n $QRS_NAMESPACE
   oc rollout status deployment cp-serviceability  -n $QRS_NAMESPACE

3. If you previously customized your Postgres storage sizes, these changes must also be applied to the EDB storage settings. The defaults for Stolon storage are:
   • de-udi-postgres: 500Gi
   • default-postgres: 220 Gi
   • soar-postgres: 220 Gi

   To check if the Stolon storage has been modified, run the following command and compare to the defaults:

   oc get pvc | grep postgres

   Data from the Stolon de-udi-postgres cluster will be migrated to the EDB default-postgres cluster. You can modify the EDB Storage settings by using the following methods:

   • Resizing the default EDB PVCs.
   • Resizing the SOAR EDB PVCs.

   There is not an existing EDB PVC at this point and the default storage settings for EDB are default-postgres: 630Gi, soar-postgres: 220 Gi

   Important:

   • The next step stops the Detection and Response Center application. QRadar Suite Software admins must not create new accounts until the upgrade process is complete.
   • The estimate time for data replication per GiB is 2 minutes.
   • Except for Connected Assets and Risk and Detection and Response Center applications, the QRadar Suite Software system remains available while the database replication runs online and new data that is written to the version 12 database replicates to version 14.
4. Install PostgreSQL version 14, and then start the data replication process from PostgreSQL version 12 to version 14 by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade start-replication --token=$(oc whoami -t) 

   The following example shows the output after you run the command.

   INFO: Checking storage configuration in cp4s-values and current DB cluster utilization, note that data from stolon instance de-udi will be replicated to edb instance default
   INFO: stolon default - storage setting: 220Gi, current database size: 127.5 Mi
   INFO: stolon de-udi - storage setting: 500Gi, current database size: 47.8 Mi
   INFO: stolon soar - storage setting: 220Gi, current database size: 98.5 Mi
   INFO: edb default - storage setting: 630Gi
   INFO: edb soar - storage setting: 220Gi
   INFO: installing edb postgres
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: edb clusters installed
   INFO: creating edb database soar/activemq
   INFO: creating edb database de-udi/atkhunts
   INFO: creating edb database de-udi/stixdata
   INFO: creating edb database default/apps
   INFO: creating edb database default/car
   INFO: creating edb database soar/co3
   INFO: creating edb database default/configstore
   INFO: creating edb database default/drc
   INFO: creating edb database default/edgegateway
   INFO: creating edb database default/cp4s_entitlements
   INFO: creating edb database default/cp4s_changelog
   INFO: creating edb database default/pulsebackend
   INFO: creating edb database default/advisor
   INFO: creating edb database default/cp4s_tis
   INFO: stopping drc services
   INFO: configuring replication publications and subscriptions
   INFO: replication configuration complete

5. Monitor the status of data replication by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade check-replication --token=$(oc whoami -t) 

   The following example output displays the replication status of each database. Before you proceed to the next step, make sure that the Copy Status is Complete and the Streaming Lag is less than 10 Mib.

   Instance  Database           Replication Status  Tables Copied  Copy Status  Streaming Lag  Streaming Status  WAL Held
   de-udi    atkhunts           Configured          5/5            Complete     0 bytes        active            56 bytes
   de-udi    stixdata           Configured          3/3            Complete     0 bytes        active            56 bytes
   default   advisor            Configured          169/169        Complete     0 bytes        active            60.5 Ki
   default   apps               Configured          22/22          Complete     0 bytes        active            60.5 Ki
   default   car                Configured          360/360        Complete     0 bytes        active            60.5 Ki
   default   car1               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car2               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car3               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car4               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car5               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car6               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car7               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car8               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   configstore        Configured          1/1            Complete     0 bytes        active            60.5 Ki
   default   cp4s_changelog     Configured          4/4            Complete     0 bytes        active            60.5 Ki
   default   cp4s_entitlements  Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   cp4s_tis           Configured          11/11          Complete     0 bytes        active            60.5 Ki
   default   drc                Configured          1381/1381      Complete     0 bytes        active            60.5 Ki
   default   edgegateway        Configured          3/3            Complete     0 bytes        active            60.5 Ki
   default   pulsebackend       Configured          0/0            Complete     0 bytes        active            60.5 Ki
   default   tii_threats        Configured          1/1            Complete     0 bytes        active            60.5 Ki
   soar      activemq           Configured          4/4            Complete     0 bytes        active            56 bytes
   soar      co3                Configured          565/565        Complete     0 bytes        active            56 bytes

   If the copy status of a database does not decrement or if the streaming lag is inactive, you can restart the replication for the database by using the following command.

   oc exec -it deploy/cp-serviceability -- postgres_upgrade reset-replication -t $(oc whoami -t) -d <database>

   For example, to restart the data replication for the configstore database, use the following command.

   oc exec -it deploy/cp-serviceability -- postgres_upgrade reset-replication -t $(oc whoami -t) -d configstore

6. Schedule a maintenance window for the upgrade.
   Important: All applications stop in the next step.
7. Complete the data replication process by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade complete-replication --token=$(oc whoami -t) 

   Tip:

   By default, the streaming lag must be less than 10 MiB to run the complete-replication command. On busy systems, replication streaming might not keep up with the live transactions. In this case, you can over ride the default setting by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade complete-replication --token=$(oc whoami -t) -s <MiB>

   To allow a 1024 MiB steaming lag, you can use the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade complete-replication --token=$(oc whoami -t) -s 1024

   If you increase the streaming lag, the replication process takes longer to complete.

   The following example shows the output after you run the command.

   INFO: replication lag within threshold, stopping applications
   INFO: deleting deployment isc-cases-activemq
   INFO: deleting deployment isc-cases-co3postgres-operator
   INFO: deleting deployment isc-cases-operator
   INFO: deleting deployment debackend
   INFO: deleting deployment ibm-aitk-orchestrator
   INFO: deleting deployment isc-aitk-thaddeus
   INFO: deleting deployment tis-udiworkers
   INFO: deleting deployment udi-udiedgeservice
   INFO: deleting deployment udi-udiendpoints
   INFO: deleting deployment udi-udiworkers
   INFO: deleting deployment isc-app-manager-ds
   INFO: deleting deployment car
   INFO: deleting deployment idrmingestion
   INFO: deleting deployment isc-cases-application
   INFO: deployment isc-cases-co3postgres-operator already deleted
   INFO: deleting deployment isc-cases-event-handler
   INFO: deployment isc-cases-operator already deleted
   INFO: deleting deployment inf-configstore
   INFO: deleting deployment drc
   INFO: deleting deployment drcapi
   INFO: deleting deployment drcimporter
   INFO: deployment drc already deleted
   INFO: deployment drcapi already deleted
   INFO: deployment drcimporter already deleted
   INFO: deployment drc already deleted
   INFO: deployment drcapi already deleted
   INFO: deployment drcimporter already deleted
   INFO: deleting deployment edgegateway-ui
   INFO: deleting deployment isc-entitlements
   INFO: deleting deployment pulsedashboard
   INFO: deleting deployment idrmdashboard
   INFO: deployment idrmingestion already deleted
   INFO: deleting deployment idrmintex
   INFO: deleting deployment idrmriskengine
   INFO: deleting deployment threat-inv-api
   INFO: deleting deployment tis-data-gateway
   INFO: databases are in sync
   INFO: removing replication subscriptions and publications
   INFO: syncing postgres sequences
   INFO: updating connection details in postgres secrets

8. Delete the ibm-cp-security-operator CSV by using the following command.

   OPERATOR_NS=<cp4s namespace or openshift-operators>
   oc get csv -n $OPERATOR_NS -o name | grep ibm-cp-security-operator | xargs oc delete -n $OPERATOR_NS

1. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
   • Using a username and password.

     oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>

   • Using a token.

     oc login --token=<token> --server=<openshift_url>

2. Set the $CP4S_NAMESPACE environment variable by typing the following command, where <cp4s_namespace> is the namespace where you are installing QRadar Suite Software.
   Important: If you install QRadar Suite Software in the all namespace mode, set the <cp4s_namespace> value as openshift-operators.

   export CP4S_NAMESPACE=<cp4s_namespace>

3. Start the upgrade process by using the following command.

   oc patch subs ibm-cp-security-operator --type merge -p '{"spec":{"channel":"v1.11"}}' -n $CP4S_NAMESPACE

4. The upgrade takes approximately 1 hour and 30 minutes. You can monitor the progress of the upgrade by using the following command:

   oc get cp4sthreatmanagement threatmgmt -n $CP4S_NAMESPACE

   When you see the following message, the upgrade is complete:

   [INFO] IBM Cloud Pak for Security deployment is complete.

   If you are using SAML authentication, follow the steps in Reconfiguring SAML authentication after upgrading to QRadar Suite Software 1.11.
   1. Log in to your QRadar Suite Software console.
   2. Click you user icon, then About.
   3. Verify that the displayed version number is 1.11.1.