Upgrading QRadar Suite Software in an air-gapped environment

The Red Hat OpenShift CLI client helps you develop, build, deploy and run your applications on any Red Hat OpenShift or Kubernetes cluster. It also includes the administrative commands for managing a cluster under the adm subcommand.

1. Download Red Hat OpenShift CLI 4.16 or 4.18 from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-4.16/ or https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-4.18/ . The file to download is called openshift-client-<platform>-<version>.tar.gz.
2. Extract the binary file that you downloaded by typing the following command, where <oc_cli_archive_file> is the name of the archive file that you downloaded.

   tar -xf <oc_cli_archive_file>

3. Modify the permissions of the binary file by typing the following command, where <oc_cli_binary> is the name of the Red Hat OpenShift binary that you extracted from the archive.
4. Move the binary file to the /usr/local/bin directory by typing the following command.

   mv <oc_cli_binary> /usr/local/bin/oc

   Tip: If this command returns a No such file or directory or Not a directory error message, create the /usr/local/bin directory by typing the following command.

   sudo mkdir /usr/local/bin

5. Make sure that the Red Hat OpenShift CLI client is working by typing the following command.

   oc version

   Tip: MacOS users might see a message that this tool cannot be opened because it is from an unidentified developer. Close this message and go to System Preferences > Security & Privacy. On the General tab, click Open Anyway or Allow Anyway. Repeat the oc version command.

Before mirroring your images, set the environment variables on your mirroring device, and connect to the internet so that you can download the corresponding CASE files.

1. Connect your mirroring device to the internet, and disconnect it from your local air-gapped network.
2. Create the following environment variables with the installer image name and the image inventory on your mirroring device by typing the following command.

   export CASE_NAME=ibm-cp-security && export CASE_VERSION=1.1.3

3. Download the IBM Security QRadar Suite Software installer and image inventory to your mirroring device by typing the following command.

   oc ibm-pak get $CASE_NAME --version $CASE_VERSION --disable-top-level-images-mode

   The CASE is saved to the ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION directory and the log file is saved to ~/.ibm-pak/logs/oc-ibm_pak.log.

   Tip: If you want to save the CASE to a directory other than your home directory, set the $IBMPAK_HOME environment variable by typing the following command.

   export IBMPAK_HOME=<working_directory>

   When you set the $IBMPAK_HOME environment variable, the CASE is saved to <working_directory>/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION and the log is saved to <working_directory>/.ibm-pak/logs/oc-ibm_pak.log.

   Important: If you change where the CASE is saved to, you must use $IBMPAK_HOME/.ibm-pak in place of ~/.ibm-pak throughout this procedure.

   Tip: If you want the installation process to be repeatable across environments, you can reuse the same saved CASE instead of downloading the CASE files again in other environments. You don't need to update versions of dependencies into the saved cache.

Mirroring images takes the image from the internet to your mirroring device, then effectively copies that image on to your air-gapped environment. After you mirror your images, you can configure your cluster and complete the air-gapped installation.

1. Set the $TARGET_REGISTRY environment variable to the IP address or FQDN and the port for your target registry by typing the following command. The target registry is the registry where your images are mirrored to and accessed by the Red Hat OpenShift Container Platform cluster.

   export TARGET_REGISTRY=<target_registry>

   For example, if your target registry is at 192.0.2.0:5000 type the following command.

   export TARGET_REGISTRY=192.0.2.0:5000

2. Generate the mirror manifests to use when you mirror the images to the target registry by typing the following command.

   oc ibm-pak generate mirror-manifests $CASE_NAME $TARGET_REGISTRY --version $CASE_VERSION

   Tip: If you want to view the list of images to be mirrored, type the following command.

   oc ibm-pak describe $CASE_NAME --version $CASE_VERSION --list-mirror-images

3. Store the authentication credentials for the IBM® Entitled Registry, cp.icr.io.
   If you are using Podman, store authentication credentials for cp.icr.io by typing the following commands.

   export REGISTRY_AUTH_FILE=~/.ibm-pak/auth.json

   podman login cp.icr.io -u cp

   The password is your IBM Entitled Registry key.

   The command stores and caches the registry credentials in the location that is specified for the $REGISTRY_AUTH_FILE environment variable.

4. Store the authentication credentials for your target registry.
   If you are using Podman, store authentication credentials for your target registry by typing the following commands.

   export REGISTRY_AUTH_FILE=~/.ibm-pak/auth.json

   podman login $TARGET_REGISTRY

   The command stores and caches the registry credentials in the location that is specified for the $REGISTRY_AUTH_FILE environment variable.

5. Mirror images to the target registry by typing the following command.

   oc image mirror \
   -f ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping.txt \
   --filter-by-os '.*'  \
   -a $REGISTRY_AUTH_FILE \
   --insecure  \
   --skip-multiple-scopes \
   --max-per-registry=1

6. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
   • Using a username and password.

     oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>

   • Using a token.

     oc login --token=<token> --server=<openshift_url>

7. Update the global image pull secret for your Red Hat OpenShift Container Platform cluster and add the credentials for your target registry.
   1. Retrieve the existing global pull secret by typing the following command, where <pull_secret_location> is the location of the file where you want to store the global pull secret configuration.

      oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' > <pull_secret_location>

   2. Add the new pull secret to the global pull secret file by typing the following command, where <username> and <password> are the username and password for your target registry.

      oc registry login --registry="$TARGET_REGISTRY" --auth-basic="<username>:<password>" --to=<pull_secret_location>

   3. Update the global pull secret in the cluster by typing the following command.

      oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location>

   4. Verify the status of the nodes by typing the following command.

      oc get MachineConfigPool -w

   After the global pull secret is updated, the configuration of your nodes is updated sequentially.

   Important: Wait until all MachineConfigPools are updated before you proceed to the next step.
8. Create the ImageContentSourcePolicy resource by typing the following command.

   oc apply -f ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/image-content-source-policy.yaml

   1. Verify that the ImageContentSourcePolicy resource is created by typing the following command.

      oc get imageContentSourcePolicy ibm-cp-security

   2. Verify the status of the nodes by typing the following command.

      oc get MachineConfigPool -w

   After the ImageContentSourcePolicy resource is created, the configuration of your nodes is updated sequentially.

   Important: Wait until all MachineConfigPools are updated before you proceed to the next step.
9. If you are using an insecure registry, you must add the local registry to the cluster insecureRegistries list by typing the following command.

   oc patch image.config.openshift.io/cluster --type=merge \
    -p '{"spec":{"registrySources":{"insecureRegistries":["'${TARGET_REGISTRY}'"]}}}'

   Important: Do not use insecure registries for production systems.
   1. Verify the status of the nodes by typing the following command.

      oc get MachineConfigPool -w

   After the insecureRegistries list is updated, the configuration of your nodes is updated sequentially.

   Important: Wait until all MachineConfigPools are updated before you proceed to the next step.

Before you prepare for an upgrade, you must schedule a maintenance window to avoid disruptions, and review the documentation for any new features or known issues that might impact the upgrade process. For more information, see What's new or changed and Known issues.

1. Back up your QRadar Suite Software data.
2. Verify that you have the latest serviceability pod in the QRadar Suite Software namespace by using the following command.

   oc rollout restart deployment cp-serviceability -n $QRS_NAMESPACE
   oc rollout status deployment cp-serviceability  -n $QRS_NAMESPACE

3. If you previously customized your Postgres storage sizes, these changes must also be applied to the EDB storage settings. The defaults for Stolon storage are:
   • de-udi-postgres: 500Gi
   • default-postgres: 220 Gi
   • soar-postgres: 220 Gi

   To check if the Stolon storage has been modified, run the following command and compare to the defaults:

   oc get pvc | grep postgres

   Data from the Stolon de-udi-postgres cluster will be migrated to the EDB default-postgres cluster. You can modify the EDB Storage settings by using the following methods:

   • Resizing the default EDB PVCs.
   • Resizing the SOAR EDB PVCs.

   There is not an existing EDB PVC at this point and the default storage settings for EDB are default-postgres: 630Gi, soar-postgres: 220 Gi

   Important:

   • The next step stops the Detection and Response Center application. QRadar Suite Software admins must not create new accounts until the upgrade process is complete.
   • The estimate time for data replication per GiB is 2 minutes.
   • Except for Connected Assets and Risk and Detection and Response Center applications, the QRadar Suite Software system remains available while the database replication runs online and new data that is written to the version 12 database replicates to version 14.
4. Install PostgreSQL version 14, and then start the data replication process from PostgreSQL version 12 to version 14 by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade start-replication --token=$(oc whoami -t) 

   The following example shows the output after you run the command.

   INFO: Checking storage configuration in cp4s-values and current DB cluster utilization, note that data from stolon instance de-udi will be replicated to edb instance default
   INFO: stolon default - storage setting: 220Gi, current database size: 127.5 Mi
   INFO: stolon de-udi - storage setting: 500Gi, current database size: 47.8 Mi
   INFO: stolon soar - storage setting: 220Gi, current database size: 98.5 Mi
   INFO: edb default - storage setting: 630Gi
   INFO: edb soar - storage setting: 220Gi
   INFO: installing edb postgres
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: processing edb install...
   INFO: edb clusters installed
   INFO: creating edb database soar/activemq
   INFO: creating edb database de-udi/atkhunts
   INFO: creating edb database de-udi/stixdata
   INFO: creating edb database default/apps
   INFO: creating edb database default/car
   INFO: creating edb database soar/co3
   INFO: creating edb database default/configstore
   INFO: creating edb database default/drc
   INFO: creating edb database default/edgegateway
   INFO: creating edb database default/cp4s_entitlements
   INFO: creating edb database default/cp4s_changelog
   INFO: creating edb database default/pulsebackend
   INFO: creating edb database default/advisor
   INFO: creating edb database default/cp4s_tis
   INFO: stopping drc services
   INFO: configuring replication publications and subscriptions
   INFO: replication configuration complete

5. Monitor the status of data replication by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade check-replication --token=$(oc whoami -t) 

   The following example output displays the replication status of each database. Before you proceed to the next step, make sure that the Copy Status is Complete and the Streaming Lag is less than 10 Mib.

   Instance  Database           Replication Status  Tables Copied  Copy Status  Streaming Lag  Streaming Status  WAL Held
   de-udi    atkhunts           Configured          5/5            Complete     0 bytes        active            56 bytes
   de-udi    stixdata           Configured          3/3            Complete     0 bytes        active            56 bytes
   default   advisor            Configured          169/169        Complete     0 bytes        active            60.5 Ki
   default   apps               Configured          22/22          Complete     0 bytes        active            60.5 Ki
   default   car                Configured          360/360        Complete     0 bytes        active            60.5 Ki
   default   car1               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car2               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car3               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car4               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car5               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car6               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car7               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   car8               Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   configstore        Configured          1/1            Complete     0 bytes        active            60.5 Ki
   default   cp4s_changelog     Configured          4/4            Complete     0 bytes        active            60.5 Ki
   default   cp4s_entitlements  Configured          8/8            Complete     0 bytes        active            60.5 Ki
   default   cp4s_tis           Configured          11/11          Complete     0 bytes        active            60.5 Ki
   default   drc                Configured          1381/1381      Complete     0 bytes        active            60.5 Ki
   default   edgegateway        Configured          3/3            Complete     0 bytes        active            60.5 Ki
   default   pulsebackend       Configured          0/0            Complete     0 bytes        active            60.5 Ki
   default   tii_threats        Configured          1/1            Complete     0 bytes        active            60.5 Ki
   soar      activemq           Configured          4/4            Complete     0 bytes        active            56 bytes
   soar      co3                Configured          565/565        Complete     0 bytes        active            56 bytes

   If the copy status of a database does not decrement or if the streaming lag is inactive, you can restart the replication for the database by using the following command.

   oc exec -it deploy/cp-serviceability -- postgres_upgrade reset-replication -t $(oc whoami -t) -d <database>

   For example, to restart the data replication for the configstore database, use the following command.

   oc exec -it deploy/cp-serviceability -- postgres_upgrade reset-replication -t $(oc whoami -t) -d configstore

6. Schedule a maintenance window for the upgrade.
   Important: All applications stop in the next step.
7. Complete the data replication process by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade complete-replication --token=$(oc whoami -t) 

   Tip:

   By default, the streaming lag must be less than 10 MiB to run the complete-replication command. On busy systems, replication streaming might not keep up with the live transactions. In this case, you can over ride the default setting by using the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade complete-replication --token=$(oc whoami -t) -s <MiB>

   To allow a 1024 MiB steaming lag, you can use the following command.

   oc exec deploy/cp-serviceability -- /opt/bin/postgres_upgrade complete-replication --token=$(oc whoami -t) -s 1024

   If you increase the streaming lag, the replication process takes longer to complete.

   The following example shows the output after you run the command.

   INFO: replication lag within threshold, stopping applications
   INFO: deleting deployment isc-cases-activemq
   INFO: deleting deployment isc-cases-co3postgres-operator
   INFO: deleting deployment isc-cases-operator
   INFO: deleting deployment debackend
   INFO: deleting deployment ibm-aitk-orchestrator
   INFO: deleting deployment isc-aitk-thaddeus
   INFO: deleting deployment tis-udiworkers
   INFO: deleting deployment udi-udiedgeservice
   INFO: deleting deployment udi-udiendpoints
   INFO: deleting deployment udi-udiworkers
   INFO: deleting deployment isc-app-manager-ds
   INFO: deleting deployment car
   INFO: deleting deployment idrmingestion
   INFO: deleting deployment isc-cases-application
   INFO: deployment isc-cases-co3postgres-operator already deleted
   INFO: deleting deployment isc-cases-event-handler
   INFO: deployment isc-cases-operator already deleted
   INFO: deleting deployment inf-configstore
   INFO: deleting deployment drc
   INFO: deleting deployment drcapi
   INFO: deleting deployment drcimporter
   INFO: deployment drc already deleted
   INFO: deployment drcapi already deleted
   INFO: deployment drcimporter already deleted
   INFO: deployment drc already deleted
   INFO: deployment drcapi already deleted
   INFO: deployment drcimporter already deleted
   INFO: deleting deployment edgegateway-ui
   INFO: deleting deployment isc-entitlements
   INFO: deleting deployment pulsedashboard
   INFO: deleting deployment idrmdashboard
   INFO: deployment idrmingestion already deleted
   INFO: deleting deployment idrmintex
   INFO: deleting deployment idrmriskengine
   INFO: deleting deployment threat-inv-api
   INFO: deleting deployment tis-data-gateway
   INFO: databases are in sync
   INFO: removing replication subscriptions and publications
   INFO: syncing postgres sequences
   INFO: updating connection details in postgres secrets

8. Choose only one of the following upgrade methods.
   • Upgrading in an air-gapped environment by using a bastion host.
   • Upgrading in an air-gapped environment by using a portable device.

oc get cp4sthreatmanagement.isc.ibm.com/threatmgmt -o jsonpath='{.spec.basicDeploymentConfiguration.adminUser}' -n <cp4s_namespace>

oc get cp4sthreatmanagement.isc.ibm.com/threatmgmt -o jsonpath='{.spec.basicDeploymentConfiguration.domain}' -n <cp4s_namespace>

oc get cp4sthreatmanagement.isc.ibm.com/threatmgmt -o jsonpath='{.spec.basicDeploymentConfiguration.storageClass}' -n <cp4s_namespace>

1. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
   • Using a username and password.

     oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>

   • Using a token.

     oc login --token=<token> --server=<openshift_url>

2. Set the $CP4S_NAMESPACE environment variable by typing the following command, where <cp4s_namespace> is the namespace where you are installing QRadar Suite Software.
   Important: If you install QRadar Suite Software in the all namespace mode, set the <cp4s_namespace> value as openshift-operators.

   export CP4S_NAMESPACE=<cp4s_namespace>

3. Extract the QRadar Suite Software CASE by typing the following command.

   tar -xf \
   ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION/ibm-cp-security-$CASE_VERSION.tgz \
   -C ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION

4. Update the parameters in the ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION/ibm-cp-security/inventory/ibmSecurityOperatorSetup/files/values.conf file. The following table lists the configurable parameters for the QRadar Suite Software installation and their descriptions.

   Table 3. QRadar Suite Software installation parameters

   Parameter	Description	Do you need to update this parameter?
   adminUser	The user that is given administrator privileges in the QRadar Suite Software System Administration account after installation. Specify a username or an email address that exists in your identity provider.	Yes
   airgapInstall	Set to true.	Yes
   clusterProxy	Set to false. Cluster-wide proxy is not supported in an air-gapped environment.	No
   domain	The fully qualified domain name (FQDN) created for QRadar Suite Software. If you don't specify an FQDN, it is generated as cp4s.<cluster_ingress_subdomain>.	No, unless you want to specify your own FQDN.
   domainCertificatePath	The path of the TLS certificate that is associated with the QRadar Suite Software domain. If the domain is not specified, the Red Hat OpenShift cluster certificates are used. For more information, see Domain name and TLS certificates.	No, unless you updated the domain parameter.
   domainCertificateKeyPath	The path of the TLS key that is associated with the QRadar Suite Software domain. If the domain is not specified, the Red Hat OpenShift cluster certificates are used. For more information, see Domain name and TLS certificates.	No, unless if you updated the domain parameter.
   customCaFilePath	The path of the custom TLS certificate associated with the QRadar Suite Software domain. For more information, see Domain name and TLS certificates.	No, unless you are using a custom or self-signed certificate.
   storageClass	The provisioned block or file storage class for all the PVCs required by QRadar Suite Software. When it is not specified, the default storage class is used. For more information, see Storage requirements.	No, unless you are using a storage class other than the default storage class for the cluster.
   backupStorageClass	Storage class for the backup and restore PVC. If this value is not set, QRadar Suite Software takes the value from the storageClass parameter.	No, unless you are using a different storage class for the backup and restore pod than you set for the storageClass parameter.
   backupStorageSize	The storage size for the backup and restore PVC. Must be 500Gi or higher.	No, unless you need the storage size for the backup and restore PVC to be greater than 500 Gi.
   imagePullPolicy	The pull policy for the images. When Red Hat OpenShift creates containers, it uses the imagePullPolicy to determine whether to pull the container image from the registry before it starts the container. Options are Always, IfNotPresent, or Never.	No
   repository	Specify the URL and port for the local Podman registry with the /cp/cp4s namespace appended. For example, example-registry:5000/cp/cp4s.	Yes
   repositoryUsername	The username to access your target registry.	Yes
   repositoryPassword	The password to access your target registry.	Yes
   deployDRC	Set to true to deploy Detection and Response Center. Set to false to skip deployment of Detection and Response Center. For more information, see Exploring security rule use cases with Detection and Response Center.	No, unless you don't want to deploy Detection and Response Center.
   deployThreatInvestigator	Set to true to deploy Threat Investigator. Set to false to skip deployment of Threat Investigator. For more information, see Investigating cases with IBM Security Threat Investigator.	No, unless you don't want to deploy Threat Investigator.

5. Install QRadar Suite Software by typing the following command.

   Table 4. QRadar Suite Software installation command arguments

   Argument	Description
   --allNamespaceMode	In this mode, the QRadar Suite Software operators are installed in the openshift-operators project (namespace). The QRadar Suite Software operators are available to all namespaces in the cluster. Note: If the allNamespaceMode is not selected, the QRadar Suite Software operators are installed in ownNamespaceMode and is only available in the Operand Namespace.
   --acceptLicense	Read the QRadar Suite Software license that is in the $HOME/ibm-cp-security/licenses directory. By accepting the license, you confirm that you read the license and accept the terms. For the QRadar Suite Software installation to proceed, the acceptLicense true parameter is added to the installation action. After QRadar Suite Software is installed, you can use the license and usage page to turn on and off applications to comply with your QRadar Suite Software license purchase. For more information, see Managing licensing and usage.

   oc ibm-pak launch -t 1 \
   $CASE_NAME \
   --version $CASE_VERSION \
   --inventory ibmSecurityOperatorSetup \
   --namespace $CP4S_NAMESPACE \
   --action install \
   --args "--acceptLicense true --inputDir ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION"

   Important: Installation takes approximately 1.5 hours. When installation is complete, the latest version of IBM Cloud Pak foundational services, and QRadar Suite Software 1.11.1 are installed.
6. Verify QRadar Suite Software installation by typing the following command.

   oc ibm-pak launch -t 1 \
   $CASE_NAME \
   --version $CASE_VERSION \
   --inventory ibmSecurityOperatorSetup \
   --namespace $CP4S_NAMESPACE \
   --action validate

[INFO] IBM Cloud Pak for Security deployment is complete.