Installing and configuring software on the source cluster for backup and restore with IBM Fusion

1.1 Installing the IBM Software Hub command-line interface

To install IBM Software Hub software on your Red Hat OpenShift Container Platform cluster, you must install the IBM Software Hub command-line interface (cpd-cli) on the workstation from which you are running the installation commands.

Who needs to complete this task?

User	Why you need the cpd-cli
Cluster administrator	Configure the image pull secret Change node settings Set up projects where IBM Software Hub will be installed
Instance administrator	Install IBM Software Hub
Registry administrator	Mirror images to the private container registry.

When do you need to complete this task?

Repeat as needed You must complete this task on any workstation from which you plan to run installation commands.

You can also complete this task if you need to use the cpd-cli to complete other tasks, such as backing up and restoring your installation or managing users.

You must install the cpd-cli on a client workstation that can connect to your cluster.

1. Download the cpd-cli from the IBM/cpd-cli repository on GitHub.

   Ensure that you download the correct package based on the operating system on the client workstation:

   Workstation operating system	Enterprise Edition	Standard Edition
   Linux	The package that you download depends on your hardware: x86_64 cpd-cli-linux-EE-14.3.0.tgz ppc64le cpd-cli-ppc64le-EE-14.3.0.tgz s390x cpd-cli-s390x-EE-14.3.0.tgz	The package that you download depends on your hardware: x86_64 cpd-cli-linux-SE-14.3.0.tgz ppc64le cpd-cli-ppc64le-SE-14.3.0.tgz s390x cpd-cli-s390x-SE-14.3.0.tgz
   Mac OS	cpd-cli-darwin-EE-14.3.0.tgz	cpd-cli-darwin-SE-14.3.0.tgz
   Windows	You must download the Linux package and run it in Windows Subsystem for Linux: cpd-cli-linux-EE-14.3.0.tgz	You must download the Linux package and run it in Windows Subsystem for Linux: cpd-cli-linux-SE-14.3.0.tgz

2. Extract the contents of the package to the directory where you want to run the cpd-cli.
3. On Mac OS, you must trust the following components of the cpd-cli:
   • cpd-cli
   • plugins/lib/darwin/config
   • plugins/lib/darwin/cpdbr
   • plugins/lib/darwin/cpdbr-oadp
   • plugins/lib/darwin/cpdctl
   • plugins/lib/darwin/cpdtool
   • plugins/lib/darwin/health
   • plugins/lib/darwin/manage
   • plugins/lib/darwin/platform-diag
   • plugins/lib/darwin/platform-mgmt

   For each component:

   1. Right-click the component and select Open.

      You will see a message with the following format:

      macOS cannot verify the developer of component-name. Are you sure you want to open it?

   2. Click Open.
4. Best practice Make the cpd-cli executable from any directory.

   By default, you must either change to the directory where the cpd-cli is located or specify the fully qualified path of the cpd-cli to run the commands.

   However, you can make the cpd-cli executable from any directory so that you only need to type cpd-cli command-name to run the commands.

   Workstation operating system	Details
   Linux	Add the following line to your ~/.bashrc file: export PATH=<fully-qualified-path-to-the-cpd-cli>:$PATH
   Mac OS	Add the following line to your ~/.bash_profile or ~/.zshrc file: export PATH=<fully-qualified-path-to-the-cpd-cli>:$PATH
   Windows	From the Windows Subsystem for Linux, add the following line to your ~/.bashrc file: export PATH=<fully-qualified-path-to-the-cpd-cli>:$PATH

5. Best practice Determine whether you need to set any of the following environment variables for the cpd-cli.

   CPD_CLI_MANAGE_WORKSPACE

   By default, the first time you run a cpd-cli manage command, the cpd-cli automatically creates the cpd-cli-workspace/olm-utils-workspace/work directory.

   The location of the directory depends on several factors:

   • If you made the cpd-cli executable from any directory, the directory is created in the directory where you run the cpd-cli commands.
   • If you did not make the cpd-cli executable from any directory, the directory is created in the directory where the cpd-cli is installed.

   You can set the CPD_CLI_MANAGE_WORKSPACE environment variable to override the default location.

   The CPD_CLI_MANAGE_WORKSPACE environment variable is especially useful if you made the cpd-cli executable from any directory. When you set the environment variable, it ensures that the files are located in one directory.

   Default value

   No default value. The directory is created based on the factors described in the preceding text.

   Valid values

   The fully qualified path where you want the cpd-cli to create the work directory. For example, if you specify /root/cpd-cli/, the cpd-cli manage plug-in stores files in the /root/cpd-cli/work directory.

   To set the CPD_CLI_MANAGE_WORKSPACE environment variable, run:

   export CPD_CLI_MANAGE_WORKSPACE=<fully-qualified-directory>

   OLM_UTILS_LAUNCH_ARGS

   You can use the OLM_UTILS_LAUNCH_ARGS environment variable to mount certificates that the cpd-cli must use in the cpd-cli container.

   Mount CA certificates

   Important: If you use a proxy server to mirror images or to download CASE packages, use the OLM_UTILS_LAUNCH_ARGS environment variable to add the CA certificates to enable the olm-utils container to trust connections through the proxy server. For more information, see Cannot access CASE packages when using a proxy server.

   You can mount CA certificates if you need to reach an external HTTPS endpoint that uses a self-signed certificate.

   Tip: Typically the CA certificates are in the /etc/pki/ca-trust directory on the workstation. If you need additional information on adding certificates to a workstation, run:

   man update-ca-trust

   Determine the correct argument for your environment:

   • If the certificates on the client workstation are in the /etc/pki/ca-trust directory, the argument is:

     " -v /etc/pki/ca-trust:/etc/pki/ca-trust"

   • If the certificates on the client workstation are in a different directory, replace <ca-loc> with the appropriate location on the client workstation:

     " -v <ca-loc>:/etc/pki/ca-trust"

   Mount Kubernetes certificates

   You can mount Kubernetes certificates if you need to use a certificate to connect to the Kubernetes API server.

   The argument depends on the location of the certificates on the client workstation. Replace <k8-loc> with the appropriate location on the client workstation:

   " -v <k8-loc>:/etc/k8scert --env K8S_AUTH_SSL_CA_CERT=/etc/k8scert"

   Default value

   No default value.

   Valid values

   The valid values depend on the arguments that you need to pass to the OLM_UTILS_LAUNCH_ARGS environment variable.

   • To pass CA certificates, specify:

     " -v <ca-loc>:/etc/pki/ca-trust"

   • To pass Kubernetes certificates, specify:

     " -v <k8-loc>:/etc/k8scert --env K8S_AUTH_SSL_CA_CERT=/etc/k8scert"

   • To pass both CA certificates and Kubernetes certificates, specify:

     " -v <ca-loc>:/etc/pki/ca-trust -v <k8-loc>:/etc/k8scert --env K8S_AUTH_SSL_CA_CERT=/etc/k8scert"

   To set the OLM_UTILS_LAUNCH_ARGS environment variable, run:

   export OLM_UTILS_LAUNCH_ARGS=" <arguments>"

   Important: If you set either of these environment variables, ensure that you add them to your installation environment variables script.
6. Run the following command to ensure that the cpd-cli is installed and running and that the cpd-cli manage plug-in has the correct version of the olm-utils-v4 image.

   cpd-cli manage restart-container

1.2 Installing the OpenShift command-line interface

The IBM Software Hub command-line interface (cpd-cli) interacts with the OpenShift command-line interface (oc CLI) to issue commands to your Red Hat OpenShift Container Platform cluster.

Who needs to complete this task?

All administrators Any users who are completing IBM Software Hub installation tasks must install the OpenShift CLI.

When do you need to complete this task?

Repeat as needed You must complete this task on any workstation from which you plan to run installation commands.

You must install a version of the OpenShift CLI that is compatible with your Red Hat OpenShift Container Platform cluster.

To install the OpenShift CLI, follow the appropriate guidance for your cluster:

Self-managed clusters

Install the version of the oc CLI that corresponds to the version of Red Hat OpenShift Container Platform that you are running. For details, see Getting started with the OpenShift CLI in the Red Hat OpenShift Container Platform documentation:

• Version 4.14
• Version 4.16
• Version 4.17
• Version 4.18
• Version 4.19
• Version 4.20

Managed OpenShift clusters

Follow the appropriate guidance for your managed OpenShift environment.

OpenShift environment	Installation instructions
IBM Cloud Satellite	See Installing the CLI in the IBM Cloud Satellite documentation.
Red Hat OpenShift on IBM Cloud	See Installing the CLI in the IBM Cloud documentation.
Azure Red Hat OpenShift (ARO)	See Install the OpenShift CLI in the Azure Red Hat OpenShift documentation
Red Hat OpenShift Service on AWS (ROSA)	See Installing the OpenShift CLI in the Red Hat OpenShift Service on AWS documentation.
Red Hat OpenShift Dedicated on Google Cloud	See Learning how to use the command-line tools for Red Hat OpenShift Dedicated in the Red Hat OpenShift Dedicated documentation.

3.1 Creating environment variables

Create the following environment variables so that you can copy commands from the documentation and run them without making any changes.

3.3 Configuring the Data Protection Application

Complete the following steps to configure the Data Protection Application (DPA) for IBM Software Hub:

1. Run the following command increase the ephemeral storage and memory limits for the Velero pods:

   oc patch dpa \
   -n ibm-backup-restore velero \
   --type merge \
   -p '{"spec": {"configuration": {"velero": {"podConfig": {"resourceAllocations": {"limits": {"ephemeral-storage": "2Gi", "memory": "4Gi"}}}}}}}'

2. Configure the plugins required by IBM Software Hub:

   DPA="velero"
   NAMESPACE="ibm-backup-restore"
   
   add_plugin_if_missing() {
     local PLUGIN_NAME="$1"
     local PLUGIN_IMAGE="$2"
     local EXPECTED_IMAGE_VERSION="${PLUGIN_IMAGE##*:}"
   
     echo -e "\nChecking plugin: $PLUGIN_NAME"
   
     EXISTING_IMAGE=$(oc get dpa "$DPA" -n "$NAMESPACE" -o json | jq -r \
       --arg name "$PLUGIN_NAME" '.spec.configuration.velero.customPlugins[]? | select(.name == $name) | .image')
     if [ -z "$EXPECTED_IMAGE_VERSION" ]; then
       echo "version tag must be defined, current arg=$PLUGIN_IMAGE"
       return 1
     fi
   
     EXISTING_IMAGE_INDEX=$(oc get dpa "$DPA" -n "$NAMESPACE" -o json | jq -r \
       --arg name "$PLUGIN_NAME" '[.spec.configuration.velero.customPlugins[]? | .name] | index($name)')
   
     INSTALLED_IMAGE_VERSION="${EXISTING_IMAGE##*:}"  # Gets everything after the last ":"
     if [ -n "$EXISTING_IMAGE" ]; then
       if [ "$INSTALLED_IMAGE_VERSION" = "$EXPECTED_IMAGE_VERSION" ]; then
         echo "Plugin '$PLUGIN_NAME' already exists with image: $EXISTING_IMAGE and in correct version, aborting DPA update"
       else
         echo "Plugin '$PLUGIN_NAME' needs to be patched. Patching DPA customPlugins with image $PLUGIN_IMAGE ..."
         oc patch dpa "$DPA" -n "$NAMESPACE" --type='json' \
           -p='[{"op":"replace","path":"/spec/configuration/velero/customPlugins/'$EXISTING_IMAGE_INDEX'","value":{"name":"'"$PLUGIN_NAME"'","image":"'"$PLUGIN_IMAGE"'"}}]'
   
         if [ $? -ne 0 ]; then
             echo "ERROR: oc patch command failed"
             return 1
         fi
       fi
     else
       echo "Plugin '$PLUGIN_NAME' not found. Adding DPA customPlugins with image $PLUGIN_IMAGE ..."
       oc patch dpa "$DPA" -n "$NAMESPACE" --type='json' -p='[{"op":"add","path":"/spec/configuration/velero/customPlugins/-","value":{"name":"'"$PLUGIN_NAME"'","image":"'"$PLUGIN_IMAGE"'"}}]'
       if [ $? -ne 0 ]; then
           echo "ERROR: oc patch command failed"
           return 1
       fi
     fi
   }
   
   add_plugin_if_missing "cpfs-oadp-plugin" "icr.io/cpopen/cpfs/cpfs-oadp-plugins:${CPFS_OADP_PLUGIN_VERSION}"
   add_plugin_if_missing "db2u-velero-plugin" "icr.io/db2u/db2u-velero-plugin:${VERSION}"
   add_plugin_if_missing "swhub-velero-plugin" "icr.io/cpopen/cpd/swhub-velero-plugin:${VERSION}"

3.4 Optional: Enabling parallel hook processing

If you install IBM Fusion version 2.11 or later, you can configure IBM Software Hub to run multiple service hooks simultaneously instead of sequentially.

You can enable parallel processing of backup hooks by setting the hook-parallel-workers parameter in the IBM Fusion guardian-configmap ConfigMap. By default, this value is treated as 0, which means hooks are executed sequentially. The maximum number of hook-parallel-workers that you can set is 35.

For more information about parallel hook processing, see Parallel hook execution in the IBM Fusion documentation.

• To set the number of hook-parallel-workers, use the following command:

  oc patch cm guardian-configmap -n ibm-backup-restore --patch '{"data":{"hook-parallel-workers": "3"}}'

• To clear the hook-parallel-workers value, use the following command:

  oc patch cm guardian-configmap -n ibm-backup-restore --type='json' -p='[{"op": "remove", "path": "/data/hook-parallel-workers"}]'

• To check the hook-parallel-workers value, use the following command:

  oc get cm guardian-configmap -n ibm-backup-restore -o yaml | yq '.data.hook-parallel-workers'

3.5 Installing the cpdbr service

Install the cpdbr service on the source cluster by doing the following steps.

1. To install the cpdbr service on the source cluster, do the following steps.
   1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

      ${OC_LOGIN}

      Remember: OC_LOGIN is an alias for the oc login command.
   2. Configure the client to set the OADP project:

      cpd-cli oadp client config set namespace=${OADP_PROJECT}

   3. In the IBM Software Hub operators project, install the cpdbr service.

      ---

      The cluster pulls images from the IBM Entitled Registry

      Environments with the scheduling service

      cpd-cli oadp install \
      --component=cpdbr-tenant \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpdbr-hooks-image-prefix=icr.io/cpopen/cpd \
      --cpfs-image-prefix=icr.io/cpopen/cpfs \
      --namespace=${OADP_PROJECT} \
      --cpd-scheduler-namespace=${PROJECT_SCHEDULING_SERVICE} \
      --log-level=debug \
      --verbose

      Tip: If the cloud object store for the IBM Fusion backup service uses a CA certificate, you can use the --backup-validation-cacert option to provide the base64 encoded certificate.

      Environments without the scheduling service

      cpd-cli oadp install \
      --component=cpdbr-tenant \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpdbr-hooks-image-prefix=icr.io/cpopen/cpd \
      --cpfs-image-prefix=icr.io/cpopen/cpfs \
      --namespace=${OADP_PROJECT} \
      --log-level=debug \
      --verbose

      Tip: If the cloud object store for the IBM Fusion backup service uses a CA certificate, you can use the --backup-validation-cacert option to provide the base64 encoded certificate.

      ---

      The cluster pulls images from a private container registry

      Note: Before you can install images from a private container registry, check that an image content source policy was configured. For details, see Configuring an image content source policy for IBM Software Hub images.

      Environments with the scheduling service

      cpd-cli oadp install \
      --component=cpdbr-tenant \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpdbr-hooks-image-prefix=${PRIVATE_REGISTRY_LOCATION}/cpopen/cpd \
      --cpfs-image-prefix=${PRIVATE_REGISTRY_LOCATION}/cpopen/cpfs \
      --namespace=${OADP_PROJECT} \
      --cpd-scheduler-namespace=${PROJECT_SCHEDULING_SERVICE} \
      --log-level=debug \
      --verbose

      Tip: If the cloud object store for the IBM Fusion backup service uses a CA certificate, you can use the --backup-validation-cacert option to provide the base64 encoded certificate.

      Environments without the scheduling service

      cpd-cli oadp install \
      --component=cpdbr-tenant \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpdbr-hooks-image-prefix=${PRIVATE_REGISTRY_LOCATION}/cpopen/cpd \
      --cpfs-image-prefix=${PRIVATE_REGISTRY_LOCATION}/cpopen/cpfs \
      --namespace=${OADP_PROJECT} \
      --log-level=debug \
      --verbose

      Tip: If the cloud object store for the IBM Fusion backup service uses a CA certificate, you can use the --backup-validation-cacert option to provide the base64 encoded certificate.

      ---

   4. Optional: Verify that the cpdbr-tenant service pod is running:

      oc get pod -n ${PROJECT_CPD_INST_OPERATORS} | grep cpdbr

   5. 1. Configure the ibmcpd-tenant parent recipe:

         cpd-cli oadp generate plan fusion parent-recipe \
         --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
         --verbose \
         --log-level=debug

      2. Verify that the ibmcpd-tenant IBM Fusion recipe version is the same as the IBM Software Hub version:

         oc get frcpe \
         -n ${PROJECT_CPD_INST_OPERATORS} \
         -l icpdsupport/generated-by-cpdbr=true,icpdsupport/version=${VERSION}

   6. If the IBM Software Hub scheduling service is installed, verify that the following recipe was installed:

      oc get frcpe ibmcpd-scheduler \
      -n ${PROJECT_SCHEDULING_SERVICE}

   7. Verify that the appropriate version of the cpdbr service was installed:

      oc exec -n ${PROJECT_CPD_INST_OPERATORS} $(oc get -n ${PROJECT_CPD_INST_OPERATORS} po -l component=cpdbr-tenant,icpdsupport/app=br-service -o name) -- /cpdbr-scripts/cpdbr-oadp version

      Example output of the command:

      Client: 
             Version: 5.3.0
             Build Date: <timestamp>
             Build Number: <xxx>

2. To upgrade the cpdbr service on the source cluster, do the following steps.
   1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

      ${OC_LOGIN}

      Remember: OC_LOGIN is an alias for the oc login command.
   2. In the IBM Software Hub operators project, upgrade the cpdbr service.

      ---

      The cluster pulls images from the IBM Entitled Registry

      Environments with the scheduling service

      cpd-cli oadp install \
      --upgrade=true \
      --component=cpdbr-tenant \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpfs-image-prefix=icr.io/cpopen/cpfs \
      --cpd-scheduler-namespace=${PROJECT_SCHEDULING_SERVICE} \
      --namespace=${OADP_PROJECT} \
      --log-level=debug \
      --verbose

      Tip: If the cloud object store for the IBM Fusion backup service uses a CA certificate, you can use the --backup-validation-cacert option to provide the base64 encoded certificate.

      Environments without the scheduling service

      cpd-cli oadp install \
      --upgrade=true \
      --component=cpdbr-tenant \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpfs-image-prefix=icr.io/cpopen/cpfs \
      --namespace=${OADP_PROJECT} \
      --log-level=debug \
      --verbose

      Tip: If the cloud object store for the IBM Fusion backup service uses a CA certificate, you can use the --backup-validation-cacert option to provide the base64 encoded certificate.

      ---

      The cluster pulls images from a private container registry

      Note: Before you can install images from a private container registry, check that an image content source policy was configured. For details, see Configuring an image content source policy for IBM Software Hub images.

      Environments with the scheduling service

      cpd-cli oadp install \
      --upgrade=true \
      --component=cpdbr-tenant \
      --namespace=${OADP_PROJECT} \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpfs-image-prefix=${PRIVATE_REGISTRY_LOCATION}/cpopen/cpfs \
      --cpd-scheduler-namespace=${PROJECT_SCHEDULING_SERVICE} \
      --log-level=debug \
      --verbose

      Tip: If the cloud object store for the IBM Fusion backup service uses a CA certificate, you can use the --backup-validation-cacert option to provide the base64 encoded certificate.

      Environments without the scheduling service

      cpd-cli oadp install \
      --upgrade=true \
      --component=cpdbr-tenant \
      --namespace=${OADP_PROJECT} \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpfs-image-prefix=${PRIVATE_REGISTRY_LOCATION}/cpopen/cpfs \
      --log-level=debug \
      --verbose

      Tip: If the cloud object store for the IBM Fusion backup service uses a CA certificate, you can use the --backup-validation-cacert option to provide the base64 encoded certificate.

      ---

   3. Optional: Verify that the cpdbr-tenant service pod is running:

      oc get pod -n ${PROJECT_CPD_INST_OPERATORS} | grep cpdbr

   4. Verify the version of the cpdbr service:

      oc exec -n ${PROJECT_CPD_INST_OPERATORS} $(oc get -n ${PROJECT_CPD_INST_OPERATORS} po -l component=cpdbr-tenant,icpdsupport/app=br-service -o name) -- /cpdbr-scripts/cpdbr-oadp version

      The command returns output with the following format:

      Client: 
             Version: 5.3.0
             Build Date: <timestamp>
             Build Number: <xxx>

   5. 1. Configure the ibmcpd-tenant parent recipe:

         cpd-cli oadp generate plan fusion parent-recipe \
         --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
         --verbose \
         --log-level=debug

      2. Verify that the ibmcpd-tenant IBM Fusion recipe version is the same as the IBM Software Hub version:

         oc get frcpe \
         -n ${PROJECT_CPD_INST_OPERATORS} \
         -l icpdsupport/generated-by-cpdbr=true,icpdsupport/version=${VERSION}

When the cpdbr service is installed on the source cluster, the cpdbr component services are deployed and required permissions and cluster role bindings (ClusterRole, ClusterRoleBinding, Role, RoleBinding) are created.