Installing the backup and restore tools

Learn how to install the tools that are required for the backup and restore of IBM Cloud Pak for AIOps.

You need to install the Red Hat OpenShift APIs for Data Protection (OADP). OADP provides default Velero plug-ins that are integrated with storage providers to support backup operations.

Note: The backup and restore tools for IBM Cloud Pak for AIOps are the same as the tools for backing up and restoring Infrastructure Automation. If you have the tools installed for Infrastructure Automation, you do not need to complete the following procedure.

Prerequisites

  • Provision the object storage (S3 bucket) for storing your backups. The backup process moves backup files to an S3 bucket, which must be in an S3 compatible location, such as Amazon Web Services (AWS).
  • Ensure that the following command line tools are installed on your workstation:
    • oc
    • jq
    • git
  • Ensure that you are logged in to your Red Hat® OpenShift® Container Platform cluster with oc login for any steps that use the Red Hat OpenShift command-line interface (CLI).

Procedure

  1. Install the Velero client CLI
  2. Install OpenShift API for Data Protection (OADP)
  3. (Offline deployments only) Mirror the backup and restore image

1. Install the Velero client CLI

Install the Velero client CLI on the workstation that you plan to use to run the scripts for backing up and restoring the cluster.

  1. Run the following command to download the Velero CLI client:

    curl -LO https://github.com/vmware-tanzu/velero/releases/download/v1.12.3/velero-v1.12.3-linux-amd64.tar.gz
    
  2. Extract the tarball:

    tar -xzf velero-v1.12.3-linux-amd64.tar.gz
    
  3. Move the extracted velero binary to /usr/local/bin:

    mv velero-v1.12.3-linux-amd64/velero /usr/local/bin/velero 
    
  4. Run the following command to verify the installation.

    velero version
    

    Example output:

    Client:
     Version: v1.12.3
     Git commit: 06d12dec4030703cb46b2699ca434cdabd9edbb0
    velero version
    

2. Install Red Hat OpenShift APIs for Data Protection (OADP)

For more information, see the Red Hat OpenShift documentation About installing OADP Opens in a new tab.

As a convenience, if you are using S3 buckets on AWS, you can use the installation script install-oadp.sh in this section to simplify the process.

If you are using Red Hat® OpenShift® Data Foundation for your backup storage, follow the instructions in Configuring ODF as backup storage for OADP.

If you are not using Red Hat® OpenShift® Data Foundation for your backup storage, use the following steps:

  1. Clone the IBM Cloud Pak for AIOps samples GitHub repository that includes the backup and restore scripts. Clone this repository by running the following command:

    git clone https://github.com/IBM/cp4waiops-samples.git
    

    If you have an offline (air-gapped) deployment of IBM Cloud Pak for AIOps, then copy the cloned scripts to your air-gapped environment.

  2. Update the installation files.

    1. Change to the install directory where the backup and restore scripts are downloaded.

      cd <Path>/bcdr/4.9.0/install
      

      Where <Path> is the path to where you cloned the IBM Cloud Pak for AIOps backup and restore scripts.

    2. Edit the install-oadp-config.json script. Configure the following settings:

      • aws_access_key_id - Access key ID to connect to your object storage S3 bucket.
      • aws_secret_access_key - Secret access key to connect to your S3 bucket.
      • bucket_name - Name of the S3 bucket where backup data will be stored.
      • bucket_region - Region where the S3 bucket is deployed.
      • namespace - Namespace name in which you want to deploy OADP. If the namespace does not exist, it is created when you run the OADP installation script.
      • backup_label - Label for creating a folder structure to organize sets of backups together.
    3. Edit the install-oadp.sh script.

      If your version of Red Hat OpenShift is higher than v4.15, update channel: stable-1.3 to channel: stable-1.4.

      Example excerpt:

      spec:
        channel: stable-1.4
        installPlanApproval: Automatic
        name: redhat-oadp-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      

  3. Run the OADP installation script:

    nohup ./install-oadp.sh > install-oadp.log &
    
  4. Check the logs by running the following command:

    tail -f install-oadp.log
    
  5. Export an environment variable with the Velero namespace.

    export OADP_NAMESPACE=<oadpNamespace>
    

    Where <oadpNamespace> is the namespace where OADP is installed.

  6. Check the Velero pods status by running the following command:

    oc get pods -n ${OADP_NAMESPACE}
    

    The Velero and Restic pods should be in a running state.

  7. Check the status of the backupStorageLocation by running the following command:

    oc get backupStorageLocation -n ${OADP_NAMESPACE}
    

    The backupStorageLocation needs to be in an available status.

3. (Offline deployments only) Mirror the backup and restore image

If the deployment that you are backing up or restoring to is an offline (air-gapped) deployment, then you must copy the backup and restore image to a registry that is available to your air-gapped environment. You can do this by using either a bastion host or a portable device. For background information on offline (air-gapped) deployments, see Installing IBM Cloud Pak for AIOps in an air-gapped environment (offline).

Prerequisites:

  • The host that you are running the mirror from must allow access to icr.io on port 443.
  • You must have a target registry available to mirror the backup and restore image into. The target registry must adhere to the requirements in Set up a target registry.

Follow the instructions in Mirroring with a bastion host or Mirroring with a portable device.

Mirroring with a bastion host

  1. Set environment variables.

    export TARGET_REGISTRY_HOST=<IP_or_FQDN_of_target_registry>
    export TARGET_REGISTRY_PORT=<port_number_of_target_registry>
    export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT
    export TARGET_REGISTRY_USER=<username>
    export TARGET_REGISTRY_PASSWORD=<password>
    export BCDR_IMAGE=<bcdr_image>
    export BCDR_UTILS_IMAGE=cp4waiops-bcdr-utils@sha256:1db594ad10da9c7049367b5dd4373e86af7e6bcb875758022e2e743c83cf486f
    export AIOPS_NAMESPACE=<aiops_namespace>
    

    Where

    • <bcdr_image> is the name of the backup and restore image, as given in the backup helm chart values.yaml file, in the form cp4waiops-bcdr@{digest}. An example value for BCDR_IMAGE is cp4waiops-bcdr@sha256:294a42a851a2717ebbc68528ab3c6bcb1ba48114ff058f1c1b537dc6aa167355.
    • <aiops_namespace> is the namespace that IBM Cloud Pak for AIOps is deployed in.

    If your bastion host must connect to the internet through a proxy, then also set the following environment variables:

    export https_proxy=http://proxy-server-hostname:port
    export http_proxy=http://proxy-server-hostname:port
    
  2. Authenticate with the target registry.

    You do not need to authenticate with the IBM Entitled Registry, as the image that you require for backup and restore is freely available.

    You need to authenticate with the target registry to enable the oc image mirror command to push the image to your target registry. Log in to your target registry to generate an authentication file containing the registry credentials, and then create an environment variable that has the location of the authentication file.

    Run the following commands:

    podman login ${TARGET_REGISTRY} -u ${TARGET_REGISTRY_USER} -p ${TARGET_REGISTRY_PASSWORD}
    export REGISTRY_AUTH_FILE=${XDG_RUNTIME_DIR}/containers/auth.json
    

    Note: The authentication file is usually at ${XDG_RUNTIME_DIR}/containers/auth.json. For more information, see the Options section in the Podman documentationOpens in a new tab.

  3. Mirror the image.

    Connect the bastion host to the internet and disconnect if from the air-gapped environment. Then run the following command to mirror the backup and restore image to the target registry:

    oc image mirror icr.io/cpopen/cp4waiops/${BCDR_IMAGE} ${TARGET_REGISTRY}/cpopen/cp4waiops/${BCDR_IMAGE} -a ${REGISTRY_AUTH_FILE} --insecure
    oc image mirror icr.io/cpopen/cp4waiops/${BCDR_UTILS_IMAGE} ${TARGET_REGISTRY}/cpopen/cp4waiops/${BCDR_UTILS_IMAGE} -a ${REGISTRY_AUTH_FILE} --insecure
    
  4. Create the ImageContentSoucePolicy.

    1. Switch to the namespace that IBM Cloud Pak for AIOps is deployed in.

      oc project ${AIOPS_NAMESPACE}
      
    2. Run the following command to create the ImageContentSourcePolicy.

      cat << EOF | oc apply -f -
      apiVersion: operator.openshift.io/v1alpha1
      kind: ImageContentSourcePolicy
      metadata:
       name: bcdr-mirrors
      spec:
       repositoryDigestMirrors:
        - mirrors:
          - ${TARGET_REGISTRY}/cpopen/cp4waiops/
          source: icr.io/cpopen/cp4waiops/
      EOF
      

Mirroring with a portable device

If you are using a portable storage device, then connect it to a host with access to the internet. If you are using a portable compute device, then connect it to the internet. Run the following steps.

  1. Set environment variables.

    export IMAGE_PATH=<image-path>
    export BCDR_IMAGE=<bcdr_image>
    export BCDR_UTILS_IMAGE=cp4waiops-bcdr-utils@sha256:1db594ad10da9c7049367b5dd4373e86af7e6bcb875758022e2e743c83cf486f
    

    Where

    • <image_path> is the directory where you want the image to be stored.
    • <bcdr_image> is the name of the backup and restore image, as given in the backup helm chart values.yaml file, in the form cp4waiops-bcdr@{digest}. An example value for BCDR_IMAGE is cp4waiops-bcdr@sha256:294a42a851a2717ebbc68528ab3c6bcb1ba48114ff058f1c1b537dc6aa167355.

    If your device must connect to the internet through a proxy, then also set the following environment variables:

    export https_proxy=http://proxy-server-hostname:port
    export http_proxy=http://proxy-server-hostname:port
    
  2. Mirror the images to the file system on your portable device.

    You do not need to authenticate with the IBM Entitled Registry, as the image that you require for backup and restore is freely available. Run the following command to mirror the image from the IBM Entitled Registry to the file system on your portable device.

    oc image mirror \
    icr.io/cpopen/cp4waiops/${BCDR_IMAGE} file://cpopen/cp4waiops/${BCDR_IMAGE} \
    --filter-by-os '.*' \
    --insecure \
    --skip-multiple-scopes \
    --dir "${IMAGE_PATH}" \
    --max-per-registry=1
    
    oc image mirror \
    icr.io/cpopen/cp4waiops/${BCDR_UTILS_IMAGE} file://cpopen/cp4waiops/${BCDR_UTILS_IMAGE} \
    --filter-by-os '.*' \
    --insecure \
    --skip-multiple-scopes \
    --dir "${IMAGE_PATH}" \
    --max-per-registry=1
    
  3. Setup the file system in the air-gapped environment.

    1. (Portable storage device only) Copy files to the air-gapped environment

      If you are using a portable storage device, you must copy the files at $IMAGE_PATH on the portable storage device to a local compute device in the air-gapped environment that has access to the target registry. If you are using a portable compute device, then these files are already present, and you can proceed to the next step.

    2. Disconnect the device that has your file system (the portable compute device or the local compute device) from the internet and connect it to the air-gapped environment.

    3. Set environment variables on the device in the air-gapped environment that has access to the target registry.

      If you are using a portable compute device, then set the following environment variables on it. If you are using a portable storage device, then set the following environment variables on your local compute device within the air-gapped environment.

      export TARGET_REGISTRY_HOST=<IP_or_FQDN_of_target_registry>
      export TARGET_REGISTRY_PORT=<port_number_of_target_registry>
      export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT
      export TARGET_REGISTRY_USER=<username>
      export TARGET_REGISTRY_PASSWORD=<password>
      export IMAGE_PATH=<image_path>
      export BCDR_IMAGE=<bcdr_image>
      export BCDR_UTILS_IMAGE=cp4waiops-bcdr-utils@sha256:1db594ad10da9c7049367b5dd4373e86af7e6bcb875758022e2e743c83cf486f
      export AIOPS_NAMESPACE=<aiops_namespace>
      

      Where

      • <bcdr_image> is the name of the backup and restore image, as given in the backup helm chart values.yaml file, in the form cp4waiops-bcdr@{digest}. An example value for BCDR_IMAGE is cp4waiops-bcdr@sha256:294a42a851a2717ebbc68528ab3c6bcb1ba48114ff058f1c1b537dc6aa167355.
      • <aiops_namespace> is the namespace that IBM Cloud Pak for AIOps is deployed in.
  4. Authenticate with the target registry.

    Authenticate with the target registry in the air-gapped environment that you are mirroring the image into. You need to authenticate with the target registry to enable the oc image mirror command to push the image to your target registry. Log in to your target registry to generate an authentication file containing the registry credentials, and then create an environment variable that has the location of the authentication file.

    Run the following commands:

    podman login ${TARGET_REGISTRY} -u ${TARGET_REGISTRY_USER} -p ${TARGET_REGISTRY_PASSWORD}
    export REGISTRY_AUTH_FILE=${XDG_RUNTIME_DIR}/containers/auth.json
    

    Note: The authentication file is usually at ${XDG_RUNTIME_DIR}/containers/auth.json. For more information, see the Options section in the Podman documentationOpens in a new tab.

  5. Mirror the images to the target registry from the file system.

    Run the following command on the device that has your file system (the portable compute device or the local compute device) to copy the image from the file system to the $TARGET_REGISTRY. Your device with the file system must be connected to both the target registry and the Red Hat® OpenShift® Container Platform cluster that you want to backup or restore IBM Cloud Pak for AIOps on.

    oc image mirror \
    file://cpopen/cp4waiops/${BCDR_IMAGE} ${TARGET_REGISTRY}/cpopen/cp4waiops/${BCDR_IMAGE} \
    --from-dir "${IMAGE_PATH}" \
    -a ${REGISTRY_AUTH_FILE} \
    --filter-by-os '.*' \
    --insecure  \
    --skip-multiple-scopes \
    --max-per-registry=1
    
    oc image mirror \
    file://cpopen/cp4waiops/${BCDR_UTILS_IMAGE} ${TARGET_REGISTRY}/cpopen/cp4waiops/${BCDR_UTILS_IMAGE} \
    --from-dir "${IMAGE_PATH}" \
    -a ${REGISTRY_AUTH_FILE} \
    --filter-by-os '.*' \
    --insecure  \
    --skip-multiple-scopes \
    --max-per-registry=1
    
  6. Create the ImageContentSoucePolicy.

    1. Switch to the namespace that IBM Cloud Pak for AIOpsis deployed in.

      oc project ${AIOPS_NAMESPACE}
      
    2. Run the following command to create the ImageContentSourcePolicy.

      cat << EOF | oc apply -f -
      apiVersion: operator.openshift.io/v1alpha1
      kind: ImageContentSourcePolicy
      metadata:
       name: bcdr-mirrors
      spec:
       repositoryDigestMirrors:
        - mirrors:
          - ${TARGET_REGISTRY}/cpopen/cp4waiops/
          source: icr.io/cpopen/cp4waiops/
      EOF
      

Troubleshooting

Red Hat OpenShift APIs for Data Protection (OADP) installation failed

If your installation of OADP fails for any reason, delete any related files and reattempt the installation.

For example, the installation script install-oadp.sh might not complete successfully. If this occurs an error similar to the following message can display:

error: unable to recognize "STDIN": no matches for kind "DataProtectionApplication" in version "oadp.openshift.io/v1alpha1"

This can occur if the network or cluster is running slow and taking too long to pull the required images. This delay causes the OADP operator to take too long to create the required custom resource.

If the preceding issue occurs, or the installation fails for another reason, run the following oc commands to delete any existing OADP installation files before attempting to install again.

  1. Delete any cluster service versions (CSV) in the project:

    oc delete csv --all -n ${OADP_NAMESPACE}
    
  2. Delete any subscriptions in the project:

    oc delete subscription --all -n ${OADP_NAMESPACE}
    
  3. Delete any deployments in the project:

    oc delete deploy --all -n ${OADP_NAMESPACE}
    
  4. Delete any daemonsets in the project:

    oc delete ds --all -n ${OADP_NAMESPACE}
    
  5. Delete the project (namespace):

    oc delete project ${OADP_NAMESPACE}
    

Any existing installation is deleted and you can reattempt the installation.

Backups are not showing after installation

If your backups do not show, complete the following steps:

  1. Correct the bucket and prefix property within the backupstoragelocation resource that is named bcdr-s3-location. The bucket and prefix name should match the value that is defined within the install-oadp-config.json file during the OADP installation.

    Run the oc edit command to edit the backupstoragelocation resource:

    oc edit backupstoragelocation bcdr-s3-location -n ${OADP_NAMESPACE}
    
  2. Delete the Velero pod by running the following command:

    oc delete pod <pod> -n ${OADP_NAMESPACE}
    

    The backups should now be available.

  3. Run the following command to list the backups:

    velero get backup