Installing the backup and restore tools
Learn how to install the tools that are required for the backup and restore of IBM Cloud Pak for AIOps.
You need to install the Red Hat OpenShift APIs for Data Protection (OADP). OADP provides default Velero plug-ins that are integrated with storage providers to support backup operations.
Note: The backup and restore tools for IBM Cloud Pak for AIOps are the same as the tools for backing up and restoring Infrastructure Automation. If you have the tools installed for Infrastructure Automation, you do not need to complete the following procedure.
Prerequisites
- Provision the object storage (S3 bucket) for storing your backups. The backup process moves backup files to an S3 bucket, which must be in an S3 compatible location, such as Amazon Web Services (AWS).
- Ensure that the following command line tools are installed on your workstation:
oc
jq
git
- Ensure that you are logged in to your Red Hat® OpenShift® Container Platform cluster with
oc login
for any steps that use the Red Hat OpenShift command-line interface (CLI).
Procedure
1. Install the Velero client CLI
Install the Velero client CLI on the workstation that you plan to use to run the scripts for backing up and restoring the cluster.
-
Run the following command to download the Velero CLI client:
curl -LO https://github.com/vmware-tanzu/velero/releases/download/v1.12.3/velero-v1.12.3-linux-amd64.tar.gz
-
Extract the tarball:
tar -xzf velero-v1.12.3-linux-amd64.tar.gz
-
Move the extracted velero binary to
/usr/local/bin
:mv velero-v1.12.3-linux-amd64/velero /usr/local/bin/velero
-
Run the following command to verify the installation.
velero version
Example output:
Client: Version: v1.12.3 Git commit: 06d12dec4030703cb46b2699ca434cdabd9edbb0 velero version
2. Install Red Hat OpenShift APIs for Data Protection (OADP)
For more information, see the Red Hat OpenShift documentation About installing OADP .
As a convenience, if you are using S3 buckets on AWS, you can use the installation script install-oadp.sh
in this section to simplify the process.
If you are using Red Hat® OpenShift® Data Foundation for your backup storage, follow the instructions in Configuring ODF as backup storage for OADP.
If you are not using Red Hat® OpenShift® Data Foundation for your backup storage, use the following steps:
-
Clone the IBM Cloud Pak for AIOps samples GitHub repository that includes the backup and restore scripts. Clone this repository by running the following command:
git clone https://github.com/IBM/cp4waiops-samples.git
If you have an offline (air-gapped) deployment of IBM Cloud Pak for AIOps, then copy the cloned scripts to your air-gapped environment.
-
Update the installation files.
-
Change to the
install
directory where the backup and restore scripts are downloaded.cd <Path>/bcdr/4.9.0/install
Where
<Path>
is the path to where you cloned the IBM Cloud Pak for AIOps backup and restore scripts. -
Edit the
install-oadp-config.json
script. Configure the following settings:aws_access_key_id
- Access key ID to connect to your object storage S3 bucket.aws_secret_access_key
- Secret access key to connect to your S3 bucket.bucket_name
- Name of the S3 bucket where backup data will be stored.bucket_region
- Region where the S3 bucket is deployed.namespace
- Namespace name in which you want to deploy OADP. If the namespace does not exist, it is created when you run the OADP installation script.backup_label
- Label for creating a folder structure to organize sets of backups together.
-
Edit the
install-oadp.sh
script.If your version of Red Hat OpenShift is higher than v4.15, update
channel: stable-1.3
tochannel: stable-1.4
.Example excerpt:
spec: channel: stable-1.4 installPlanApproval: Automatic name: redhat-oadp-operator source: redhat-operators sourceNamespace: openshift-marketplace
-
-
Run the OADP installation script:
nohup ./install-oadp.sh > install-oadp.log &
-
Check the logs by running the following command:
tail -f install-oadp.log
-
Export an environment variable with the Velero namespace.
export OADP_NAMESPACE=<oadpNamespace>
Where
<oadpNamespace>
is the namespace where OADP is installed. -
Check the Velero pods status by running the following command:
oc get pods -n ${OADP_NAMESPACE}
The Velero and Restic pods should be in a running state.
-
Check the status of the
backupStorageLocation
by running the following command:oc get backupStorageLocation -n ${OADP_NAMESPACE}
The
backupStorageLocation
needs to be in an available status.
3. (Offline deployments only) Mirror the backup and restore image
If the deployment that you are backing up or restoring to is an offline (air-gapped) deployment, then you must copy the backup and restore image to a registry that is available to your air-gapped environment. You can do this by using either a bastion host or a portable device. For background information on offline (air-gapped) deployments, see Installing IBM Cloud Pak for AIOps in an air-gapped environment (offline).
Prerequisites:
- The host that you are running the mirror from must allow access to
icr.io
on port 443. - You must have a target registry available to mirror the backup and restore image into. The target registry must adhere to the requirements in Set up a target registry.
Follow the instructions in Mirroring with a bastion host or Mirroring with a portable device.
Mirroring with a bastion host
-
Set environment variables.
export TARGET_REGISTRY_HOST=<IP_or_FQDN_of_target_registry> export TARGET_REGISTRY_PORT=<port_number_of_target_registry> export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT export TARGET_REGISTRY_USER=<username> export TARGET_REGISTRY_PASSWORD=<password> export BCDR_IMAGE=<bcdr_image> export BCDR_UTILS_IMAGE=cp4waiops-bcdr-utils@sha256:1db594ad10da9c7049367b5dd4373e86af7e6bcb875758022e2e743c83cf486f export AIOPS_NAMESPACE=<aiops_namespace>
Where
<bcdr_image>
is the name of the backup and restore image, as given in the backup helm chartvalues.yaml
file, in the formcp4waiops-bcdr@{digest}
. An example value for BCDR_IMAGE iscp4waiops-bcdr@sha256:294a42a851a2717ebbc68528ab3c6bcb1ba48114ff058f1c1b537dc6aa167355
.<aiops_namespace>
is the namespace that IBM Cloud Pak for AIOps is deployed in.
If your bastion host must connect to the internet through a proxy, then also set the following environment variables:
export https_proxy=http://proxy-server-hostname:port export http_proxy=http://proxy-server-hostname:port
-
Authenticate with the target registry.
You do not need to authenticate with the IBM Entitled Registry, as the image that you require for backup and restore is freely available.
You need to authenticate with the target registry to enable the
oc image mirror
command to push the image to your target registry. Log in to your target registry to generate an authentication file containing the registry credentials, and then create an environment variable that has the location of the authentication file.Run the following commands:
podman login ${TARGET_REGISTRY} -u ${TARGET_REGISTRY_USER} -p ${TARGET_REGISTRY_PASSWORD} export REGISTRY_AUTH_FILE=${XDG_RUNTIME_DIR}/containers/auth.json
Note: The authentication file is usually at
${XDG_RUNTIME_DIR}/containers/auth.json
. For more information, see the Options section in the Podman documentation.
-
Mirror the image.
Connect the bastion host to the internet and disconnect if from the air-gapped environment. Then run the following command to mirror the backup and restore image to the target registry:
oc image mirror icr.io/cpopen/cp4waiops/${BCDR_IMAGE} ${TARGET_REGISTRY}/cpopen/cp4waiops/${BCDR_IMAGE} -a ${REGISTRY_AUTH_FILE} --insecure oc image mirror icr.io/cpopen/cp4waiops/${BCDR_UTILS_IMAGE} ${TARGET_REGISTRY}/cpopen/cp4waiops/${BCDR_UTILS_IMAGE} -a ${REGISTRY_AUTH_FILE} --insecure
-
Create the
ImageContentSoucePolicy
.-
Switch to the namespace that IBM Cloud Pak for AIOps is deployed in.
oc project ${AIOPS_NAMESPACE}
-
Run the following command to create the
ImageContentSourcePolicy
.cat << EOF | oc apply -f - apiVersion: operator.openshift.io/v1alpha1 kind: ImageContentSourcePolicy metadata: name: bcdr-mirrors spec: repositoryDigestMirrors: - mirrors: - ${TARGET_REGISTRY}/cpopen/cp4waiops/ source: icr.io/cpopen/cp4waiops/ EOF
-
Mirroring with a portable device
If you are using a portable storage device, then connect it to a host with access to the internet. If you are using a portable compute device, then connect it to the internet. Run the following steps.
-
Set environment variables.
export IMAGE_PATH=<image-path> export BCDR_IMAGE=<bcdr_image> export BCDR_UTILS_IMAGE=cp4waiops-bcdr-utils@sha256:1db594ad10da9c7049367b5dd4373e86af7e6bcb875758022e2e743c83cf486f
Where
<image_path>
is the directory where you want the image to be stored.<bcdr_image>
is the name of the backup and restore image, as given in the backup helm chartvalues.yaml
file, in the formcp4waiops-bcdr@{digest}
. An example value for BCDR_IMAGE iscp4waiops-bcdr@sha256:294a42a851a2717ebbc68528ab3c6bcb1ba48114ff058f1c1b537dc6aa167355
.
If your device must connect to the internet through a proxy, then also set the following environment variables:
export https_proxy=http://proxy-server-hostname:port export http_proxy=http://proxy-server-hostname:port
-
Mirror the images to the file system on your portable device.
You do not need to authenticate with the IBM Entitled Registry, as the image that you require for backup and restore is freely available. Run the following command to mirror the image from the IBM Entitled Registry to the file system on your portable device.
oc image mirror \ icr.io/cpopen/cp4waiops/${BCDR_IMAGE} file://cpopen/cp4waiops/${BCDR_IMAGE} \ --filter-by-os '.*' \ --insecure \ --skip-multiple-scopes \ --dir "${IMAGE_PATH}" \ --max-per-registry=1
oc image mirror \ icr.io/cpopen/cp4waiops/${BCDR_UTILS_IMAGE} file://cpopen/cp4waiops/${BCDR_UTILS_IMAGE} \ --filter-by-os '.*' \ --insecure \ --skip-multiple-scopes \ --dir "${IMAGE_PATH}" \ --max-per-registry=1
-
Setup the file system in the air-gapped environment.
-
(Portable storage device only) Copy files to the air-gapped environment
If you are using a portable storage device, you must copy the files at $IMAGE_PATH on the portable storage device to a local compute device in the air-gapped environment that has access to the target registry. If you are using a portable compute device, then these files are already present, and you can proceed to the next step.
-
Disconnect the device that has your file system (the portable compute device or the local compute device) from the internet and connect it to the air-gapped environment.
-
Set environment variables on the device in the air-gapped environment that has access to the target registry.
If you are using a portable compute device, then set the following environment variables on it. If you are using a portable storage device, then set the following environment variables on your local compute device within the air-gapped environment.
export TARGET_REGISTRY_HOST=<IP_or_FQDN_of_target_registry> export TARGET_REGISTRY_PORT=<port_number_of_target_registry> export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT export TARGET_REGISTRY_USER=<username> export TARGET_REGISTRY_PASSWORD=<password> export IMAGE_PATH=<image_path> export BCDR_IMAGE=<bcdr_image> export BCDR_UTILS_IMAGE=cp4waiops-bcdr-utils@sha256:1db594ad10da9c7049367b5dd4373e86af7e6bcb875758022e2e743c83cf486f export AIOPS_NAMESPACE=<aiops_namespace>
Where
<bcdr_image>
is the name of the backup and restore image, as given in the backup helm chartvalues.yaml
file, in the formcp4waiops-bcdr@{digest}
. An example value for BCDR_IMAGE iscp4waiops-bcdr@sha256:294a42a851a2717ebbc68528ab3c6bcb1ba48114ff058f1c1b537dc6aa167355
.<aiops_namespace>
is the namespace that IBM Cloud Pak for AIOps is deployed in.
-
-
Authenticate with the target registry.
Authenticate with the target registry in the air-gapped environment that you are mirroring the image into. You need to authenticate with the target registry to enable the
oc image mirror
command to push the image to your target registry. Log in to your target registry to generate an authentication file containing the registry credentials, and then create an environment variable that has the location of the authentication file.Run the following commands:
podman login ${TARGET_REGISTRY} -u ${TARGET_REGISTRY_USER} -p ${TARGET_REGISTRY_PASSWORD} export REGISTRY_AUTH_FILE=${XDG_RUNTIME_DIR}/containers/auth.json
Note: The authentication file is usually at
${XDG_RUNTIME_DIR}/containers/auth.json
. For more information, see the Options section in the Podman documentation.
-
Mirror the images to the target registry from the file system.
Run the following command on the device that has your file system (the portable compute device or the local compute device) to copy the image from the file system to the $TARGET_REGISTRY. Your device with the file system must be connected to both the target registry and the Red Hat® OpenShift® Container Platform cluster that you want to backup or restore IBM Cloud Pak for AIOps on.
oc image mirror \ file://cpopen/cp4waiops/${BCDR_IMAGE} ${TARGET_REGISTRY}/cpopen/cp4waiops/${BCDR_IMAGE} \ --from-dir "${IMAGE_PATH}" \ -a ${REGISTRY_AUTH_FILE} \ --filter-by-os '.*' \ --insecure \ --skip-multiple-scopes \ --max-per-registry=1
oc image mirror \ file://cpopen/cp4waiops/${BCDR_UTILS_IMAGE} ${TARGET_REGISTRY}/cpopen/cp4waiops/${BCDR_UTILS_IMAGE} \ --from-dir "${IMAGE_PATH}" \ -a ${REGISTRY_AUTH_FILE} \ --filter-by-os '.*' \ --insecure \ --skip-multiple-scopes \ --max-per-registry=1
-
Create the
ImageContentSoucePolicy
.-
Switch to the namespace that IBM Cloud Pak for AIOpsis deployed in.
oc project ${AIOPS_NAMESPACE}
-
Run the following command to create the
ImageContentSourcePolicy
.cat << EOF | oc apply -f - apiVersion: operator.openshift.io/v1alpha1 kind: ImageContentSourcePolicy metadata: name: bcdr-mirrors spec: repositoryDigestMirrors: - mirrors: - ${TARGET_REGISTRY}/cpopen/cp4waiops/ source: icr.io/cpopen/cp4waiops/ EOF
-
Troubleshooting
- Red Hat OpenShift APIs for Data Protection (OADP) installation failed
- Backups are not showing after installation
Red Hat OpenShift APIs for Data Protection (OADP) installation failed
If your installation of OADP fails for any reason, delete any related files and reattempt the installation.
For example, the installation script install-oadp.sh
might not complete successfully. If this occurs an error similar to the following message can display:
error: unable to recognize "STDIN": no matches for kind "DataProtectionApplication" in version "oadp.openshift.io/v1alpha1"
This can occur if the network or cluster is running slow and taking too long to pull the required images. This delay causes the OADP operator to take too long to create the required custom resource.
If the preceding issue occurs, or the installation fails for another reason, run the following oc
commands to delete any existing OADP installation files before attempting to install again.
-
Delete any cluster service versions (CSV) in the project:
oc delete csv --all -n ${OADP_NAMESPACE}
-
Delete any subscriptions in the project:
oc delete subscription --all -n ${OADP_NAMESPACE}
-
Delete any deployments in the project:
oc delete deploy --all -n ${OADP_NAMESPACE}
-
Delete any daemonsets in the project:
oc delete ds --all -n ${OADP_NAMESPACE}
-
Delete the project (namespace):
oc delete project ${OADP_NAMESPACE}
Any existing installation is deleted and you can reattempt the installation.
Backups are not showing after installation
If your backups do not show, complete the following steps:
-
Correct the
bucket
andprefix
property within thebackupstoragelocation
resource that is namedbcdr-s3-location
. Thebucket
andprefix
name should match the value that is defined within theinstall-oadp-config.json
file during the OADP installation.Run the
oc edit
command to edit thebackupstoragelocation
resource:oc edit backupstoragelocation bcdr-s3-location -n ${OADP_NAMESPACE}
-
Delete the Velero pod by running the following command:
oc delete pod <pod> -n ${OADP_NAMESPACE}
The backups should now be available.
-
Run the following command to list the backups:
velero get backup