OADP online backup partially fails with timeout error

Creating an online backup with the OADP backup and restore utility partially fails due to a timeout error.

Symptoms

You see error messages such as in the following examples.

Output from the cpd-cli oadp backup status command

 message: /Timed out awaiting reconciliation of volumesnapshot zen/velero-data-c-db2oltp-wkc-db2u-0-6kk7p
 name: /data-c-db2oltp-wkc-db2u-0 message: /Error backing up item error: /error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, 
namespace=${PROJECT_CPD_INST_OPERANDS}, name=velero-data-c-db2oltp-wkc-db2u-0-6kk7p): 
rpc error: code = Unknown desc = failed to get volumesnapshot ${PROJECT_CPD_INST_OPERANDS}/velero-data-c-db2oltp-wkc-db2u-0-6kk7p: 
client rate limiter Wait returned an error: context deadline exceeded

From the Velero pod log or the Velero backup and restore log

time="<timestamp>" level=error msg="Error backing up item" backup=oadp-operator/bkpcpds4851-tenant-online-b1 
error="error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=${PROJECT_CPD_INST_OPERANDS}, 
name=velero-zen-metastore-edb-1-cjxlh): rpc error: code = Unknown desc = timed out waiting for the condition" logSource="/remote-source/velero/app/pkg/backup/backup.go:448" name=zen-metastore-edb-1

Resolving the problem

Rerun the cpd-cli oadp backup create command with the --csi-snapshot-timeout option set to an appropriate duration to make sure that the backup process will successfully complete. For example:

cpd-cli oadp backup create <volume-backup-name> \
--tenant-operator-namespace ${PROJECT_CPD_INST_OPERATORS} \
--include-resources='namespaces,persistentvolumeclaims,persistentvolumes,volumesnapshot,volumesnapshotcontent,customresourcedefinitions.apiextensions.k8s.io' \
--selector='icpdsupport/empty-on-nd-backup notin (true),icpdsupport/ignore-on-nd-backup notin (true)' \ \
--hook-kind=checkpoint \
--snapshot-volumes \
--csi-snapshot-timeout=30m \
--log-level=debug \
--verbose