IBM Cloud Pak foundational services backup and restore for Helm chart deployments

Edit online

Learn how to back up and restore foundational services that you installed without Operator Lifecycle Manager (OLM) with Helm chart. You can schedule backup and restore of foundational services by using the Red Hat OpenShift API for Data Protection (OADP) operator. Make sure that you use the stable-1.3 channel of the OADP operator.

Prerequisites

Edit online

If you plan to restore foundational services on a new cluster and currently use an ImageContentSourcePolicy resource, reconfigure this resource on the new cluster. For more information, see Configure the cluster.
Set up any Amazon S3-compatible storage. For example, you can create a bucket in IBM Cloud Object Storage. For more information, see IBM Cloud Object Storage .
When you add a service credential to the bucket, include the hash-based message authentication code (HMAC). For more information, see Service credentials . From the Cloud Object Storage navigation menu, gather the following information:
- access key id, which can be found on the Service credentials page that is associated with the bucket.
- secret access key, which can be found on the Service credentials page that is associated with the bucket.
- bucket name, which can be found on the Buckets page.
- bucket region, which can be found on the Buckets page.
- root directory name, which is the path in the bucket where you want to store the backups.
- s3 URL, which is the endpoint URL of the bucket.
  - The endpoint URL must start with http:// or https://.

Note: If the cluster being backed up or restored to uses s390x for architecture, any velero CLI commands must be run on an alternate cluster that does not use s390x and has oc access to the original (usually using oc login). Velero CLI does not yet support s390x.

Backup and restore for multiple instances of foundational services

Edit online

If you have multiple instances of foundational services that are deployed on a single cluster, create the backup resources and add labels to resources for each tenant.

Backing up foundational services

Edit online

Complete the following steps to back up the installed foundational services.

Create the backup resources

Edit online

You need the following resources for completing the backup procedures.

Log in to your OpenShift cluster command-line interface (CLI) by using the oc login command.
Create a namespace for Velero objects. The following example creates the velero namespace. For more information about Velero, see Velero documentation .
```
oc project velero
```
Install the Red Hat OADP operator in the velero namespace. For more information, see About installing OADP .
Create a secret named cloud-credentials with the access key id and secret access key credentials.
1. Open any editor and place the following credentials in a file named credentials-velero.
```
vi credentials-velero
```
2. Insert the following content in the file:
```
[default]
aws_access_key_id=<access_key_id>
aws_secret_access_key=<secret_access_key>
```
3. Create the secret.
```
oc create secret generic cloud-credentials -n velero --from-file cloud=credentials-velero
```
From your OpenShift cluster console OperatorHub page, install the OADP operator from the stable-1.3 channel, which provides the Velero 1.9 API. The API is needed for foundational services backup and restore. For more information, see OpenShift Container Platform documentation .

Create a DataProtectionApplication object.

Note: The provider is aws even if you are not using AWS Object Storage.

If you use Red Hat® OpenShift® Container Platform version 4.19 and onward, use the following DataProtectionApplication custom resource (CR).

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <resource_name>
  namespace: velero
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    argocd.argoproj.io/sync-wave: '20'
spec:
  backupLocations:
    - velero:
        config:
          profile: default
          region: <bucket_region>
          s3ForcePathStyle: 'true'
          s3Url: <s3_URL>
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: <bucket_name>
          prefix: <root_directory_name>
        provider: aws
  configuration:
    nodeAgent:
      enable: true
      uploaderType: kopia
    velero:
      defaultPlugins:
        - openshift
        - aws
      podConfig:
        resourceAllocations:
          limits:
            cpu: '1'
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi

If you use a version of OpenShift Container Platform before version 4.19, use the following DataProtectionApplication CR.

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <resource_name>
  namespace: velero
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    argocd.argoproj.io/sync-wave: '20'
spec:
  backupLocations:
    - velero:
        config:
          profile: default
          region: <bucket_region>
          s3ForcePathStyle: 'true'
          s3Url: <s3_URL>
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: <bucket_name>
          prefix: <root_directory_name>
        provider: aws
  configuration:
    restic:
      enable: true
    velero:
      defaultPlugins:
        - openshift
        - aws
      podConfig:
        resourceAllocations:
          limits:
            cpu: '1'
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi

Add labels to resources

Edit online

You can add labels to resources automatically by running the script or by manually adding labels. Complete one of the following procedures.

Note: If you choose to label resources automatically by running the script, make sure that all resources are labeled. You can compare the labels that are added by the script with the manual steps to make sure that all necessary labels are in place.

Labelling the resources automatically by running the script

Edit online

Run the following commands to fetch and download the env.properties file and the label-common-services.sh script and save them in the same folder.

wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/backup/common-service/label-common-service.sh
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/backup/common-service/env.properties

Open the env.properties file, edit the required variables and save the changes.

Note: The OPERATOR_NS="" variable must be properly set for the script to work. Other variables have default values. You can change these values to fit your environment.

vi env.properties

The env.properties file contains the following variables:

# Change the following values to match your environment
OPERATOR_NS="" # Pass the namespace where the cs operator is installed

# Pass the namespace where the cs services are installed
# Leave it as empty string if the services are installed in the same namespace as the operator
SERVICES_NS="" 
CONTROL_NS="" # Pass the control namespace if it is needed to be backed up

#Pass any additional namespaces in the tenant that are not the operator or services namespace. Comma delimited
TETHERED_NS=""

# Change to the namespace where cert-manager, licensing and LSR are installed
CERT_MANAGER_NAMESPACE="ibm-cert-manager" 
LICENSING_NAMESPACE="ibm-licensing"
LSR_NAMESPACE="ibm-lsr"

# enable labeling of cert manager, licensing, and LSR resources by changing value to 1
ENABLE_CERT_MANAGER=0
ENABLE_LICENSING=0
ENABLE_LSR=0

# Change to 1 to enable private catalog
ENABLE_PRIVATE_CATALOG=0

#change to 1 to label catalog sources in openshift-marketplace
ENABLE_DEFAULT_CS=0

# Add additional CatalogSources separated by comma which are withour ".spec.publisher: IBM"
# For example: "my-catalog,my-catalog2,my-catalog3"
ADDITIONAL_SOURCES=""

Use the following command to run the label-common-service.sh script.
```
./label-common-service.sh --no-olm
```
While running the script, you can pass the following flags: --operator-ns, --services-ns, --tethered-ns. To view a full list of options, run the folllowing command ./label-common-service.sh -h

Add labels to the Licensing service configmaps:

Find the licensing namespace:
```
oc get pods -A | grep licensing
```

Get the label-licensing-configmaps.sh file.

wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/backup/licensing/label-licensing-configmaps.sh

Add executable permission.
```
chmod +x label-licensing-configmaps.sh
```

Run the script.

./label-licensing-configmaps.sh <namespace from previous step>

Add labels to the Cert Manager resources:
1. Get the label-cert-manager.sh file.
```
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/backup/cert-manager/label-cert-manager.sh
```
2. Add executable permission.
```
chmod +x label-cert-manager.sh
```
3. Run the script.
```
./label-cert-manager.sh
```
Note: The label-cert-manager.sh script searches all namespaces for cert manager resources. The search might require elevated privileges to run.

Manually adding labels to resources

Edit online

Labelling the general foundational services resources

Edit online

Before you begin, set the namespace where you installed foundational services as the default namespace.

oc project <namespace-where-foundational-services-are-installed>

You need to label the currently installed resources to identify them during restoration.

If you use IBM License Service Reporter, see Backing up the License Service Reporter instance.

Add a label to the common-service-maps configmap in the kube-public namespace (if present):

oc label configmap common-service-maps -n kube-public foundationservices.cloudpak.ibm.com=configmap --overwrite=true

If using a custom hostname, custom TLS secret, or both, label the common-web-ui-config configmap in each namespace it is present:
```
oc label configmap common-web-ui-config foundationservices.cloudpak.ibm.com=configmap --overwrite=true -n <namespace present>
```
This configmap can be found by using the command:
```
oc get cm -A | grep common-web-ui-config
```
If using a custom hostname, custom TLS secret, or both, label the cs-onprem-tenant-config configmap in each namespace it is present:
```
oc label configmap cs-onprem-tenant-config foundationservices.cloudpak.ibm.com=configmap --overwrite=true -n <namespace present>
```
This configmap can be found by using the command:
```
oc get cm -A | grep cs-onprem-tenant-config
```
Add a label to the namespaces where you installed foundational services, the namespace where IBM Cert Manager is installed (default namespace is ibm-cert-manager), the namespace where IBM Licensing is installed (default namespace is ibm-licensing), and workload namespaces that use foundational services. It is possible there are multiple namespaces with foundational services installed. Make sure to label each of them.
1. Find the namespaces where the services are installed.
  - Find the namespace where IBM Cert Manager is installed:
```
oc get pods -A | grep cert-manager
```
  - Find the namespace where IBM Licensing is installed:
```
oc get pods -A | grep licensing
```
  - Determine the workload namespaces. First, check whether the common-service-maps configmap exists:
```
oc get cm common-service-maps -n kube-public
```
    If the common-service-maps configmap exists, make sure to label each namespace listed in requested-from-namespace and controlNamespace values:
```
oc get cm common-service-maps -n kube-public -o yaml
```
    If the common-service-maps configmap is not present, label each namespace that is using the common service instance for smoother restoration process.
2. Label the namespaces.
```
oc label namespace <namespace-where-foundational-services-is-installed> foundationservices.cloudpak.ibm.com=namespace --overwrite=true
oc label namespace <namespace-where-cert-manager-is-installed> foundationservices.cloudpak.ibm.com=namespace --overwrite=true
oc label namespace <namespace-where-licensing-is-installed> foundationservices.cloudpak.ibm.com=namespace --overwrite=true
oc label namespace <requested-from-namespace> foundationservices.cloudpak.ibm.com=namespace --overwrite=true
oc label namespace <controlNamespace> foundationservices.cloudpak.ibm.com=namespace --overwrite=true
```
Add a label to the common-service CR:
```
oc label commonservices common-service foundationservices.cloudpak.ibm.com=commonservice --overwrite=true
```
Note: If your cluster has more than one (that is, in SOD scenarios), labeling each of them will not negatively impact the restore process.

Add a label to the commonservices.operator.ibm.com customresourcedefinition (CRD):

oc label customresourcedefinition commonservices.operator.ibm.com foundationservices.cloudpak.ibm.com=crd --overwrite=true

Add a label to the entitlement secret, if you have one in your cluster:

Find all entitlement keys on the cluster:

oc get secret -A | grep ibm-entitlement-key

Label each entitlement key:

oc label secret ibm-entitlement-key foundationservices.cloudpak.ibm.com=entitlementkey --overwrite=true -n <namespace>

Add a label to the global pull secret, if you have one in your cluster:

oc label secret pull-secret -n openshift-config foundationservices.cloudpak.ibm.com=pull-secret --overwrite=true

Add a label to the OperandRequests:
1. Find operand requests to label:
```
oc get operandrequests -A
```
2. Label each OperandRequest:
```
oc label operandrequests <operand request name> foundationservices.cloudpak.ibm.com=operand --overwrite=true -n <namespace>
```
Note: Typically, foundational services OperandRequests are named common-service so any OperandRequest with this name should be labeled. However, there might be more OperandRequests to label other than the ones that are named common-service as Cloud Paks might name their operand requests something different. There are some that do not need to be labeled because they are created automatically when certain services are requested such as ibm-iam-request. There is no harm in labeling this request.

Identifying Helm release secrets

Edit online

In the next steps, you need to label the Helm release secrets for your resources. The secrets are located in the namespace where the Helm charts are deployed (release namespace) and their names follow the sh.helm.release.v1.${chart_release_name}.v1 pattern. Use one of the following methods to find your Helm release secrets:

Run the helm ls --all-namespaces command to list all Helm charts that are installed on the cluster with their namespaces. This is enough information to label the secrets.
Look at one of the resources that belong to a specific Helm chart and check the following annotations: helm.sh/release-namespace and helm.sh/release-name.
Run the oc get secrets -A | grep sh.helm.release.v1 command to list all Helm charts that are installed on the cluster. Label all secrets in reply. Exclude only the secrets that are not a part of the tenant which you mean you restore.

Labelling the cluster-scoped resources

Edit online

Add labels to ODLM resources:

oc label crds operandbindinfos.operator.ibm.com operandconfigs.operator.ibm.com operandregistries.operator.ibm.com operandrequests.operator.ibm.com operatorconfigs.operator.ibm.com -n <operator namespace> foundationservices.cloudpak.ibm.com=odlm-cluster

Add labels to foundational services operator:

oc label clusterrole ibm-common-service-operator-$OPERATOR_NS -n <operator namespace> foundationservices.cloudpak.ibm.com=cs-cluster
oc label clusterrolebinding ibm-common-service-operator-$OPERATOR_NS -n <operator namespace> foundationservices.cloudpak.ibm.com=cs-cluster

Label the foundational services operator Helm release secret if the cluster-scoped chart was installed via helm install and not helm template | oc apply. If a cluster-scoped chart is installed using a method different than "helm install", it does not have an associated Helm secret.

oc label secret <foundational services operator Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=cs-cluster

Add labels to IM resources:

oc label crd clients.oidc.security.ibm.com authentications.operator.ibm.com -n <operator namespace> foundationservices.cloudpak.ibm.com=iam-cluster
oc label clusterrole ibm-iam-operator-$OPERATOR_NS -n <operator namespace> foundationservices.cloudpak.ibm.com=iam-cluster
oc label clusterrolebinding ibm-iam-operator-$OPERATOR_NS -n <operator namespace> foundationservices.cloudpak.ibm.com=iam-cluster

Label the IM Helm release secret if the cluster-scoped chart was installed via helm install and not helm template | oc apply. If a cluster-scoped chart is installed using a method different than "helm install", it does not have an associated Helm secret.

oc label secret <IM Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=iam-cluster

Add labels to UI resources:

oc label crd commonwebuis.operators.ibm.com navconfigurations.foundation.ibm.com switcheritems.operators.ibm.com -n <operator namespace> foundationservices.cloudpak.ibm.com=ui-cluster

Add labels to EDB PostgreSQL resources:

oc label crd backups.postgresql.k8s.enterprisedb.io clusters.postgresql.k8s.enterprisedb.io poolers.postgresql.k8s.enterprisedb.io scheduledbackups.postgresql.k8s.enterprisedb.io clusterimagecatalogs.postgresql.k8s.enterprisedb.io imagecatalogs.postgresql.k8s.enterprisedb.io publications.postgresql.k8s.enterprisedb.io subscriptions.postgresql.k8s.enterprisedb.io databases.postgresql.k8s.enterprisedb.io -n <operator namespace> foundationservices.cloudpak.ibm.com=edb-cluster
oc label clusterrole postgresql-operator-controller-manager-$OPERATOR_NS -n <operator namespace> foundationservices.cloudpak.ibm.com=edb-cluster
oc label clusterrolebinding postgresql-operator-controller-manager-$OPERATOR_NS -n <operator namespace> foundationservices.cloudpak.ibm.com=edb-cluster
oc label validatingwebhookconfiguration postgresql-operator-validating-webhook-configuration -n <operator namespace> foundationservices.cloudpak.ibm.com=edb-cluster
oc label mutatingwebhookconfiguration postgresql-operator-mutating-webhook-configuration -n <operator namespace> foundationservices.cloudpak.ibm.com=edb-cluster

Label the EDB PostgreSQL Helm release secret if the cluster-scoped chart was installed via helm install and not helm template | oc apply. If a cluster-scoped chart is installed using a method different than "helm install", it does not have an associated Helm secret.

oc label secret <EDB Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=edb-cluster

Add labels to zen resources:

oc label crd zenservices.zen.cpd.ibm.com zenextensions.zen.cpd.ibm.com -n <operator namespace> foundationservices.cloudpak.ibm.com=zen-cluster
oc label clusterrole ibm-zen-operator-cluster-role -n <operator namespace> foundationservices.cloudpak.ibm.com=zen-cluster
oc label clusterrolebinding ibm-zen-operator-cluster-role-binding -n <operator namespace> foundationservices.cloudpak.ibm.com=zen-cluster

Label the zen Helm release secret if the cluster-scoped chart was installed via helm install and not helm template | oc apply. If a cluster-scoped chart is installed using a method different than "helm install", it does not have an associated Helm secret.

oc label secret <zen Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=zen-cluster

Add labels to namespace-scope (NSS) operator resources:

oc label clusterrole ibm-namespace-scope-operator-$OPERATOR_NS -n <operator namespace> foundationservices.cloudpak.ibm.com=nss-cluster
oc label clusterrolebinding ibm-namespace-scope-operator-$OPERATOR_NS -n <operator namespace> foundationservices.cloudpak.ibm.com=nss-cluster
oc label customresourcedefinition namespacescopes.operator.ibm.com -n <operator namespace> foundationservices.cloudpak.ibm.com=nss-cluster

Label the NSS Helm release secret if the cluster-scoped chart was installed via helm install and not helm template | oc apply. If a cluster-scoped chart is installed using a method different than "helm install", it does not have an associated Helm secret.

oc label secret <NSS Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=nss-cluster

Labelling the namespace-scoped resources

Edit online

Add labels to namespace-scope (NSS) operator resources:

oc label deployment ibm-namespace-scope-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=nss
oc label role ibm-namespace-scope-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=nss
oc label rolebinding ibm-namespace-scope-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=nss
oc label role nss-managed-role-from-$OPERATOR_NS -n <operator namespace> foundationservices.cloudpak.ibm.com=nss
oc label role nss-managed-role-from-$OPERATOR_NS -n <all tenant namespaces> foundationservices.cloudpak.ibm.com=nss
oc label role rolebinding nss-managed-role-from-$OPERATOR_NS -n <operator namespace> foundationservices.cloudpak.ibm.com=nss
oc label role rolebinding nss-managed-role-from-$OPERATOR_NS -n <all tenant namespaces> foundationservices.cloudpak.ibm.com=nss
oc label serviceaccount ibm-namespace-scope-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=nss
oc label namespacescopes.operator.ibm.com common-service -n <operator namespace> foundationservices.cloudpak.ibm.com=nss

Label the NSS Helm release secret:

oc label secret <NSS Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=nss

Add labels for ODLM operator:

oc label deploy operand-deployment-lifecycle-manager -n <operator namespace> foundationservices.cloudpak.ibm.com=odlm-chart
oc label serviceaccount operand-deployment-lifecycle-manager -n <operator namespace> foundationservices.cloudpak.ibm.com=odlm-chart
oc label role operand-deployment-lifecycle-manager -n <operator namespace> foundationservices.cloudpak.ibm.com=odlm-chart
oc label role operand-deployment-lifecycle-manager -n <services namespace> foundationservices.cloudpak.ibm.com=odlm-chart
oc label role operand-deployment-lifecycle-manager -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=odlm-chart
oc label rolebinding operand-deployment-lifecycle-manager -n <operator namespace> foundationservices.cloudpak.ibm.com=odlm-chart
oc label rolebinding operand-deployment-lifecycle-manager -n <services namespace> foundationservices.cloudpak.ibm.com=odlm-chart
oc label rolebinding operand-deployment-lifecycle-manager -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=odlm-chart

Label the common Helm release secret for ODLM and foundational services operator:

oc label secret <ODLM and foundational services Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=odlm-chart

Add labels for foundational services operator:

oc label deployment ibm-common-service-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=cs-chart
oc label serviceaccount ibm-common-service-operator -n <operator namespacev foundationservices.cloudpak.ibm.com=cs-chart
oc label role ibm-common-service-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=cs-chart
oc label role ibm-common-service-operator -n <services namespace> foundationservices.cloudpak.ibm.com=cs-chart
oc label role ibm-common-service-operator -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=cs-chart
oc label rolebinding ibm-common-service-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=cs-chart
oc label rolebinding ibm-common-service-operator -n <services namespace> foundationservices.cloudpak.ibm.com=cs-chart
oc label rolebinding ibm-common-service-operator -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=cs-chart

Add labels for IM operator

oc label deployment ibm-iam-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=iam-chart
oc label serviceaccount ibm-iam-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=iam-chart
oc label role ibm-iam-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=iam-chart
oc label role ibm-iam-operator -n <services namespace> foundationservices.cloudpak.ibm.com=iam-chart
oc label role ibm-iam-operator -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=iam-chart
oc label rolebinding ibm-iam-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=iam-chart
oc label rolebinding ibm-iam-operator -n <services namespace> foundationservices.cloudpak.ibm.com=iam-chart
oc label rolebinding ibm-iam-operator -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=iam-chart

Label the common Helm release secret for IM and UI operator:

oc label secret <IM and UI operator Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=iam-chart

Add labels for the UI operator:

oc label deployment ibm-commonui-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=ui-chart
oc label serviceaccount ibm-commonui-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=ui-chart
oc label role ibm-commonui-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=ui-chart
oc label role ibm-commonui-operator -n <services namespace> foundationservices.cloudpak.ibm.com=ui-chart
oc label role ibm-commonui-operator -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=ui-chart
oc label rolebinding ibm-commonui-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=ui-chart
oc label rolebinding ibm-commonui-operator -n <services namespace> foundationservices.cloudpak.ibm.com=ui-chart
oc label rolebinding ibm-commonui-operator -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=ui-chart

Add labels for the EDB PostgreSQL operator:

Run the following command to get the name of the EDB PostgreSQL deployment: deploy=$(oc get deploy -n $OPERATOR_NS | grep postgresql-operator-controller-manager | awk '{print $1}’).

oc label deployment $deploy -n <operator namespace> label foundationservices.cloudpak.ibm.com=edb-chart
oc label configmap cloud-native-postgresql-image-list postgresql-operator-default-monitoring -n <operator namespace> foundationservices.cloudpak.ibm.com=edb-chart
oc label service postgresql-operator-webhook-service -n <operator namespace> foundationservices.cloudpak.ibm.com=edb-chart
oc label serviceaccount postgresql-operator-manager -n <operator namespace> foundationservices.cloudpak.ibm.com=edb-chart
oc label role postgresql-operator-controller-manager -n <operator namespace> foundationservices.cloudpak.ibm.com=edb-chart
oc label role postgresql-operator-controller-manager -n <services namespace> foundationservices.cloudpak.ibm.com=edb-chart
oc label role postgresql-operator-controller-manager -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=edb-chart
oc label rolebinding postgresql-operator-controller-manager -n <operator namespace> foundationservices.cloudpak.ibm.com=edb-chart
oc label rolebinding postgresql-operator-controller-manager -n <services namespace> foundationservices.cloudpak.ibm.com=edb-chart
oc label rolebinding postgresql-operator-controller-manager -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=edb-chart

Label the EDB PostgreSQL Helm release secret:

oc label secret <EDB PostgreSQL Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=edb-chart

Add labels for the zen operator:

Get the name of the zenservice deployment name and namespace.

oc label deploy ibm-zen-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=zen-chart
oc label role ibm-zen-operator-role -n <operator namespace> foundationservices.cloudpak.ibm.com=zen-chart
oc label role ibm-zen-operator-role -n <services namespace> foundationservices.cloudpak.ibm.com=zen-chart
oc label role ibm-zen-operator-role -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=zen-chart
oc label rolebinding ibm-zen-operator-rolebinding -n <operator namespace> foundationservices.cloudpak.ibm.com=zen-chart
oc label rolebinding ibm-zen-operator-rolebinding -n <services namespace> foundationservices.cloudpak.ibm.com=zen-chart
oc label rolebinding ibm-zen-operator-rolebinding -n <all tenant namespaces if applicable> foundationservices.cloudpak.ibm.com=zen-chart
oc label serviceaccount ibm-zen-operator-serviceaccount -n <operator namespace> foundationservices.cloudpak.ibm.com=zen-chart
oc label zenservice $zen_service -n <zenservice namespace> foundationservices.cloudpak.ibm.com=zen-chart

Label the zen Helm release secret:

oc label secret <zen Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=zen-chart

Labelling the resources for standalone services

Edit online

Add labels for License Service (ibm-licensing operator)

oc label clusterrole ibm-license-service ibm-license-service-restricted ibm-licensing-default-reader ibm-licensing-operator -n <licensing namespace> foundationservices.cloudpak.ibm.com=ls-cluster
oc label clusterrolebinding ibm-license-service ibm-license-service-restricted ibm-licensing-default-reader ibm-licensing-operator ibm-license-service-cluster-monitoring-view -n <licensing namespace> foundationservices.cloudpak.ibm.com=ls-cluster
oc label customresourcedefinition ibmlicensingdefinitions.operator.ibm.com ibmlicensingquerysources.operator.ibm.com ibmlicensings.operator.ibm.com -n <operator namespace> foundationservices.cloudpak.ibm.com=ls-cluster

Label the Licensing Helm release secret if the cluster-scoped chart was installed via helm install and not helm template | oc apply. If a cluster-scoped chart is installed using a method different than "helm install", it does not have an associated Helm secret.

oc label secret <Licensing Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=ls-cluster

Namespace-scoped labels for License Service:

oc label ibmlicensing instance -n <licensing namespace> foundationservices.cloudpak.ibm.com=ls-chart
oc label deployment ibm-licensing-operator -n <licensing namespace> foundationservices.cloudpak.ibm.com=ls-chart
oc label role ibm-license-service ibm-license-service-restricted ibm-licensing-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=ls-chart
oc label rolebinding ibm-license-service ibm-license-service-restricted ibm-licensing-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=ls-chart
oc label serviceaccount ibm-license-service ibm-license-service-restricted ibm-licensing-default-reader ibm-licensing-operator -n <operator namespace> foundationservices.cloudpak.ibm.com=ls-chart

Label the Licensing Helm release secret:

oc label secret <Licensing Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=ls-chart

Add labels for Cert manager

oc label clusterrole ibm-cert-manager-operator -n <cert manager namespace> foundationservices.cloudpak.ibm.com=ibm-cm-chart
oc label customresourcedefinition challenges.acme.cert-manager.io orders.acme.cert-manager.io certificaterequests.cert-manager.io certificates.cert-manager.io clusterissuers.cert-manager.io issuers.cert-manager.io certmanagerconfigs.operator.ibm.com -n <cert manager namespace> foundationservices.cloudpak.ibm.com=ibm-cm-chart
oc label deployment ibm-cert-manager-operator -n <cert manager namespace> foundationservices.cloudpak.ibm.com=ibm-cm-chart
oc label serviceaccount -n $CERT_MANAGER_NAMESPACE ibm-cert-manager-operator -n <cert manager namespace> foundationservices.cloudpak.ibm.com=ibm-cm-chart
oc label role -n $CERT_MANAGER_NAMESPACE ibm-cert-manager-operator-leader-election-role -n <cert manager namespace> foundationservices.cloudpak.ibm.com=ibm-cm-chart
oc label rolebinding -n $CERT_MANAGER_NAMESPACE ibm-cert-manager-operator-leader-election-rolebinding -n <cert manager namespace> foundationservices.cloudpak.ibm.com=ibm-cm-chart

Label the Cert manager release secret for namespace scope, and for cluster-scoper deployments if installed via helm install and not helm template | oc apply.

oc label secret <Cert manager Helm release secret name> -n <secret namespace> foundationservices.cloudpak.ibm.com=ibm-cm-chart

Backup common-service-db

Edit online

Note: If you use an external database for storing your Identity Management (IM) and Zen data, you are responsible for backup and restore of this data.

Get the common-service-db backup resources.

wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/common-service-db/cs-db-backup-deployment.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/common-service-db/cs-db-backup-pvc.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/common-service-db/cs-db-br-script-cm-4.6.10.4.11.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/common-service-db/cs-db-role.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/common-service-db/cs-db-rolebinding.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/common-service-db/cs-db-sa.yaml

If you use foundational services version 4.19.0 or later, use the following paths instead:

wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/common-service-db/cs-db-backup-deployment.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/common-service-db/cs-db-backup-pvc.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/common-service-db/cs-db-br-script-cm.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/common-service-db/cs-db-role.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/common-service-db/cs-db-rolebinding.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/common-service-db/cs-db-sa.yaml

Update the backup files.
- Replace <cs-db namespace> with the namespace where common-service-db instance is running.
- Replace the <storage class> with the storage class that the current IM deployment uses.
Add the PVC to the cluster.
```
oc apply -f cs-db-backup-pvc.yaml
```
Add the cs-db-br-script-cm-4.6.10.4.11.yaml to the correct namespace:
```
oc apply -f cs-db-br-script-cm-4.6.10.4.11.yaml
```
If you use foundational services version 4.19.0 or later, add the cs-db-br-script-cm.yaml to the correct namespace:
```
oc apply -f cs-db-br-script-cm.yaml
```

Give the common-service-db backup necessary permissions

oc apply -f cs-db-sa.yaml

oc apply -f cs-db-role.yaml

oc apply -f cs-db-rolebinding.yaml

Add the deployment to the cluster.

oc apply -f cs-db-backup-deployment.yaml

Back up Zen

Edit online

Locate zenservice instances.
```
oc get zenservice -A
```

Label each zenservice.

oc label zenservice <zenservice name> foundationservices.cloudpak.ibm.com=zen --overwrite=true -n <namespace>

Back up Zen MetastoreDB

Edit online

Note: Repeat this step for each namespace where a zenservice instance is installed.

Get the Zen 5 backup resources.

wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/zen5-backup-deployment.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/zen5-backup-pvc.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/zen5-br-scripts-cm.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/zen5-sa.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/zen5-role.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/zen5-rolebinding.yaml

If you use foundational services version 4.19.0 or later, use the following paths instead:

wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/zen5-backup-deployment.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/zen5-backup-pvc.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/zen5-br-scripts-cm.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/zen5-sa.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/zen5-role.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/ibm-pg/zen5-rolebinding.yaml

Update the backup files.

By default, the zen5-backup-pvc.yaml needs to replace the following parameters:
- Replace <zenservice namespace> with the namespace where the zenservice instance is running.
- Replace the <storage class> with either the storage class that common-service-db deployment uses, or with any storage class that has Retain ReclaimPolicy.
In the zen5-backup-deployment.yaml file, replace all instances of <zenservice namespace> with the namespace where the zenservice instance is running. There are four; two are parameters for the velero backup and restore commands.

By default, the backup and restore commands (represented by .spec.template.metadata.annotations.pre.hook.backup.velero.io/command & .spec.template.metadata.annotations.post.hook.restore.velero.io/command) are scheduled to run in the <zenservice namespace> namespace as parameters to the scripts called in the commands. Edit both commands' first parameter values to match the namespace that the deployment is created in.

By default, the restore command (represented by .spec.template.metadata.annotations.post.hook.restore.velero.io/command) is set to run against zenservice named <zenservice name>. Update the second parameter to match the name of the zenservice in the target namespace.

In the zen5-br-scripts-cm.yaml and zen5-sa.yaml, make sure to replace the namespace value <zenservice namespace> with the zenservice namespace in use for each instance of zenservice in use.
Add the PVC to the cluster.
```
oc apply -f zen5-backup-pvc.yaml
```
Add the zen5-br-scripts-cm.yaml to the correct namespace
```
oc apply -f zen5-br-scripts-cm.yaml
```
Give the Zen 5 backup necessary permissions
- For each namespace with a zenservice to backup, create a service account. Replace the <zenservice namespace> value before applying.
```
oc apply -f zen5-sa.yaml
```
- Once per zenservice namespace, apply the Role for the zen backup. Replace the <zenservice namespace> value before applying.
```
oc apply -f zen5-role.yaml
```
- Create the RoleBinding to connect the ServiceAccount to the Role.
  1. Edit the zen5-rolebinding.yaml file to add the ServiceAccount created earlier and replace the <zenservice namespace> value.
```
vi zen5-rolebinding.yaml
```
  2. Apply the zen5-rolebinding.yaml file
```
oc apply -f zen5-rolebinding.yaml
```

Add the deployment to the cluster.

oc apply -f zen5-backup-deployment.yaml

Create a backup resource

Edit online

Create a backup resource for the velero namespace.

Get the schedule-common-services.yaml file.

wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/schedule-common-services.yaml

Update the schedule-common-services.yaml file based on your backup requirements. For more information, see Velero Schedule API Type . By default, the backup runs once a day and is deleted 48 hours later.

The following configurations in the schedule-common-services.yaml file are important:
- schedule:, which is a CRON expression. CRON uses the server time, which is usually the Coordinated Universal Time unless configured to be something else.
- ttl, which is the time to live for the backup.
- storageLocation, which is the same storage location that you used when you set up OADP. The command oc get backupstoragelocations.velero.io -n <velero namespace> can be used to get the name.
- velero, which is the namespace where you installed OADP.

Create the resource.

oc apply -f schedule-common-services.yaml

Verify whether the backup schedule was created.
```
velero schedule get
```
After the first scheduled time passes, you can verify whether the backup ran. Look for a schedule name and timestamp.
```
velero backup get
```
Verify whether the backup was successful and check the details to see if all resources are saved.
```
velero backup describe <__BACKUP_NAME__> --details
```

Reconfiguring an external PostgreSQL database

Edit online

Complete this step only if you use an externally provided PostgreSQL database. Otherwise, skip this step and proceed with Restoring foundational services.

If you use an externally provided PostgreSQL database, you must reconfigure your external PostgreSQL database solution and restore its data. For more information, see Configuring an external PostgreSQL database for IM. Make sure that IS_EMBEDDED field is properly set to "false" in the common-service-db-im configmap.

Restoring foundational services

Edit online

Complete the following steps to restore foundational services.

Before you restore foundational services, set up Velero on the new cluster. Follow the instructions in the Create the backup resources section.

For troubleshooting issues that may arise during restore, see IBM Cloud Pak foundational services Installation Troubleshooting.

Download the necessary files for restoring different resources:

wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-namespace.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-entitlementkey.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-configmap.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-crd.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-commonservice.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-licensing.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-cert-manager.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-operands.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-cs-db.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-zen5-data.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-nss.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/no-olm/restore-cluster-charts.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/no-olm/restore-im-ns-charts.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/no-olm/restore-installer-ns-charts.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/no-olm/restore-zen-ns-chart.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/no-olm/restore-ibm-cm-chart.yaml

Restore the foundational services namespaces by using the restore-namespace.yaml file.
1. Get the name of the Velero backup that you plan to use for restoring.
```
velero backup get
```
  Replace __BACKUP_NAME__ in the following commands with the Velero backup name.
  
  Verify whether the backup was successful and check the details to see if all resources are saved.
```
velero backup describe <__BACKUP_NAME__> --details
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-namespace.yaml
```
3. Restore the namespace.
```
oc apply -f restore-namespace.yaml
```
  You can check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
4. Verify whether the namespace is restored. Your namespace must be listed in the command output.
```
oc get namespace
```
  Proceed with the next step after the namespace is restored.
5. Change the default project to the restored common service namespace.
```
oc project <namespace-where-foundational-services-are-installed>
```
Restore the entitlement key.
1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-entitlementkey.yaml
```
2. Restore the entitlement key.
```
oc apply -f restore-entitlementkey.yaml
```
3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
4. Verify whether the entitlement key is restored.
```
oc get secret
```
Restore common-service-maps configmap.
1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-configmap.yaml
```
2. Restore the configmap.
```
oc apply -f restore-configmap.yaml
```
3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
4. Verify whether the configmap is restored.
```
oc get configmap common-service-maps -n kube-public
```
Restore the commonservices.operator.ibm.com customresourcedefinition (CRD).
1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-crd.yaml
```
2. Restore the CRD.
```
oc apply -f restore-crd.yaml
```
3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
4. Verify whether the crd is restored.
```
oc get customresourcedefinition | grep commonservices.operator.ibm.com
```
Restore the common-service CR.
1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-commonservice.yaml
```
2. Restore the CR.
```
oc apply -f restore-commonservice.yaml
```
3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
4. Verify whether the commonservice is restored.
```
oc get commonservice
```
  If the foundational services are not restored, delete the restore resource and apply it again:
  1. Delete the resource.
```
oc delete -f restore-commonservice.yaml
```
  2. Restore the CR.
```
oc apply -f restore-commonservice.yaml
```
    Wait for 30 seconds and check again for the CommonService resource.
Restore Cert Manager operator chart.
1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-ibm-cm-chart.yaml
```
2. Restore the Cert manager operator.
```
oc apply -f restore-ibm-cm-chart.yaml
```
3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
4. Verify whether the Cert Manager operator and operand pods are up.
```
oc get pods -n <Cert Manager namespace>
```
  The reply should contain the operator, cainjector, controller, and webhook pods.
Restore cert manager resource.
1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-cert-manager.yaml
```
2. Restore the cert manager resource.
```
oc apply -f restore-cert-manager.yaml
```
3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
4. Verify whether the certificates are restored.
```
oc get certificates
```
Restore License Service.
1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-licensing.yaml
```
2. Restore the configmap.
```
oc apply -f restore-licensing.yaml
```
3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
4. Verify whether the configmap is restored.
```
oc get configmap | grep licensing
```
If you use IBM License Service Reporter, see Backing up the License Service Reporter instance.
Restore the Helm charts.
1. Restore the cluster scope resources:
  1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi no-olm/restore-cluster-scope.yaml
```
  2. Restore the cluster scope resources.
```
oc apply -f no-olm/restore-cluster-scope.yaml
```
  3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
  4. Watch the foundational services namespace for the operand-deployment-lifecycle-manager and common-service operator to be running:
```
oc get pod -n <cs namespace> -w
```
2. Restore the namespace scope resources if you use Separation of Duties (SOD).
  1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-nss.yaml
```
  2. Restore the namespace scope resources.
```
oc apply -f restore-nss.yaml
```
  3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
  4. Watch the foundational services namespace for the operand-deployment-lifecycle-manager and common-service operator to be running:
```
oc get pod -n <cs namespace> -w
```
    Troubleshooting: In case of issues with generating new installation plans for updates or new installations, see OLM is unable to generate new install plans.
  5. Wait for the namespace scope configmap to come ready.
3. Restore the foundational services operator and ODLM namespace charts.
  1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi no-olm/restore-installer-ns-charts.yaml
```
  2. Restore the cluster scope resources.
```
oc apply -f no-olm/restore-installer-ns-charts.yaml
```
  3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
4. Restore the IM, UI, and EDB namespace charts.
  1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi no-olm/restore-im-ns-charts.yaml
```
  2. Restore the cluster scope resources.
```
oc apply -f no-olm/restore-im-ns-charts.yaml
```
  3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
5. Restore the OperandRequests.
  1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-operands.yaml
```
  2. Restore the operands.
```
oc apply -f restore-operands.yaml
```
  3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
  4. Verify whether the operands are restored.
```
oc get operandrequest
```
  5. Verify whether operand requests are reconciled.
    Give ODLM time to reconcile one or more restored operand requests but new operators and their operands should be seen deploying shortly after the restore completes. Check the operand requests' status fields and the ODLM logs for any issues.
  6. If using a custom hostname, TLS secret, or both, wait for the platform-identity pods to come ready:
    - Verify that the cs-onprem-tenant-config configmap is present:
```
oc get cm -n <namespace where hostname is changed or custom TLS secret used> | grep cs-onprem-tenant-config
```
    - Wait for the platform-identity-management, platform-identity-provider, and platform-auth-service pods to come ready in the same namespace.
    - Make sure to update the custom hostname to reflect a change in cluster if necessary. For example, the structure of the route is <route name>.cluster1.com. If you are no longer on cluster1 but now on cluster2, the route needs to be updated from <route name>.cluster1.com to <route name>.cluster2.com.
    - If using a custom TLS secret, it is best to re-create this secret on the new cluster by using the same name. In this case, if the secret was carried over to the new cluster, it would need to be replaced.
    - Follow the instructions here https://www.ibm.com/docs/en/cloud-paks/foundational-services/4.3?topic=cc-updating-custom-hostname-tls-secret-by-using-configmap.
6. Restore zenservice instances.
  1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-zen-ns-charts.yaml
```
  2. Restore the zenservice instances.
```
oc apply -f restore-zen-ns-charts.yaml
```
  3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
  4. Wait for the zenservice instances to come ready. Once the Progress field is 100%, the instance is ready. The following command will continuously output the percentage of all the zenservices on the cluster.
```
oc get zenservice -A -w -o yaml | grep Progress:
```
    If the restored zenservice contains fields to configure zenCustomRoute, do the following:
    - Verify the secret used (if the field exists) is present in the zenservice namespace in the target cluster.
    - Update the value in the zenservice CR for the route. For example, the structure of the route is <route name>.cluster1.com. If you are no longer on cluster1 but now on cluster2, the route needs to be updated from <route name>.cluster1.com to <route name>.cluster2.com. no-olm/
Restore common-service-db
1. Get the restore object
```
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-cs-db.yaml
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created previously.
```
vi restore-cs-db.yaml
```
3. Restore the cs-db data.
```
oc apply -f restore-cs-db.yaml
```
4. Check restore progress. Proceed with the next step after restore is complete.
```
velero restore get
```
5. Check logs of the velero restore to verify that the data was restored
```
velero restore logs restore-cs-db-data
```
  Troubleshooting: If the logs or the data indicate that the restore was not successful, apply the following workaround:
  1. Get the restore job:
```
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/common-service-db/cs-db-restore-job.yaml
```
  2. Replace <cs-db namespace> with the namespace where common-service-db instance is running.
  3. Delete the existing cs-db-backup deployment and cs-db-backup pod.
```
oc delete deploy cs-db-backup -n <namespace>
```
  4. Run the restore job.
```
oc apply -f cs-db-restore-job.yaml
```
  Note: The secondary steps that are listed here must be run only if the restore logs indicate that the restore was not run. If the storage class used on the backup cluster does not match the storage class that is used on the target cluster, the restore fails. Adapting to different storage classes across clusters is a current limitation of velero.
Restore Zen data.
1. Complete this step only if you use an externally provided PostgreSQL database. Reconfigure the external PostgreSQL database instance. For more information, see Configuring an external PostgreSQL database for IM.
2. Restore zen data.
  1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
vi restore-zen5-data.yaml
```
  2. Restore the Zen data.
```
oc apply -f restore-zen5-data.yaml
```
  3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.
```
velero restore get
```
```
velero restore describe <__RESTORE_NAME__> --details
```
  4. Check logs of the velero restore to verify that the data was restored
```
velero restore logs restore-zen5-data
```
    - Search for restore_zen5 to find relevant logs. If it is not present, the restore did not run. If the logs or the data indicate that the restore was not successful, the following steps can be taken as a workaround:
      1. Get the Zen 5 restore job resource.
        
        wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/zen/zen5-restore-job.yaml
      2. Delete the existing zen5-backup deployment
        
        oc delete deploy zen5-backup -n <namespace>
      3. Wait for the zen5-backup pods to fully delete (fully gone, not Terminating)
      4. Give the zen5 backup necessary permissions if the necessary ServiceAccount, Role, and RoleBinding are not already present.
        
        Check if permissions exist:
        
        oc get sa -n <zenservice namespace> | grep zen5 oc get role | grep zen5 oc get rolebinding | grep zen5
        
        Get the zen5-sa.yaml, zen5-role.yaml, & zen5-rolebinding.yaml files.
        
        wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/zen5-sa.yaml wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/zen5-role.yaml wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/zen5-rolebinding.yaml
        
        For each namespace with a zenservice to backup, edit the service account file zen5-sa.yaml to deploy in the corresponding namespace
        
        oc apply -f zen5-sa.yaml
        
        Once per zenservice namespace, apply the zen5-role.yaml file to create the Role for the zen backup. Replace the <zenservice namespace> value before applying.
        
        oc apply -f zen5-role.yaml
        
        Create the RoleBinding to connect the ServiceAccounts to the Role.
        
        Edit the zen5-rolebinding.yaml file to add each ServiceAccount created earlier. Replace the <zenservice namespace> value before applying.
        vi zen5-rolebinding.yaml
        
        Apply the zen5-rolebinding.yaml file
        oc apply -f zen5-rolebinding.yaml
      5. Edit the zen5-restore-job.yaml file. The default namespace is set to zen. The parameters for the underlying restore_zen5.sh are defaulted to the zen namespace and test-zen zenservice name. Update both of these parameters to reflect the proper namespace and zenservice respectively.
      6. Apply the zen5-restore-job.yaml file
        
        oc apply -f zen5-restore-job.yaml
      7. Wait for the job to complete, then check the logs of the zen5-restore-job pod to verify restore completed.
      8. Repeat as needed for each namespace with a zenservice instance installed.
  5. Wait for the zenservice instances to come ready. Once the Progress field is 100%, the instance is ready. The following command will continuously output the percentage of all the zenservices on the cluster.
```
oc get zenservice -A -w -o yaml | grep Progress:
```
  Fee the following troubleshooting tips:
  - Make sure that there is only one zen5-backup or one zen5-restore-job pod in a namespace at any given time as they compete for the same PVC.
  - If the zen5-restore-job pod is stuck in ContainerCreating:
    1. delete the deployment zen5-backup
    2. make sure the zen5-backup pod is fully deleted (not Terminating)
    3. delete the zen5-restore-job job and its pod (not Terminating)
    4. ensure that the configmap zen5-br-configmap, pvc zen5-backup-pvc, role zen5-backup-role, rolebinding zen5-backup-rolebinding, and service account zen5-backup-sa are present in the namespace
    5. reapply the zen5-restore-job yaml
  - If the configmap zen5-br-configmap is not present, it can be downloaded from:
```
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/zen5-br-scripts-cm.yaml.
```
    Make sure to edit the namespace field before applying with the following command:
```
oc apply -f zen5-br-scripts-cm.yaml
```
  - Velero restore is less predictable than backup when restoring databases. There is no harm to deleting a Completed velero restore object (that is, restore-cs-db-data or restore-zen5-data), deleting the accompanying deployment and pvc, waiting for these items to be fully deleted, then re-creating the velero restore object to try again. Should this still not work, the secondary instructions by using the cs-db-restore-job.yaml and zen5 restore job can be used on an individual namespace basis. There is no harm to running the restore in a namespace that has already been restored.

If you use a custom route for the restored zenservice and you are restoring to a new cluster, update the value of zenCustomRoute in the zenservice CR to reflect the new hostname and re-trigger the iam-config job. Run the following commands:

oc -n <zenservice namespace> patch zenservice  <zenservice name>  --type='merge' -p '{"spec":{"zenCustomRoute":{"route_host":"<updated route>"}}}'
oc -n  <zenservice namespace> patch zenservice <zenservice name>  --type='merge' -p '{"spec":{"reconcile":true}}'
oc get job -n  <zenservice namespace> iam-config-job -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | oc replace --force -f -

All restoration tasks are completed.

Verify whether foundational services are properly restored.

Verify the pods:
```
oc get pods
```
All pods must be running.
Verify the subscriptions:
```
oc get subscriptions
```
Subscriptions of all installed services must be listed.
Verify that the Identity and Access section of the cp-console shows the users and teams that your organization added in the original cluster.

For backing up and restoring Identity Management (IM) components, see Identity management backup and restore.

For migrating existing OIDC and SAML configurations, see Migrating identity management.

General Troubleshooting:

Edit online

If a restore process is stopped in the New phase when you view with velero restore get, restart the velero pod in the namespace where OADP is installed. After the velero pod restarts, the status of the restore process must change to InProgress.