IBM Cloud Pak foundational services backup and restore for clusters with a single instance of foundational services

• If you plan to restore foundational services on a new cluster and currently use an ImageContentSourcePolicy resource, reconfigure this resource on the new cluster. For more information, see Configure the cluster.

• Set up any Amazon S3-compatible storage. For example, you can create a bucket in IBM Cloud Object Storage. For more information, see IBM Cloud Object Storage .

• When you add a service credential to the bucket, include the hash-based message authentication code (HMAC). For more information, see Service credentials . From the Cloud Object Storage navigation menu, gather the following information:

  • access key id, which can be found on the Service credentials page that is associated with the bucket.

  • secret access key, which can be found on the Service credentials page that is associated with the bucket.

  • bucket name, which can be found on the Buckets page.

  • bucket region, which can be found on the Buckets page.

  • root directory name, which is the path in the bucket where you want to store the backups.

  • s3 URL, which is the endpoint URL of the bucket.

    • The endpoint URL must start with http:// or https://.

If you use an external database for storing your Identity Management (IM) and Zen data, you are responsible for backup and restore of this data.

Complete the following steps to back up the installed foundational services.

Complete this step only if you use an externally provided PostgreSQL database. Otherwise, skip this step and proceed with Restoring foundational services.

If you use an externally provided PostgreSQL database, you must reconfigure your external PostgreSQL database solution and restore its data. For more information, see Configuring an external PostgreSQL database for IM. Make sure that IS_EMBEDDED field is properly set to "false" in the common-service-db-im configmap.

Complete the following steps to restore foundational services.

Before you restore foundational services, set up Velero on the new cluster. Follow the instructions in the Create the backup resources section.

For troubleshooting issues that may arise during restore, see IBM Cloud Pak foundational services Installation Troubleshooting.

Download the necessary files for restoring different resources:

wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-namespace.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-entitlementkey.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-pull-secret.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-catalog.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-operatorgroup.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-configmap.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-crd.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-commonservice.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-subscriptions.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-licensing.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-cert-manager.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-operands.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-cs-db.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-zen.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-zen5-data.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-singleton-subscriptions.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-nss.yaml
wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-ums.yaml

1. Restore the foundational services namespaces by using the restore-namespace.yaml file.

   1. Get the name of the Velero backup that you plan to use for restoring.

      velero backup get
      

      Replace __BACKUP_NAME__ in the following commands with the Velero backup name.

      Verify whether the backup was successful and check the details to see if all resources are saved.

      velero backup describe <__BACKUP_NAME__> --details
      

   2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

      vi restore-namespace.yaml
      

   3. Restore the namespace.

      oc apply -f restore-namespace.yaml
      

      You can check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

      velero restore get
      

      velero restore describe <__RESTORE_NAME__> --details
      

   4. Verify whether the namespace is restored. Your namespace must be listed in the command output.

      oc get namespace
      

      Proceed with the next step after the namespace is restored.

   5. Change the default project to the restored common service namespace.

      oc project <namespace-where-foundational services-are-installed>
      

2. Restore the entitlement key.

   1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

      vi restore-entitlementkey.yaml
      

   2. Restore the entitlement key.

      oc apply -f restore-entitlementkey.yaml
      

   3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

      velero restore get
      

      velero restore describe <__RESTORE_NAME__> --details
      

   4. Verify whether the entitlement key is restored.

      oc get secret
      

3. Restore the pull secret.

   1. Save the current pull secret.

      oc get secret pull-secret -n openshift-config -o yaml > original-pull-secret.yaml
      

   2. Delete the current pull secret from the openshift-config namespace.

      oc delete secret pull-secret -n openshift-config
      

   3. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

      vi restore-pull-secret.yaml
      

   4. Restore the pull secret.

      oc apply -f restore-pull-secret.yaml
      

   5. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

      velero restore get
      

      velero restore describe <__RESTORE_NAME__> --details
      

   6. Verify whether the pull secret is restored.

      oc get secret -n openshift-config | grep pull
      

4. Restore the catalog.

   1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

      vi restore-catalog.yaml
      

   2. Restore the catalog.

      oc apply -f restore-catalog.yaml
      

   3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

      velero restore get
      

      velero restore describe <__RESTORE_NAME__> --details
      

   4. Verify whether the catalog source is restored.

      oc get catalogsource -n openshift-marketplace | grep ibm
      

   5. Verify whether the ibm-operator-catalog pod is running.

      oc get pod -n openshift-marketplace -w
      

      Note: If using IBM Cert Manager, IBM Licensing, or cloud-native-postgresql-catalog catalog source, verify that their pods are Running as well.

   If the pods are running, proceed with the next step.

5. Restore the operator group.

   1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

      vi restore-operatorgroup.yaml
      

   2. Restore the operator group.

      oc apply -f restore-operatorgroup.yaml
      

   3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

      velero restore get
      

      velero restore describe <__RESTORE_NAME__> --details
      

   4. Verify whether the operator group is restored.

      oc get operatorgroup
      

6. Restore common-service-maps configmap.

   1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

      vi restore-configmap.yaml
      

   2. Restore the configmap.

      oc apply -f restore-configmap.yaml
      

   3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

      velero restore get
      

      velero restore describe <__RESTORE_NAME__> --details
      

   4. Verify whether the configmap is restored.

      oc get configmap common-service-maps -n kube-public
      

7. Restore the commonservices.operator.ibm.com customresourcedefinition (CRD).

   1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

      vi restore-crd.yaml
      

   2. Restore the CRD.

      oc apply -f restore-crd.yaml
      

   3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

      velero restore get
      

      velero restore describe <__RESTORE_NAME__> --details
      

   4. Verify whether the crd is restored.

      oc get customresourcedefinition | grep commonservices.operator.ibm.com
      

8. Restore the common-service CR.

   1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

      vi restore-commonservice.yaml
      

   2. Restore the CR.

      oc apply -f restore-commonservice.yaml
      

   3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

      velero restore get
      

      velero restore describe <__RESTORE_NAME__> --details
      

   4. Verify whether the commonservice is restored.

      oc get commonservice
      

      If the foundational services are not restored, delete the restore resource and apply it again:

      1. Delete the resource.

         oc delete -f restore-commonservice.yaml
         

      2. Restore the CR.

         oc apply -f restore-commonservice.yaml
         

         Wait for 30 seconds and check again for the CommonService resource.

9. Restore the singleton subscriptions.

   1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

      vi restore-singleton-subscriptions.yaml
      

   2. Restore the subscriptions.

      oc apply -f restore-singleton-subscriptions.yaml
      

   3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

      velero restore get
      

      velero restore describe <__RESTORE_NAME__> --details
      

   4. Watch the namespaces where Cert Manager and License Service are deployed for the Cert Manager and Licensing operators to be running. By default Cert Manager and License Service are deployed in ibm-cert-manager and ibm-licensing namespaces.

      oc get pod -n <cs namespace> -w
      

10. Restore cert manager resource.

    1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

       vi restore-cert-manager.yaml
       

    2. Restore the cert manager resource.

       oc apply -f restore-cert-manager.yaml
       

    3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

       velero restore get
       

       velero restore describe <__RESTORE_NAME__> --details
       

    4. Verify whether the certificates are restored.

       oc get certificates
       

11. Restore the subscriptions.

    1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

       vi restore-subscriptions.yaml
       

    2. Restore the subscriptions.

       oc apply -f restore-subscriptions.yaml
       

    3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

       velero restore get
       

       velero restore describe <__RESTORE_NAME__> --details
       

    4. Watch the foundational services namespace for the operand-deployment-lifecycle-manager to be running:

       oc get pod -n <cs namespace> -w
       

    See the following notes:

    • If not using IBM Cert Manager, IBM Common Service Operator deployment fails unless a third-party Cert Manager is installed on the cluster beforehand.
    • If using SOD, it is likely the ibm-common-service-operator not come ready after restoring the subscriptions and subsequently will not deploy ODLM. This is expected and will resolve after running the next step.

    Troubleshooting: In case of issues with generating new installation plans for updates or new installations, see OLM is unable to generate new install plans.

12. Restore the namespace scope resources if you use Separation of Duties (SOD).

    1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

       vi restore-nss.yaml
       

    2. Restore the namespace scope resources.

       oc apply -f restore-nss.yaml
       

    3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

       velero restore get
       

       velero restore describe <__RESTORE_NAME__> --details
       

    4. Watch the foundational services namespace for the operand-deployment-lifecycle-manager and common-service operator to be running:

       oc get pod -n <cs namespace> -w
       

    Troubleshooting: In case of issues with generating new installation plans for updates or new installations, see OLM is unable to generate new install plans.

13. Restore Licensing service configmap.

    1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

       vi restore-licensing.yaml
       

    2. Restore the configmap.

       oc apply -f restore-licensing.yaml
       

    3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

       velero restore get
       

       velero restore describe <__RESTORE_NAME__> --details
       

    4. Verify whether the configmap is restored.

       oc get configmap | grep licensing
       

14. If you use IBM License Service Reporter, see Backing up the License Service Reporter instance.

15. Restore Usage Metering Service if you use it.

    1. Substitute the __BACKUP_NAME__ value with the name of the backup resource that you created in a previous step.

       vi restore-ums.yaml
       

    2. Restore the operands.

       oc apply -f restore-ums.yaml
       

    3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

       velero restore get
       

       velero restore describe <__RESTORE_NAME__> --details
       

    4. Verify that the ibmusagemeterings.operator.ibm.com CR, ibmservicemeterdefinitions.operator.ibm.com CRs, and the ibm-usage-metering-events configmap are restored.

    5. Wait for the ibm-usage-metering-operator operator to install and verify that the instance pod comes up.

16. Restore the OperandRequests.

    1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

       vi restore-operands.yaml
       

    2. Restore the operands.

       oc apply -f restore-operands.yaml
       

    3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

       velero restore get
       

       velero restore describe <__RESTORE_NAME__> --details
       

    4. Verify whether the operands are restored.

       oc get operandrequest
       

    5. Verify whether operand requests are reconciled.

       Give ODLM time to reconcile one or more restored operand requests but new operators and their operands should be seen deploying shortly after the restore completes. Check the operand requests' status fields and the ODLM logs for any issues.

    6. If using a custom hostname, TLS secret, or both, wait for the platform-identity pods to come ready:

       • Verify that the cs-onprem-tenant-config configmap is present:

         oc get cm -n <namespace where hostname is changed or custom TLS secret used> | grep cs-onprem-tenant-config
         

       • Wait for the platform-identity-management, platform-identity-provider, and platform-auth-service pods to come ready in the same namespace.

       • Make sure to update the custom hostname to reflect a change in cluster if necessary. For example, the structure of the route is <route name>.cluster1.com. If you are no longer on cluster1 but now on cluster2, the route needs to be updated from <route name>.cluster1.com to <route name>.cluster2.com.

       • If using a custom TLS secret, it is best to re-create this secret on the new cluster by using the same name. In this case, if the secret was carried over to the new cluster, it would need to be replaced.

       • Follow the instructions here https://www.ibm.com/docs/en/cloud-paks/foundational-services/4.3?topic=cc-updating-custom-hostname-tls-secret-by-using-configmap.

17. Restore common-service-db.

    1. Get the restore object.

       wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/restore-cs-db.yaml
       

    2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created previously.

       vi restore-cs-db.yaml
       

    3. Restore the cs-db data.

       oc apply -f restore-cs-db.yaml
       

    4. Check restore progress. Proceed with the next step after restore is complete.

       velero restore get
       

    5. Check logs of the velero restore to verify that the data was restored.

       velero restore logs restore-cs-db-data
       

       Troubleshooting: If the logs or the data indicate that the restore was not successful, apply the following workaround:

       1. Get the restore job:

          wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/restore/common-service-db/cs-db-restore-job.yaml
          

       2. Replace <cs-db namespace> with the namespace where common-service-db instance is running.

       3. Delete the existing cs-db-backup deployment and cs-db-backup pod.

          oc delete deploy cs-db-backup -n <namespace>
          

       4. Run the restore job.

          oc apply -f cs-db-restore-job.yaml
          

       Note: The secondary steps that are listed here must be run only if the restore logs indicate that the restore was not run. If the storage class used on the backup cluster does not match the storage class that is used on the target cluster, the restore fails. Adapting to different storage classes across clusters is a current limitation of velero.

18. Restore Zen and Zen data. 1. Complete this step only if you use an externally provided PostgreSQL database. Reconfigure the external PostgreSQL database instance. For more information, see Configuring an external PostgreSQL database for IM. 2. Restore zenservice instances.

    1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

       vi restore-zen.yaml
       

    2. Restore the zenservice instances.

       oc apply -f restore-zen.yaml
       

    3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

       velero restore get
       

       velero restore describe <__RESTORE_NAME__> --details
       

    4. Wait for the zenservice instances to come ready. Once the Progress field is 100%, the instance is ready. The following command will continuously output the percentage of all the zenservices on the cluster.

       oc get zenservice -A -w -o yaml | grep Progress:
       

    5. Complete this step only if you use an extrnally provided PostreSQL database. Once the the progress status of zenservice is at 100%, suspend the backup cronjob with the following command:

       oc patch cj zen-metastore-backup-cron-job
       --namespace ${ZEN_NAMESPACE}
       --type=merge
       --patch '{"spec": {"suspend": true}}'
       

    If the restored zenservice contains fields to configure zenCustomRoute, do the following:

    • Verify the secret used (if the field exists) is present in the zenservice namespace in the target cluster.
    • Update the value in the zenservice CR for the route. For example, the structure of the route is <route name>.cluster1.com. If you are no longer on cluster1 but now on cluster2, the route needs to be updated from <route name>.cluster1.com to <route name>.cluster2.com.

    1. Restore zen data.

       1. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.

          vi restore-zen5-data.yaml
          

       2. Restore the Zen data.

          oc apply -f restore-zen5-data.yaml
          

       3. Check the progress and the details of the restore by using the following commands. Proceed with the next step after the status shows as Completed.

          velero restore get
          

          velero restore describe <__RESTORE_NAME__> --details
          

       4. Check logs of the velero restore to verify that the data was restored.

          velero restore logs restore-zen5-data
          

          • Search for restore_zen5 to find relevant logs. If it is not present, the restore did not run. If the logs or the data indicate that the restore was not successful, see The data restore step ends with the PartiallyFailed status.

       5. Wait for the zenservice instances to come ready. Once the Progress field is 100%, the instance is ready. The following command will continuously output the percentage of all the zenservices on the cluster.

          oc get zenservice -A -w -o yaml | grep Progress:
          

       See the following troubleshooting tips:

       • Make sure that there is only one zen5-backup or one zen5-restore-job pod in a namespace at any given time as they compete for the same PVC.

       • If the zen5-restore-job pod is stuck in ContainerCreating:

         1. Delete the deployment zen5-backup.
         2. Make sure the zen5-backup pod is fully deleted (not Terminating).
         3. Delete the zen5-restore-job job and its pod (not Terminating).
         4. Ensure that the configmap zen5-br-configmap, pvc zen5-backup-pvc, role zen5-backup-role, rolebinding zen5-backup-rolebinding, and service account zen5-backup-sa are present in the namespace.
         5. Reapply the zen5-restore-job yaml.

       • If the configmap zen5-br-configmap is not present, it can be downloaded from:

         wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts-adopter/velero/schedule/zen5-br-scripts-cm.yaml.
         

         Make sure to edit the namespace field before applying with the following command:

         oc apply -f zen5-br-scripts-cm.yaml
         

       • Velero restore is less predictable than backup when restoring databases. There is no harm to deleting a Completed velero restore object (that is, restore-cs-db-data or restore-zen5-data), deleting the accompanying deployment and pvc, waiting for these items to be fully deleted, then re-creating the velero restore object to try again. Should this still not work, the secondary instructions by using the cs-db-restore-job.yaml and zen5 restore job can be used on an individual namespace basis. There is no harm to running the restore in a namespace that has already been restored.

19. If you use a custom route for the restored zenservice and you are restoring to a new cluster, update the value of zenCustomRoute in the zenservice CR to reflect the new hostname and re-trigger the iam-config job. Run the following commands:

    oc -n <zenservice namespace> patch zenservice  <zenservice name>  --type='merge' -p '{"spec":{"zenCustomRoute":{"route_host":"<updated route>"}}}'
    oc -n  <zenservice namespace> patch zenservice <zenservice name>  --type='merge' -p '{"spec":{"reconcile":true}}'
    oc get job -n  <zenservice namespace> iam-config-job -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | oc replace --force -f -
    

All restoration tasks are completed.

Verify whether foundational services are properly restored.

• Verify the pods:

  oc get pods
  

  All pods must be running.

• Verify the subscriptions:

  oc get subscriptions
  

  Subscriptions of all installed services must be listed.

• Verify that the Identity and Access section of the cp-console shows the users and teams that your organization added in the original cluster.

For backing up and restoring Identity Management (IM) components, see Identity management backup and restore.

For migrating existing OIDC and SAML configurations, see Migrating identity management.