Backing up and restoring IBM Cloud Pak for Integration

Label the instances to back up

The backup process backs up any resource that has an appropriate backup label. You must therefore ensure that every resource that you want to backup has such a label.

The following steps show how to add labels to the instances in Cloud Pak for Integration by using the Red Hat OpenShift CLI. If you want to do more advanced actions such as adding your own labels or labeling only some instances, see the More information about labels section later in this topic.

The following procedure uses the CLI to add labels.

1. (Prerequisite step) Ensure that you have the Red Hat OpenShift command-line interface (CLI) installed. For more information, see Getting started with the OpenShift CLI.

2. Log in to your OpenShift Container Platform system as an OpenShift cluster administrator by running the oc login command.

3. If you installed Cloud Pak for Integration into a single namespace, run the following command to make that namespace the default namespace. If you installed Cloud Pak for Integration into all namespaces, you can skip this step; later commands will run against all namespaces.

   oc project <cloud-pak-for-integration-namespace>
   

   For <cloud-pak-for-integration-namespace>, enter the namespace into which you installed the operators.

4. Add labels to the operators. For more information about oc label command usage, see oc label in the Red Hat OpenShift documentation.

   • Add labels to the catalog sources, which make the operators available for you to install:

     oc label catalogsource ibm-integration-platform-navigator-catalog backup.integration.ibm.com/component=catalogsource -n openshift-marketplace
     oc label catalogsource ibm-integration-asset-repository-catalog backup.integration.ibm.com/component=catalogsource -n openshift-marketplace
     oc label catalogsource appconnect-operator-catalogsource backup.appconnect.ibm.com/component=catalogsource -n openshift-marketplace
     oc label catalogsource ibm-datapower-operator-catalog backup.datapower.ibm.com/component=catalogsource -n openshift-marketplace
     oc label catalogsource ibm-eventstreams backup.eventstreams.ibm.com/component=catalogsource -n openshift-marketplace
     oc label catalogsource opencloud-operators backup.integration.ibm.com/component=catalogsource -n openshift-marketplace
     oc label catalogsource ibm-apiconnect-catalog backup.apiconnect.ibm.com/component=catalogsource -n openshift-marketplace
     oc label catalogsource ibmmq-operator-catalogsource backup.mq.ibm.com/component=catalogsource -n openshift-marketplace
     oc label catalogsource ibm-eventendpointmanagement-catalog backup.events.ibm.com/component=catalogsource -n openshift-marketplace
     

     The catalog source names in this example list are the default names when the catalog sources are created by using the oc ibm-pak tool. If you create your catalog sources by using a different method (such as installation by using the CLI or a CICD pipeline), you may need to enter different catalog source names.

   • Add labels to the operator subscriptions that are part of Cloud Pak for Integration, with the following exceptions:

     • Do not add labels for operators that were installed automatically, such as EDB. (You only need to add labels for operators that you installed.)
     • Do not add labels for the IBM DataPower Gateway operator if it was installed as a dependency of the IBM API Connect operator. If you do back up the subscription, the operator will not be installed when it is restored.

     oc label subscription ibm-integration-platform-navigator backup.integration.ibm.com/component=subscription
     oc label subscription ibm-integration-asset-repository backup.integration.ibm.com/component=subscription
     oc label subscription ibm-appconnect backup.appconnect.ibm.com/component=subscription
     oc label subscription datapower-operator backup.datapower.ibm.com/component=subscription
     oc label subscription ibm-eventstreams backup.eventstreams.ibm.com/component=subscription
     oc label subscription ibm-common-service-operator backup.integration.ibm.com/component=subscription
     oc label subscription ibm-apiconnect backup.apiconnect.ibm.com/component=subscription
     oc label subscription ibm-mq backup.mq.ibm.com/component=subscription
     oc label subscription ibm-eventendpointmanagement backup.events.ibm.com/component=subscription
     

     The subscription names in this example list are the default names when the operator is installed using the OpenShift web console. If you create your subscriptions using a different method (such as installation by using the CLI or a CICD pipeline), you may need to enter different subscription names.

   • Add labels to the operator groups. This step is required only if you installed the operators in A specific namespace on the cluster mode. If you installed the operators in All namespaces on the cluster mode, you do not need to backup the operator groups.

     oc label operatorgroup --all backup.integration.ibm.com/component=operatorgroup
     oc label operatorgroup --all backup.appconnect.ibm.com/component=operatorgroup
     oc label operatorgroup --all backup.eventstreams.ibm.com/component=operatorgroup
     oc label operatorgroup --all backup.datapower.ibm.com/component=operatorgroup
     oc label operatorgroup --all backup.apiconnect.ibm.com/component=operatorgroup
     oc label operatorgroup --all backup.mq.ibm.com/component=operatorgroup
     oc label operatorgroup --all backup.events.ibm.com/component=operatorgroup
     

5. Add labels to all the Cloud Pak for Integration instances on the OpenShift cluster that support OADP backup and restore:

   oc label platformnavigator --all --all-namespaces backup.integration.ibm.com/component=platformnavigator  
   oc label integrationassembly --all --all-namespaces backup.integration.ibm.com/component=integrationassembly
   oc label messagingserver --all --all-namespaces backup.integration.ibm.com/component=messagingserver
   oc label messagingqueue --all --all-namespaces backup.integration.ibm.com/component=messagingqueue
   oc label messagingchannel --all --all-namespaces backup.integration.ibm.com/component=messagingchannel
   oc label messaginguser --all --all-namespaces backup.integration.ibm.com/component=messaginguser
   oc label validatingadmissionpolicy --all --all-namespaces backup.integration.ibm.com/component=validatingadmissionpolicy
   oc label validatingadmissionpolicybinding --all --all-namespaces backup.integration.ibm.com/component=validatingadmissionpolicybinding
   oc label assetrepository --all --all-namespaces backup.integration.ibm.com/component=assetrepository
   oc label configuration --all --all-namespaces backup.appconnect.ibm.com/component=configuration
   oc label api --all --all-namespaces backup.apiconnect.ibm.com/component=api
   oc label product --all --all-namespaces backup.apiconnect.ibm.com/component=product
   oc label dashboard --all --all-namespaces backup.appconnect.ibm.com/component=dashboard
   oc label designerauthoring --all --all-namespaces backup.appconnect.ibm.com/component=designerauthoring
   oc label integrationruntime --all --all-namespaces backup.appconnect.ibm.com/component=integrationruntime
   oc label integrationserver --all --all-namespaces backup.appconnect.ibm.com/component=integrationserver
   oc label switchserver --all --all-namespaces backup.appconnect.ibm.com/component=switchserver
   oc label eventstreams --all --all-namespaces backup.eventstreams.ibm.com/component=eventstreams
   oc label eventgateway --all --all-namespaces backup.events.ibm.com/component=eventgateway
   oc label datapowerservice --all --all-namespaces backup.datapower.ibm.com/component=datapowerservice
   oc label kafkaconnect --all --all-namespaces backup.eventstreams.ibm.com/component=kafkaconnect
   oc label kafkatopic --all --all-namespaces backup.eventstreams.ibm.com/component=kafkatopic
   oc label kafkauser --all --all-namespaces backup.eventstreams.ibm.com/component=kafkauser
   oc label kafkabridge --all --all-namespaces backup.eventstreams.ibm.com/component=kafkabridge
   oc label kafkaconnector --all --all-namespaces backup.eventstreams.ibm.com/component=kafkaconnector
   oc label kafkarebalance --all --all-namespaces backup.eventstreams.ibm.com/component=kafkarebalance
   oc label apiconnectclusters --all --all-namespaces backup.apiconnect.ibm.com/component=apiconnectcluster
   oc label commonservice --all --all-namespaces backup.integration.ibm.com/component=commonservice
   

   If API Connect subsystems are deployed without the API Connect cluster CR, then label the API Connect subsystems and certificate issuers. Run these commands in each namespace where API Connect subsystems are deployed:

   oc label managementclusters --all backup.apiconnect.ibm.com/component=managementcluster
   oc label certificates.cert-manager.io ingress-ca backup.apiconnect.ibm.com/component=managementcluster 
   oc label certificates.cert-manager.io gateway-client-client backup.apiconnect.ibm.com/component=managementcluster
   oc label certificates.cert-manager.io portal-admin-client backup.apiconnect.ibm.com/component=managementcluster
   oc label certificates.cert-manager.io analytics-ingestion-client backup.apiconnect.ibm.com/component=managementcluster 
   oc label certificates.cert-manager.io api-manager-ca backup.apiconnect.ibm.com/component=managementcluster
   oc label secret api-manager-ca backup.apiconnect.ibm.com/component=managementcluster
   oc label secret portal-admin-client backup.apiconnect.ibm.com/component=managementcluster
   oc label secret gateway-client-client backup.apiconnect.ibm.com/component=managementcluster  
   
   oc label portalclusters --all backup.apiconnect.ibm.com/component=portalcluster
   oc label secret ingress-ca backup.apiconnect.ibm.com/component=portalcluster
   oc label issuers.cert-manager.io ingress-issuer backup.apiconnect.ibm.com/component=portalcluster                                                                            
   oc label issuers.cert-manager.io selfsigning-issuer backup.apiconnect.ibm.com/component=portalcluster
   
   oc label analyticsclusters --all backup.apiconnect.ibm.com/component=analyticscluster
   oc label issuers.cert-manager.io ingress-issuer backup.apiconnect.ibm.com/component=analyticscluster                                                                     
   oc label issuers.cert-manager.io selfsigning-issuer backup.apiconnect.ibm.com/component=analyticscluster
   oc label secret ingress-ca backup.apiconnect.ibm.com/component=analyticscluster 
   
   oc label gatewayclusters --all backup.apiconnect.ibm.com/component=gatewaycluster
   oc label issuers.cert-manager.io ingress-issuer backup.apiconnect.ibm.com/component=gatewaycluster
   oc label issuers.cert-manager.io selfsigning-issuer backup.apiconnect.ibm.com/component=gatewaycluster
   oc label secret gateway-service backup.apiconnect.ibm.com/component=gatewaycluster
   oc label secret gateway-peering backup.apiconnect.ibm.com/component=gatewaycluster
   oc label secret ingress-ca backup.apiconnect.ibm.com/component=gatewaycluster
   oc label secret admin-secret backup.apiconnect.ibm.com/component=gatewaycluster
   oc label secret api-manager-ca backup.apiconnect.ibm.com/component=gatewaycluster
   oc label certificates.cert-manager.io gateway-service backup.apiconnect.ibm.com/component=gatewaycluster 
   oc label certificates.cert-manager.io gateway-peering backup.apiconnect.ibm.com/component=gatewaycluster
   

   For API Connect two data center disaster recovery deployments, follow the extra configuration steps in Using OADP for API Connect two data center disaster recovery instances.

   Queue managers need two labels to be set. For each Queue manager, run this patch command:

   oc patch queuemanager <queue manager name> --type merge --patch '{"metadata":{"labels":{"backup.mq.ibm.com/component":"queuemanager"}},"spec":{"labels":{"backup.mq.ibm.com/component":"nopodbackup"}}}'
   

   You only need to label the Queue managers you created. Queue managers owned by assemblies or Messaging servers should not be labeled and will be restored by their owning resource.

   You must also label any configmaps and secrets that are used to configure the Queue managers:

   oc label configmap <configmap-name> backup.mq.ibm.com/component=configmap
   oc label secret <secret-name> backup.mq.ibm.com/component=secret
   

6. Label the pull secret for your entitlement key. This is the secret that you created if you installed Cloud Pak for Integration in an online environment. For more information, see Installing.

   oc label secret ibm-entitlement-key backup.integration.ibm.com/component=secret
   

   This must be run in each namespace you have deployed instances that use the pull secret.

7. Label the namespaces where you have instances, including the namespace where the foundational services workload is deployed. If you are restoring on a different OpenShift cluster, ensure that the target (restore) cluster does not have any namespaces with the same names you are backing up.

   oc label namespace <your-namespace> backup.integration.ibm.com/component=namespace
   

8. If you are backing up API or API Product Kubernetes resources, label any ConfigMaps and secrets that you create. These ConfigMaps and secrets are referenced when you use API and API Product Kubernetes resources.

   oc label configmap <configmap-name> backup.integration.ibm.com/component=configmap
   oc label secret <secret-name> backup.integration.ibm.com/component=secret
   

   If you are backing up an API or API Product that uses an integration runtime, you must also backup and restore the integration runtime.

9. If you are backing up Enterprise gateway instances, label any secrets that those instances reference, such as secrets that contain user credentials:

   oc label secret <secret-name> backup.integration.ibm.com/component=secret
   

Create and configure the "Backup" Kubernetes resource

OADP uses a Kubernetes resource of kind Backup to specify which resources are backed up and where the backups are stored.

1. Copy the following YAML to create a draft copy for modification:

   apiVersion: velero.io/v1
   kind: Backup
   metadata:
     name: integration-backup
     namespace: openshift-adp
   spec:
     ttl: 720h0m0s
     defaultVolumesToFsBackup: false
     includeClusterResources: true
     includedNamespaces:
     - '*'
     orLabelSelectors:
     - matchExpressions:
       - key: backup.integration.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - assetrepository
         - platformnavigator
         - integrationassembly
         - messagingserver
         - messagingqueue
         - messagingchannel
         - messaginguser
         - validatingadmissionpolicy
         - validatingadmissionpolicybinding
         - commonservice
         - secret
         - configmap
         - namespace
     - matchExpressions:
       - key: backup.apiconnect.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - api
         - product
         - apiconnectcluster
         - portalcluster
         - analyticscluster
         - managementcluster
         - gatewaycluster
         - secret
         - configmap
     - matchExpressions:
       - key: backup.datapower.ibm.com/component
         operator: In
         values:
         - catalogsource 
         - operatorgroup
         - subscription
         - datapowerservice
         - secret
         - configmap
     - matchExpressions:
       - key: backup.appconnect.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - configuration
         - dashboard
         - designerauthoring
         - integrationruntime
         - integrationserver
         - switchserver
         - secret
         - configmap
     - matchExpressions:
       - key: backup.mq.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - queuemanager
         - secret
         - configmap
     - matchExpressions:
       - key: backup.eventstreams.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - eventstreams
         - kafkaconnect
         - kafkatopic
         - kafkauser
         - kafkabridge
         - kafkaconnector
         - kafkarebalance
         - secret
         - configmap
     - matchExpressions:
       - key: backup.events.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - eventgateway
     - matchExpressions:
       - key: foundationservices.cloudpak.ibm.com
         operator: In
         values:
         - keycloak
   

   Important: If the DataPower service was deployed as part of API Connect, remove the - datapowerservice line from the section for key: backup.datapower.ibm.com/component in the YAML.
2. (Optional) Modify the matchExpressions sections as needed for your installation:

   • Remove an operator from the backup by removing the matchExpressions section for it. For example, if you don't want to backup any instance that is managed by the IBM App Connect operator, remove the matchExpression section that has a key value of backup.appconnect.ibm.com/component.
   • Add or remove label values to back up components within an operator. For example, if you want to exclude operators because you back them up in a different way, remove the catalogsource, operatorgroup, and subscription label values. If you added your own labels with custom text, ensure that those labels are present.

   A matchExpressions section for the IBM App Connect operator might look like this:

   - matchExpressions:
     - key: backup.appconnect.ibm.com/component
       operator: In
       values:
       - my-custom-label
       - catalogsource
       - operatorgroup
       - subscription
       - dashboard
       - designerauthoring
       - integrationruntime
       - integrationserver
       - secret
       - configmap
   

   Match expressions are useful if you installed multiple instances into a single namespace and you want to backup a subset of them. For more information about matchExpressions, see Resources that support set-based requirements in the Kubernetes documentation.

3. When your updates are complete, apply the modified YAML to create the Backup Kubernetes resource in your cluster. The backup process starts when you create the resource, and runs in the background, avoiding disruption to your system. You can create the resource by using the Red Hat OpenShift console or the CLI.

   Using the OpenShift web console:

   1. Click the Plus (Import YAML) icon to open the YAML editor.
   2. Paste the modified YAML into the editor.
   3. Click Create. The Backup Kubernetes resource is created in the namespace that is listed in the YAML.

   Using the CLI:

   1. Save your modified YAML in a text file.

   2. Log in to your OpenShift cluster by running the oc login command.

   3. Set the OADP namespace (openshift-adp by default) as the default namespace by running the following command:

      oc project openshift-adp
      

   4. Apply the YAML by running the following command, where <filename> is the name of the file that you saved:

      oc apply -f <filename>.yaml
      

   When the backup completes successfully, the status of your backup instance under the Backup tab of the OADP operator is completed. You can also view the status by running the following command:

   oc get backup.velero.io <integration_backup> -n openshift-adp -o jsonpath='{.status.phase}'
   

4. For a new backup, or if you modified your system since your last backup, test the backup to ensure that you can successfully recover your system if a failure occurs. To test the backup, restore it into a new OpenShift cluster while your primary system is running, then check that your restored instances work as expected.

Create and configure the "Restore" Kubernetes resource

OADP uses a Kubernetes resource of kind Restore to specify which resources are restored and where from.

1. Copy the following YAML to create a draft copy for modification:

   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: integration-restore
     namespace: openshift-adp
   spec:
     backupName: <integration-backup>
     includeClusterResources: true
     existingResourcePolicy: update
     restorePVs: true
     restoreStatus:
       includedResources:
       - apis.apiconnect.ibm.com
       - products.apiconnect.ibm.com
       - integrationkeycloakclients.keycloak.integration.ibm.com
       - integrationkeycloakusers.keycloak.integration.ibm.com
     hooks: {}
     includedNamespaces:
     - '*'
     itemOperationTimeout: 1h0m0s
     orLabelSelectors:
     - matchExpressions:
       - key: backup.integration.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - assetrepository
         - platformnavigator
         - integrationassembly
         - messagingserver
         - messagingqueue
         - messagingchannel
         - messaginguser
         - validatingadmissionpolicy
         - validatingadmissionpolicybinding
         - commonservice
         - secret
         - configmap
         - namespace
     - matchExpressions:
       - key: backup.apiconnect.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - api
         - product
         - apiconnectcluster
         - portalcluster
         - analyticscluster
         - managementcluster
         - secret
         - configmap
     - matchExpressions:
       - key: backup.datapower.ibm.com/component
         operator: In
         values:
         - catalogsource 
         - operatorgroup
         - subscription
         - datapowerservice
         - secret
         - configmap
     - matchExpressions:
       - key: backup.appconnect.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - configuration
         - dashboard
         - designerauthoring
         - integrationruntime
         - integrationserver
         - switchserver
         - secret
         - configmap
     - matchExpressions:
       - key: backup.mq.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - queuemanager
         - secret
         - configmap
     - matchExpressions:
       - key: backup.eventstreams.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - eventstreams
         - kafkaconnect
         - kafkatopic
         - kafkauser
         - kafkabridge
         - kafkaconnector
         - kafkarebalance
         - secret
         - configmap
     - matchExpressions:
       - key: backup.events.ibm.com/component
         operator: In
         values:
         - catalogsource
         - operatorgroup
         - subscription
         - eventgateway
     - matchExpressions:
       - key: foundationservices.cloudpak.ibm.com
         operator: In
         values:
           - keycloak
   

   Important: If the DataPower service was deployed as part of API Connect, remove the - datapowerservice line from the section for key: backup.datapower.ibm.com/component in the YAML.
2. Replace <integration-backup> with the name of the backup to restore. If you are restoring into a new OpenShift cluster that you prepared as described earlier, you can view a list of all the backups that were created in your old cluster, on the Backup tab of the OADP operator. You can also list the backups by running this command:

   oc get backups.velero.io -n openshift-adp

3. (Optional) Edit the matchExpressions sections in the YAML code to add or remove resources that you want to restore, as you did for the Backup Kubernetes resource.

   Important: Some resources have dependencies on others, so even if you don't want to restore a resource, you might need to restore it so that dependent resources can start successfully after the restore operation. For example, the Integration dashboard (dashboard) and Integration design (designerauthoring) instances require the IBM Cloud Pak for Integration operator (subscription under backup.integration.ibm.com/component).

4. Use your updated YAML to create the Restore Kubernetes resource in your OpenShift cluster, in the same way that you created the Backup Kubernetes resource. The restore process starts when you create the resource. When the restore completes successfully, the status of your restore instance in the Restore tab of the OADP operator is completed. You can also view the status by running the following command:

   oc get restore.velero.io -n openshift-adp <integration-restore> -o jsonpath='{.status.phase}'
   

   Tip: Some objects, such as pods, might remain in an error state until the associated operator recreates all the related resources (such as configmaps or secrets).

Check the restored backup

Verify that the expected Cloud Pak for Integration resources are restored and functioning correctly. For example:

• Platform UI: Check that you can access the user interface.
• Assembly: Check that the IntegrationAssembly Kubernetes resource has a status of Ready.
• Automation assets: Check that you can see the expected assets in the restored instance.
• Enterprise gateway: Check that the DataPowerService Kubernetes resource has a status of Ready. OADP backs up and restores only the stateless components of the instance.

If you are using the backup to test your preparedness for a failure, your primary system should still be running. Some instances might require further configuration to test in this scenario. For example:

• Kafka cluster: Because of the georeplication feature, you should be able to see information coming from the primary cluster.
• Integration runtime: Use DNS router configuration to route data from the primary cluster to the test cluster, to imitate a failover situation and check that the restored backup in the new OpenShift cluster works as expected.