Backing up your environment

It is important to back up your data so that you can resume work as quickly and effectively as possible.

Before you begin

Note: The IBM Cloud Pak® for Business Automation disaster recovery solution requires backing up and restoring both Cloud Pak for Business Automation components and Cloud Pak foundational services. The Cloud Pak for Business Automation disaster recovery documentation provides backup and restore instructions only for Cloud Pak for Business Automation components.

For working instructions to back up and restore Cloud Pak foundational services, see IBM Cloud Pak foundational services backup and restore.

For all mentions of icp4adeploy in the provided examples, replace it with the value you set for metadata.name in your IBM Cloud Pak for Business Automation custom resource (CR) file.

Before you start to back up your environment, stop your environment to prevent changes in your persistent volumes (PVs) and database. If you do not stop your environment, your PV data and databases might not be backed up properly.

You can scale down all your environment pods to 0 by running the following commands:

oc scale deploy ibm-cp4a-operator --replicas=0
oc scale deploy ibm-pfs-operator  --replicas=0
oc scale deploy ibm-content-operator  --replicas=0
for i in `oc get deploy -o name |grep icp4adeploy`; do oc scale $i --replicas=0; done
for i in `oc get sts -o name |grep icp4adeploy`; do oc scale $i --replicas=0; done

About this task

Tip: Back up each environment in your multiple-zone clusters regularly. The shorter the time in between two backups the less data you can potentially lose. Configure the cert-manager to set up the TLS key and certificate secrets.

Use the following steps to back up Cloud Pak for Business Automation in a multiple-zone environment.

Procedure

Make copies of the Cloud Pak custom resource (CR) files that are used in the primary and secondary environments. The custom resource (CR) file for a secondary environment has a different hostname from the primary environment.

Back up the security definitions in the following table. For more information, see Creating secrets to protect sensitive configuration data.

Table 1. Secrets to back up
Secrets	Example secret name
Cloud Pak for Business Automation secrets	`icp4adeploy-cpe-oidc-secret` `admin-user-details`
Images pull secret. Not present in an airgap environment.	`ibm-entitlement-key`
Lightweight Directory Access Protocol (LDAP) secret	`ldap-bind-secret`
LDAP SSL certificate secret. Required if you enabled SSL connection for LDAP. You must also back up the certificate file.	`ldap-ssl-cert`
Database SSL certificate secret. Required if you enabled an SSL connection for the database. You must also back up the certificate file. For examples of secret names, see Preparing the databases.	If you are using Db2, an example is: `ibm-dba-db2-cacert`
Shared encryption key secret	`ibm-iaws-shared-key-secret`
IBM Business Automation Workflow secret	`ibm-baw-wfs-server-db-secret`
Process Federation Server admin secret	`ibm-pfs-admin-secret`
IBM Business Automation Application secrets	`ibm-aae-app-engine-secret/icp4adeploy-workspace-aae-app-engine-admin-secret`
Resource Registry secret	`icp4adeploy-rr-admin-secret`
Database credential for Document Processing.	`ibm-aca-db-secret`
CP4BA database SSL secret. Required if you enabled SSL connection for the CP4BA database.	`ibm-cp4ba-db-ssl-secret-for-<dbServerAlias>`
The Automation Document Processing secret is configured in preparation for use with document processing.	`ibm-adp-secret`
The IBM Github secret contains a certificate to secure a connection to the required Git server.	`git-tls-secret`
IBM Business Automation Navigator secret	`ibm-ban-secret`
IBM FileNet® Content Manager secret	`ibm-fncm-secret`
IBM Business Automation Studio secret	`ibm-bas-admin-secret`
Application Engine playback server secret	`ibm-playback-server-admin-secret`
IBM Workflow Process Service Runtime admin secret	`<cr_name>-wfps-admin-secret`

Backup your configmaps if required, you may have modified configmap specific to your use cases

Back up your PVC definitions and PV definitions based on your type of provisioning:

If you are using static provisioning, back up your PVC definitions, PV definitions, and the content in the PV.

If you are using dynamic provisioning, the operator creates the PV and PVC definitions automatically, so you need to back up the PVC definition and the content in the PV. To back up the PVC definitions, get each definition and modify the format so that the PVC can be deployed. The following sample script gets all the PV and PVC definitions. Reference the list of PVC definitions that are related to their capabilities and remove the ones that you don't need.

#!/bin/sh 
 
NS=ibm-cp4ba
pvcbackup() {
    oc get pvc -n $NS --no-headers=true | while read each
    do
        pvc=`echo $each | awk '{ print $1 }'`

        if [[ "$pvc" == ibm-bts-cnpg*  ]] ; then
          # skip BTS PVCs
          continue 
        fi

        echo "---" >> pvc.yaml
        kubectl get pvc $pvc -o yaml \
            | yq eval 'del(.status, .metadata.finalizers, .metadata.resourceVersion, .metadata.uid, .metadata.annotations, .metadata.creationTimestamp, .metadata.selfLink, .metadata.managedFields, .metadata.ownerReferences, .spec.volumeMode, .spec.volumeName)' -  >> pvc.yaml
    
    done
} 
pvcbackup

Back up the content in the PV. When you restore the environment, you can create the PVC by using the backup definition and by copying the content to the corresponding PV.

The following table lists PVC definitions that you must back up and restore (see the Needs to be backed up or replicated column) if you have them, and others that you might also choose to back up.

Table 2. PVC definitions to back up
Component	Custom resource template persistent volume claim name	Description	Needs to be backed up or replicated
IBM Business Automation Navigator	`icn-asperastore`	IBM Business Automation Navigator storage for Aspera.	No
	`icn-cfgstore`	Business Automation Navigator Liberty configuration.	Yes
	`icn-logstore`	Liberty and Business Automation Navigator logs. Multiple IBM Content Navigator pods write logs here.	No
	`icn-pluginstore`	Business Automation Navigator custom plug-ins.	No
	`icn-vw-cachestore`	Business Automation Navigator storage for the Daeja ViewONE cache.	No
	`icn-vw-logstore`	Business Automation Navigator viewer logs for the Daeja ViewONE.	No

Do not back up the following PVC definitions. If you back up these definitions, you might encounter an error.

data-iaf-system-elasticsearch-es-data-0
iaf-system-elasticsearch-es-snap-main-pvc
ibm-bts-cnpg-bawent-cp4ba-bts-1
user-home-pvc

Depending on the capabilities that you are using, you must back up more PVC definitions. See the following links:

Back up all the content in the PVs. You can choose which files to restore on your environment later. The generated folder names for dynamically provisioned PVs are not static. For example, the folder name might look similar to bawent-cmis-cfgstore-pvc-ctnrs-pvc-e5241e0c-3811-4c0d-8d0f-cb66dd67f672. The folder name is different for each deployment, so you must use a mapping folder to back up the content. The following script can be used to create backups of your PVs.

#!/bin/sh 
 
NS=bawent 
SOURCE_PV_DIR=/home/pv/2400 
BACKUP_PV_DIR=/home/backup 
 
pvbackup() { 
    oc get pvc -n $NS --no-headers=true | while read each 
    do 
        pvc=`echo $each | awk '{ print $1 }'` 
        pv=`echo $each | awk '{ print $3 }'`

        if [[ "$pvc" == ibm-bts-cnpg*  ]] ; then
            # skip BTS PVCs
            continue 
        fi

        if [  -d "$SOURCE_PV_DIR/$NS-$pvc-$pv" ] 
        then 
            echo "copying pv $pv " 
            mkdir -p $BACKUP_PV_DIR/$pvc 
            cp -r -a $SOURCE_PV_DIR/$NS-$pvc-$pv/.  $BACKUP_PV_DIR/$pvc 
            echo "" 
        else 
            echo "NOT FOUND for $pvc" 
        fi 
    done 
} 
 
pvbackup

If you are using IBM Workflow Process Service, make sure to back up PVCs prefixed with datasave.
1. Use the kuberenetes command to get its definition and back up the necessary parts in this definition.
2. Back up files under folder <the_folder_for_datasave_PV>/messaging and keep the user:group information.
Make copies of the following files:
- JDBC drivers that depend on your database type. For more information, see the following links:
- Customized files that you put in the components PV for runtime. For example, customized font files.
- The configuration files that you use to set up your persistent storage, and your database server.
If you have a database, back up the secure definition that is used to store the database username and password, and the configuration files that you used to set up your database server.

If you have a database, back up the data in your database by using your preferred method. The following table shows databases that need to be backed up.

Table 3. Databases that need to be backed up for each capability
Capability	Databases that need to be backed up
IBM Automation Decision Services	MongoDB databases that you are using for the decision designer or the decision runtime.
IBM Automation Document Processing	Engine base database Engine tenant databases
IBM Automation Workstream Services	The Db2®, Oracle, PostgreSQL, or SQL Server database that you are using.
IBM Business Automation Workflow	The Db2, Oracle, PostgreSQL, or SQL Server database that you are using.
IBM FileNet Content Manager	The databases for the Global Configuration Database and your object store.
IBM Operational Decision Manager	Decision Center database Decision Server database Database information can be found under the section `datasource_configuration` of the custom resource file.
IBM Workflow Process Service Authoring	The default EDB PostgreSQL, or your own PostgreSQL database.
IBM Workflow Process Service Runtime	Your embedded or external PostgreSQL database.

To configure backup and recovery for PostgreSQL, see Backup and Recovery.

If you are using Db2, you can complete an online or offline backup by completing the following steps.

Run the following commands to complete an offline backup. If you want to do an online backup, you must also complete this step.

mkdir -p /home/db2inst1/backup/2400 
db2 backup db  TOSDB  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  GCDDB  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  AAEDB  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  ICNDB  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  BAWDB  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  DOCSDB  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  DOSDB  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024
db2 backup db  BASDB  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024
db2 backup db  APPDB  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024
db2 backup db  ADPBASE  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024
db2 backup db  PROJ1  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024
db2 backup db  DEVOS1  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024
db2 backup db  AEOS  to /home/db2inst1/backup/2400  WITH 2 BUFFERS BUFFER 1024

If you want an online backup, complete the following steps.

Enable archival logging for each database in the environment. You can also configure the interval between each backup.

mkdir  -p   /home/db2inst1/archive/TOSDB 
db2 update db cfg for TOSDB using LOGINDEXBUILD  on 
db2 update db cfg for TOSDB using LOGARCHMETH1   disk:/home/db2inst1/archive/TOSDB   
 
mkdir  -p   /home/db2inst1/archive/GCDDB 
db2 update db cfg for GCDDB using LOGINDEXBUILD  on 
db2 update db cfg for GCDDB using LOGARCHMETH1   disk:/home/db2inst1/archive/GCDDB   
 
mkdir  -p   /home/db2inst1/archive/AAEDB 
db2 update db cfg for AAEDB using LOGINDEXBUILD  on 
db2 update db cfg for AAEDB using LOGARCHMETH1   disk:/home/db2inst1/archive/AAEDB   
 
mkdir  -p   /home/db2inst1/archive/ICNDB 
db2 update db cfg for ICNDB using LOGINDEXBUILD  on 
db2 update db cfg for ICNDB using LOGARCHMETH1   disk:/home/db2inst1/archive/ICNDB   
 
mkdir  -p   /home/db2inst1/archive/BAWDB 
db2 update db cfg for BAWDB using LOGINDEXBUILD  on 
db2 update db cfg for BAWDB using LOGARCHMETH1   disk:/home/db2inst1/archive/BAWDB   
 
mkdir  -p   /home/db2inst1/archive/DOCSDB 
db2 update db cfg for DOCSDB using LOGINDEXBUILD  on 
db2 update db cfg for DOCSDB using LOGARCHMETH1   disk:/home/db2inst1/archive/DOCSDB   
 
mkdir  -p   /home/db2inst1/archive/DOSDB 
db2 update db cfg for DOSDB using LOGINDEXBUILD  on 
db2 update db cfg for DOSDB using LOGARCHMETH1   disk:/home/db2inst1/archive/DOSDB 

mkdir  -p   /home/db2inst1/archive/BASDB 
db2 update db cfg for BASDB using LOGINDEXBUILD  on 
db2 update db cfg for BASDB using LOGARCHMETH1   disk:/home/db2inst1/archive/BASDB 

mkdir  -p   /home/db2inst1/archive/APPDB 
db2 update db cfg for APPDB using LOGINDEXBUILD  on 
db2 update db cfg for APPDB using LOGARCHMETH1   disk:/home/db2inst1/archive/APPDB 

mkdir  -p   /home/db2inst1/archive/ADPBASE 
db2 update db cfg for ADPBASE using LOGINDEXBUILD  on 
db2 update db cfg for ADPBASE using LOGARCHMETH1   disk:/home/db2inst1/archive/ADPBASE 

mkdir  -p   /home/db2inst1/archive/PROJ1 
db2 update db cfg for PROJ1 using LOGINDEXBUILD  on 
db2 update db cfg for PROJ1 using LOGARCHMETH1   disk:/home/db2inst1/archive/PROJ1

mkdir  -p   /home/db2inst1/archive/DEVOS1 
db2 update db cfg for DEVOS1 using LOGINDEXBUILD  on 
db2 update db cfg for DEVOS1 using LOGARCHMETH1   disk:/home/db2inst1/archive/DEVOS1

mkdir  -p   /home/db2inst1/archive/AEOS 
db2 update db cfg for AEOS using LOGINDEXBUILD  on 
db2 update db cfg for AEOS using LOGARCHMETH1   disk:/home/db2inst1/archive/AEOS

Terminate your database connections to prevent errors during the backup.
```
db2 force applications all 
```

Complete the online backup by running the following commands.

mkdir -p /home/db2inst1/backup/2400/online
db2 backup db TOSDB online to /home/db2inst1/backup/2400/online 
db2 backup db GCDDB online to /home/db2inst1/backup/2400/online 
db2 backup db AAEDB online to /home/db2inst1/backup/2400/online 
db2 backup db ICNDB online to /home/db2inst1/backup/2400/online 
db2 backup db BAWDB online to /home/db2inst1/backup/2400/online 
db2 backup db DOCSDB online to /home/db2inst1/backup/2400/online 
db2 backup db DOSDB online to /home/db2inst1/backup/2400/online 
db2 backup db BASDB online to /home/db2inst1/backup/2400/online 
db2 backup db APPDB online to /home/db2inst1/backup/2400/online 
db2 backup db ADPBASE online to /home/db2inst1/backup/2400/online 
db2 backup db PROJ1 online to /home/db2inst1/backup/2400/online 
db2 backup db DEVOS1 online to /home/db2inst1/backup/2400/online 
db2 backup db AEOS online to /home/db2inst1/backup/2400/online

If you upgraded from version 21.0.3 to 24.0.0, you need to back up the platform-oidc-credentials secret.
Make copies of the security definitions that are used to protect the configuration data in the primary and secondary environments. Get WLP_CLIENT_ID and WLP_CLIENT_SECRET from the ibm-common-services/platform-oidc-credentials secret on the backup cluster and keep them for later usage.
If you use Business Automation Insights, back up the data.
Business Automation Insights stores data in two different places.
- OpenSearch contains the time series and the summaries.
  For more information, see Taking and restoring snapshots of OpenSearch data.
- Flink contains the state of event processing.
  For more information, see Restarting from a checkpoint or savepoint.
In addition, you are responsible for putting in place backup and restore processes for the Kafka server, which is configured through Cloud Pak for Business Automation.
If necessary, back up the Lightweight Directory Access Protocol (LDAP) files. Different types of LDAP servers have different backup methods. Make sure that the restored data in the LDAP database is the same as the source LDAP.
For IBM Security Directory Server see IBM Security Directory Server backup and restore.
To back up Business Teams Service (BTS), see Backing up and restoring.
Complete the backup procedures for the following components that you configured in your environment.
- IBM Automation Decision Services: Backing up Automation Decision Services
- IBM FileNet Content Manager: Backup and recovery of a container environment
- IBM Business Automation Navigator: Backing up Content Navigator

What to do next

When the back up is complete, scale up your environment pods by running the following commands.

for i in `oc get deploy -o name |grep icp4adeploy`; do echo "  start $i" ; oc scale $i --replicas=1; done
for i in `oc get sts -o name |grep icp4adeploy`; do echo "  start $i" ; oc scale $i --replicas=1; done
echo "  start operators ..."
oc scale deploy ibm-cp4a-operator --replicas=1
oc scale deploy ibm-pfs-operator  --replicas=1
oc scale deploy ibm-content-operator  --replicas=1