Backing up your environment

It is important to back up your data so that you can resume work as quickly and effectively as possible.

Before you begin

Note: The IBM Cloud Pak® for Business Automation disaster recovery solution requires backing up and restoring both Cloud Pak for Business Automation components and Cloud Pak foundational services. The Cloud Pak for Business Automation disaster recovery documentation provides backup and restore instructions only for Cloud Pak for Business Automation components.

For working instructions to back up and restore Cloud Pak foundational services, see IBM Cloud Pak foundational services backup and restore External link opens a new window or tab.

For all mentions of icp4adeploy in the provided examples, replace it with the value you set for metadata.name in your IBM Cloud Pak for Business Automation custom resource (CR) file.

Offline backup

An offline backup gives you a complete and consistent backup of your environment. Before you start an offline backup, stop the environment to prevent changes to persistent volumes (PVs) and databases.

You can scale down all your environment pods to 0 by running the following commands:

oc scale deploy ibm-cp4a-operator --replicas=0
oc scale deploy ibm-pfs-operator  --replicas=0
oc scale deploy ibm-content-operator  --replicas=0
for i in `oc get deploy -o name |grep icp4adeploy`; do oc scale $i --replicas=0; done
for i in `oc get sts -o name |grep icp4adeploy`; do oc scale $i --replicas=0; done
Online backup

An online backup allows you to backup more frequently when you have a short Recovery-Point-Objective (RPO). An online backup relies on database Point-In-Time Recovery (PTIR) to allow a restore that is consistent with the databases and data in persistent volumes.

You do not need to stop or scale down your environment for an online backup. All pods can continue to run, and the system operates normally while the backup is in-progress.

Consider the following points before you schedule your backups.
  • Backup all your databases at the same time or as close as they can possibly be. If your CP4BA capabilities depend on consistency across multiple databases, including external application databases, coordinate the timing of your backups.
  • When you make an online backup, target a time with minimal or no activity to reduce the risk of data inconsistency between the different databases.
  • Database backup features, behavior, and procedures depend on the chosen vendor.

About this task

Tip: Back up each environment in your multiple-zone clusters regularly. The shorter the time in between two backups the less data you can potentially lose. Configure the cert-manager to set up the TLS key and certificate secrets.

If your secrets and configMaps do not have the cp4ba.ibm.com/backup-type=mandatory label, then add it to help identify these resources for back up. The following commands add the label to a secret defined in the "my_secret.yaml" file.

oc label -f - cp4ba.ibm.com/backup-type=mandatory --local -o yaml > my_secret.yaml
oc apply -f my_secret.yaml

Use the following steps to back up Cloud Pak for Business Automation in a multiple-zone environment.

Procedure

  1. Make copies of the Cloud Pak custom resource (CR) files that are used in the primary and secondary environments. The custom resource (CR) file for a secondary environment has a different hostname from the primary environment.
  2. Back up the security definitions in the following table. For more information, see Creating secrets to protect sensitive configuration data.
    Table 1. Secrets to back up
    Secrets Example secret name
    Cloud Pak for Business Automation secrets
    1. icp4adeploy-cpe-oidc-secret
    2. admin-user-details
    Images pull secret. Not present in an airgap environment. ibm-entitlement-key
    Lightweight Directory Access Protocol (LDAP) secret ldap-bind-secret
    LDAP SSL certificate secret. Required if you enabled SSL connection for LDAP. Also back up the certificate file. ldap-ssl-cert
    Database SSL certificate secret. Required if you enabled an SSL connection for the database. Also back up the certificate file. For examples of secret names, see Preparing the databases. If you are using Db2®, an example is: ibm-dba-pg-cacert
    Shared encryption key secret ibm-iaws-shared-key-secret
    IBM Business Automation Workflow secret ibm-baw-wfs-server-db-secret
    Process Federation Server admin secret ibm-pfs-admin-secret
    IBM Business Automation Application secrets ibm-aae-app-engine-secret/icp4adeploy-workspace-aae-app-engine-admin-secret
    Resource Registry secret icp4adeploy-rr-admin-secret
    Database credential for Document Processing. ibm-aca-db-secret
    CP4BA database SSL secret. Required if you enabled SSL connection for the CP4BA database. ibm-cp4ba-db-ssl-secret-for-<dbServerAlias>
    The Automation Document Processing secret is configured in preparation for use with document processing. ibm-adp-secret
    The IBM® GitHub secret contains a certificate to secure a connection to the required Git server. git-tls-secret
    IBM Business Automation Navigator secret ibm-ban-secret
    IBM Content Cortex secret ibm-fncm-secret
    IBM Business Automation Studio secret ibm-bas-admin-secret
    Application Engine playback server secret ibm-playback-server-admin-secret
    IBM Workflow Process Service Runtime admin secret <cr_name>-wfps-admin-secret
  3. Back up all the configMaps that are referenced in the CP4BA deployment CR.
    Table 2. ConfigMaps to back up
    Capability ConfigMaps to back up
    All Cloud Pak for Business Automation deploymnts

    zen5-br-configmap

    common-service-maps

    platform-auth-idp

    ibm-licensing-cloudpaks

    ibm-licensing-products

    ibm-licensing-services

    label-licensing-configmaps

    common-web-ui-config

    cs-onprem-tenant-config
    Decision Intelligence Client Managed Software

    No DICMS-specific configMaps to back up, except if you explicitly create some and reference them in the CR.
    Operational Decision Manager

    No ODM-specific configMaps to back up, except if you explicitly create some and reference them in the CR.
    Workflow, BAS, BAA, IAWS

    No Workflow-specific configMaps to back up, except if you explicitly create some and reference them in the CR.
    IBM Content Cortex

    No FNCM-specific configMaps to back up, except if you explicitly create some and reference them in the CR.
    Business Automation Insights

    No BAI-specific configMaps to back up, except if you explicitly create some and reference them in the CR.

    flink-operator-config

    flink-config-content-insights-engine-flink

    Document Processing

    No ADP-specific configMaps to back up, except if you explicitly create some and reference them in the CR.
  4. Back up your PVC definitions and PV definitions based on your type of provisioning:
    • If you are using static provisioning, back up your PVC definitions, PV definitions, and the content in the PV.
    • If you are using dynamic provisioning, the operator creates the PVC definitions automatically, so you need to back up the PVC definition. To back up the PVC definitions, get each definition and modify the format so that the PVC can be deployed. The following sample script gets all the PVC definitions. Reference the list of PVC definitions that are related to their capabilities and remove the ones that you don't need.
      #!/bin/sh 
 
NS=ibm-cp4ba
pvcbackup() {
    oc get pvc -n $NS --no-headers=true | while read each
    do
        pvc=`echo $each | awk '{ print $1 }'`

        if [[ "$pvc" == ibm-bts-cnpg*  ]] ; then
          # skip BTS PVCs
          continue 
        fi

        echo "---" >> pvc.yaml
        kubectl get pvc $pvc -o yaml \
            | yq eval 'del(.status, .metadata.finalizers, .metadata.resourceVersion, .metadata.uid, .metadata.annotations, .metadata.creationTimestamp, .metadata.selfLink, .metadata.managedFields, .metadata.ownerReferences, .spec.volumeMode, .spec.volumeName)' -  >> pvc.yaml
    
    done
} 
pvcbackup

    The following table lists PVC definitions that you must back up and restore (see the Needs to be backed up or replicated column) if you have them, and others that you might also choose to back up.

    Table 3. PVC definitions to back up
    Component Custom resource template persistent volume claim name Description Needs to be backed up or replicated
    IBM Business Automation Navigator icn-asperastore IBM Business Automation Navigator storage for Aspera®. No
      icn-cfgstore Business Automation Navigator Liberty configuration. Yes
      icn-logstore Liberty and Business Automation Navigator logs. Multiple IBM Content Navigator pods write logs here. No
      icn-pluginstore Business Automation Navigator custom plug-ins. No
      icn-vw-cachestore Business Automation Navigator storage for the Daeja® ViewONE cache. No
      icn-vw-logstore Business Automation Navigator viewer logs for the Daeja ViewONE. No

    Do not back up the following PVC definitions. If you back up these definitions, you might encounter an error.

    • data-iaf-system-elasticsearch-es-data-0
    • iaf-system-elasticsearch-es-snap-main-pvc
    • ibm-bts-cnpg-bawent-cp4ba-bts-1
    • user-home-pvc

    Depending on the capabilities that you are using, you must back up more PVC definitions. See the following links:

  5. Back up all the content in the PVs if you are using dynamic provisioning. When you restore the environment, you can use the backup definition and copy the content to the corresponding PV to create the PVC. You can choose which files to restore on your environment later. The generated folder names for dynamically provisioned PVs are not static. For example, the folder name might look similar to bawent-cmis-cfgstore-pvc-ctnrs-pvc-e5241e0c-3811-4c0d-8d0f-cb66dd67f672. The folder name is different for each deployment, so you must use a mapping folder to back up the content. The following script can be used to create backups of your PVs. 
    #!/bin/sh 
 
NS=bawent 
SOURCE_PV_DIR=/home/pv/2600 
BACKUP_PV_DIR=/home/backup 
 
pvbackup() { 
    oc get pvc -n $NS --no-headers=true | while read each 
    do 
        pvc=`echo $each | awk '{ print $1 }'` 
        pv=`echo $each | awk '{ print $3 }'`

        if [[ "$pvc" == ibm-bts-cnpg*  ]] ; then
            # skip BTS PVCs
            continue 
        fi

        if [  -d "$SOURCE_PV_DIR/$NS-$pvc-$pv" ] 
        then 
            echo "copying pv $pv " 
            mkdir -p $BACKUP_PV_DIR/$pvc 
            cp -r -a $SOURCE_PV_DIR/$NS-$pvc-$pv/.  $BACKUP_PV_DIR/$pvc 
            echo "" 
        else 
            echo "NOT FOUND for $pvc" 
        fi 
    done 
} 
 
pvbackup
  6. If you are using IBM Workflow Process Service, make sure to back up PVCs prefixed with datasave.
    1. Use the kuberenetes command to get its definition and back up the necessary parts in this definition.
    2. Back up files under folder <the_folder_for_datasave_PV>/messaging and keep the user:group information.
  7. Make copies of the following files:
  8. If you have a database, back up the secure definition that is used to store the database username and password, and the configuration files that you used to set up your database server.
  9. If you have a database, back up the data in your database by using your preferred method. The following table shows databases that need to be backed up.
    Table 4. Databases that need to be backed up for each capability
    Capability Databases that need to be backed up
    IBM Decision Intelligence Client Managed Software PostgreSQL databases that you are using for the decision designer or the decision runtime.
    IBM Automation Document Processing
    • Engine base database
    • Engine tenant databases
    IBM Automation Workstream Services The Db2, Oracle, PostgreSQL, or SQL Server database that you are using.
    IBM Business Automation Workflow The Db2, Oracle, PostgreSQL, or SQL Server database that you are using.
    IBM Content Cortex The databases for the Global Configuration Database and your object store.
    IBM Operational Decision Manager
    • Decision Center database
    • Decision Server database

    Database information can be found under the section datasource_configuration of the custom resource file.
    IBM Workflow Process Service Authoring The default CNPG or your own PostgreSQL database.
    IBM Workflow Process Service Runtime Your embedded or external PostgreSQL database.

    To configure backup and recovery for PostgreSQL, see Backup and Recovery External link opens a new window or tab.

    If you are using Db2, you can complete an online or offline backup by completing the following steps.

    Run the following commands to complete an offline backup. If you want to do an online backup, you must also complete this step. 

    mkdir -p /home/db2inst1/backup/2600 
db2 backup db  TOSDB  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  GCDDB  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  AAEDB  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  ICNDB  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  BAWDB  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  DOCSDB  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  DOSDB  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024
db2 backup db  BASDB  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  APPDB  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  ADPBASE  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  PROJ1  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024 
db2 backup db  DEVOS1  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024
db2 backup db  AEOS  to /home/db2inst1/backup/2600  WITH 2 BUFFERS BUFFER 1024

    If you want an online backup, complete the following steps.

    1. Enable archival logging for each database in the environment. You can also configure the interval between each backup. 
      mkdir  -p   /home/db2inst1/archive/TOSDB 
db2 update db cfg for TOSDB using LOGINDEXBUILD  on 
db2 update db cfg for TOSDB using LOGARCHMETH1   disk:/home/db2inst1/archive/TOSDB   
 
mkdir  -p   /home/db2inst1/archive/GCDDB 
db2 update db cfg for GCDDB using LOGINDEXBUILD  on 
db2 update db cfg for GCDDB using LOGARCHMETH1   disk:/home/db2inst1/archive/GCDDB   

mkdir  -p   /home/db2inst1/archive/AAEDB 
db2 update db cfg for AAEDB using LOGINDEXBUILD  on 
db2 update db cfg for AAEDB using LOGARCHMETH1   disk:/home/db2inst1/archive/AAEDB  
 
mkdir  -p   /home/db2inst1/archive/ICNDB 
db2 update db cfg for ICNDB using LOGINDEXBUILD  on 
db2 update db cfg for ICNDB using LOGARCHMETH1   disk:/home/db2inst1/archive/ICNDB   
 
mkdir  -p   /home/db2inst1/archive/BAWDB 
db2 update db cfg for BAWDB using LOGINDEXBUILD  on 
db2 update db cfg for BAWDB using LOGARCHMETH1   disk:/home/db2inst1/archive/BAWDB   
 
mkdir  -p   /home/db2inst1/archive/DOCSDB 
db2 update db cfg for DOCSDB using LOGINDEXBUILD  on 
db2 update db cfg for DOCSDB using LOGARCHMETH1   disk:/home/db2inst1/archive/DOCSDB   
 
mkdir  -p   /home/db2inst1/archive/DOSDB 
db2 update db cfg for DOSDB using LOGINDEXBUILD  on 
db2 update db cfg for DOSDB using LOGARCHMETH1   disk:/home/db2inst1/archive/DOSDB 

mkdir  -p   /home/db2inst1/archive/BASDB 
db2 update db cfg for BASDB using LOGINDEXBUILD  on 
db2 update db cfg for BASDB using LOGARCHMETH1   disk:/home/db2inst1/archive/BASDB 

mkdir  -p   /home/db2inst1/archive/APPDB 
db2 update db cfg for APPDB using LOGINDEXBUILD  on 
db2 update db cfg for APPDB using LOGARCHMETH1   disk:/home/db2inst1/archive/APPDB 

mkdir  -p   /home/db2inst1/archive/ADPBASE 
db2 update db cfg for ADPBASE using LOGINDEXBUILD  on 
db2 update db cfg for ADPBASE using LOGARCHMETH1   disk:/home/db2inst1/archive/ADPBASE 

mkdir  -p   /home/db2inst1/archive/PROJ1 
db2 update db cfg for PROJ1 using LOGINDEXBUILD  on 
db2 update db cfg for PROJ1 using LOGARCHMETH1   disk:/home/db2inst1/archive/PROJ1

mkdir  -p   /home/db2inst1/archive/DEVOS1 
db2 update db cfg for DEVOS1 using LOGINDEXBUILD  on 
db2 update db cfg for DEVOS1 using LOGARCHMETH1   disk:/home/db2inst1/archive/DEVOS1

mkdir  -p   /home/db2inst1/archive/AEOS 
db2 update db cfg for AEOS using LOGINDEXBUILD  on 
db2 update db cfg for AEOS using LOGARCHMETH1   disk:/home/db2inst1/archive/AEOS
    2. Terminate your database connections to prevent errors during the backup. 
      db2 force applications all
    3. Complete the online backup by running the following commands. 
      mkdir -p /home/db2inst1/backup/2600/online
db2 backup db TOSDB online to /home/db2inst1/backup/2600/online 
db2 backup db GCDDB online to /home/db2inst1/backup/2600/online 
db2 backup db AAEDB online to /home/db2inst1/backup/2600/online
db2 backup db ICNDB online to /home/db2inst1/backup/2600/online 
db2 backup db BAWDB online to /home/db2inst1/backup/2600/online 
db2 backup db DOCSDB online to /home/db2inst1/backup/2600/online 
db2 backup db DOSDB online to /home/db2inst1/backup/2600/online 
db2 backup db BASDB online to /home/db2inst1/backup/2600/online 
db2 backup db APPDB online to /home/db2inst1/backup/2600/online 
db2 backup db ADPBASE online to /home/db2inst1/backup/2600/online 
db2 backup db PROJ1 online to /home/db2inst1/backup/2600/online 
db2 backup db DEVOS1 online to /home/db2inst1/backup/2600/online 
db2 backup db AEOS online to /home/db2inst1/backup/2600/online
  10. For each Business Automation Workflow or Workflow Process Service instance which is either federated or where full text search is enabled, you must back up the indexed data of the BPD runtime, the saved search definitions, and the reusable queries as follows:
    Note: Creating backups of case instances indexes is not supported, instead it is recommended that you rebuild case indexes as described in Rebuilding a case index External link opens a new window or tab.
    • For instances running on containers where full text search is enabled, or for instances running on traditional WebSphere® Application Server and where Business Automation Workflow is configured to perform the indexing of BPD tasks and process instances into the Federated Data Repository (FDR) External link opens a new window or tab, you must take regular snapshots of the Opensearch or Elasticsearch index related to the instance BPD runtime, and restore them on the backup environment as documented in the Failover support section of Understanding the federated data repository BPD indexing External link opens a new window or tab.
    • If Process Federation Server is configured to federate a Business Automation Workflow instance running on-premise on traditional WebSphere Application Server, and if the indexing of this instance is configured in Process Federation Server with an <ibmPfs_bpdIndexer>, take regular snapshots of the Opensearch or Elasticsearch index of this federated system by following the procedure described at Backing up and restoring Process Federation Server indexes in the federated data repository External link opens a new window or tab.
    • For standalone Process Federation Server deployments on containers, the Federated Data Repository (FDR) stores federated index data in an OpenSearch or Elasticsearch cluster. Because the snapshot and restore procedures differ between these two products, take regular snapshots by using the procedure that matches your FDR deployment:
      • For the IBM-provided OpenSearch that is deployed internally with Process Federation Server, follow the same _snapshot API steps that are used for Business Automation Insights data. For more information, see Taking and restoring snapshots of OpenSearch data.
      • For an externally managed OpenSearch cluster, see Take and restore snapshots External link opens a new window or tab in the OpenSearch documentation.
      • For an externally managed Elasticsearch cluster, see Snapshot and restore External link opens a new window or tab in the Elasticsearch documentation.
    • For all Business Automation Workflow or Workflow Process Service instances where full text search is enabled or which are federated using Process Federation Server, you must regularly back up the saved search definitions and reusable queries, and restore them on the backup environment. The Process Federation Server API is exposed on the standalone Process Federation Server instance, but an embedded Process Federation Server also exposes its own API on each Business Automation Workflow and Workflow Process Service instance running on containers where full text search is enabled. It is recommended that you perform backups of saved search definitions and reusable queries on all running Process Federation Server instances (standalone Process Federation Server and embedded Process Federation Server instances):
  11. If you use Business Automation Insights, back up the data.
    Business Automation Insights stores data in two different places. In addition, you are responsible for putting in place backup and restore processes for the Kafka server, which is configured through Cloud Pak for Business Automation.
  12. If necessary, back up the Lightweight Directory Access Protocol (LDAP) files. Different types of LDAP servers have different backup methods. Make sure that the restored data in the LDAP database is the same as the source LDAP.
    For IBM Security Directory Server see IBM Security® Directory Server backup and restore External link opens a new window or tab.
  13. To back up Business Teams Service (BTS), see Backing up and restoring External link opens a new window or tab.
  14. Complete the backup procedures for the following components that you configured in your environment.

What to do next

When the back up is complete, scale up your environment pods by running the following commands. 
for i in `oc get deploy -o name |grep icp4adeploy`; do echo "  start $i" ; oc scale $i --replicas=1; done
for i in `oc get sts -o name |grep icp4adeploy`; do echo "  start $i" ; oc scale $i --replicas=1; done
echo "  start operators ..."
oc scale deploy ibm-cp4a-operator --replicas=1
oc scale deploy ibm-pfs-operator  --replicas=1
oc scale deploy ibm-content-operator  --replicas=1