Backing up your environments

It is important to back up your data so that you can resume work as quickly and effectively as possible.

About this task

Tip: Make regular backups of each environment in your multiple-zone clusters. The shorter the time in between two backups the less data you can potentially lose. Configure the cert-manager to set up the TLS key and certificate secrets.

Use the following steps to back up IBM Cloud Pak® for Business Automation in a multiple-zone environment.

To be able to run backup operations on MongoDB, you must install MongoDB Database Tools.

Procedure

  1. Make copies of the Cloud Pak custom resource (CR) files that are used in the primary and secondary environments.
    The custom resource (CR) file for a secondary environment has a different hostname to the primary environment.
  2. Make copies of the security definitions that are used to protect the configuration data in the primary and secondary environments. Get the WLP_CLIENT_ID and WLP_CLIENT_SECRET from ibm-common-services/platform-oidc-credentials secret on the backup cluster and keep them for later usage.
  3. Make copies of the persistent volumes (PV) and persistent volume claims (PVC) in the primary and secondary environments.
    • PV and PVC definition files.
    • The files stored on the persistent volume (PV).
  4. Back up the PVs, PVCs, and files stored on the PV for datadir-zen-metastoredb.
  5. Back up the secret admin-user-details, which stores the admin password for the zen database.
  6. Back up all databases by using database commands.
  7. If you use Automation Decision Services, back up the secrets and your MongoDB collections.

    By default, the database connection is encapsulated in a secret that is named ibm-dba-ads-mongo-secret.

    The secret ibm-dba-ads-designer-secret encapsulates the encryptionKeys key, which is used to cipher and decipher secrets in Decision Designer.

    The secret ibm-dba-ads-runtime-secret encapsulates the encryptionKeys key, which is used to cipher and decipher secrets in the decision runtime.

    Make a copy of the secrets. To retrieve the secrets, run the following commands:
    kubectl --namespace $ADS_RELEASE_NAMESPACE get secret ibm-dba-ads-designer-secret -o yaml
    kubectl --namespace $ADS_RELEASE_NAMESPACE get secret ibm-dba-ads-runtime-secret -o yaml

    Adapt the name of the secrets if you changed them in the CR with the ads_configuration.decision_designer.admin_secret_name and ads_configuration.decision_runtime.admin_secret_name parameters.

    Recommend the users to synchronize their decision services with a remote Git repository to minimize risk of data loss.

    Decision Designer stores its data in multiple collections in three different databases that are defined by the mongoUri, gitMongoUri, and runtimeMongoUri properties of the MongoDB admin secret. The default name of this secret is ibm-dba-ads-mongo-secret. For more information, see mongo.admin_secret_name in Decision Designer parameters).

    The following Decision Designer collections can be restored from the database that is specified by the mongoUri parameter:
    • Credentials
    • JobHistory
    • Library
    • MachineLearningProvider
    • MachineLearningProviderPerSolution
    • Permission
    • credentials-service.ChangeLog
    • credentials-service.DbVersion
    • run-service.ChangeLog
    • run-service.DbVersion
    • rest-api.ChangeLog
    • rest-api.DbVersion
    The following Decision Designer collections can be restored from the database that is specified by the gitMongoUri parameter:
    • Branch
    • ImportReport
    • Repo
    • Resource
    • Revision
    • StagingResource
    • Tag
    • UserStaging
    • fs.chunks
    • fs.files
    • git-service.ChangeLog
    • git-service.DbVersion

    The database that is specified by the mongoHistoryUri parameter contains non critical data that cannot be restored.

    The decision runtime stores the decision archives either in a Persistent Volume (default) or in an S3 object storage bucket (referenced by the parameter ads_configuration.decision_runtime.archive_storage_type). The metadata are stored in the MongoDB database that is referenced by the runtimeMongoUri parameter. The decision archives store and the MongoDB database must be backed up simultaneously when no decision service management requests take place (no decision archive deployment, for example). All collections of the MongoDB runtime database must be backed up and restored.

    To back up the MongoDB service:
    1. Get the value of the secrets and the name of the MongoDB service.
      DESIGNER_ENCRYPTION_KEYS=$(kubectl get secret --namespace $ADS_RELEASE_NAMESPACE ibm-dba-ads-designer-secret -o jsonpath="{.data.encryptionKeys}" | base64 --decode)
      RUNTIME_ENCRYPTION_KEYS=$(kubectl get secret --namespace $ADS_RELEASE_NAMESPACE ibm-dba-ads-runtime-secret -o jsonpath="{.data.encryptionKeys}" | base64 --decode)
      MONGO_URI=$(kubectl get secret --namespace $ADS_RELEASE_NAMESPACE ibm-dba-ads-mongo-secret -o jsonpath="{.data.mongoUri}" | base64 --decode)
      GIT_MONGO_URI=$(kubectl get secret --namespace $ADS_RELEASE_NAMESPACE ibm-dba-ads-mongo-secret -o jsonpath="{.data.gitMongoUri}" | base64 --decode)
      RUNTIME_MONGO_URI=$(kubectl get secret --namespace $ADS_RELEASE_NAMESPACE ibm-dba-ads-mongo-secret -o jsonpath="{.data.runtimeMongoUri}" | base64 --decode)
      echo "DESIGNER_ENCRYPTION_KEYS: $DESIGNER_ENCRYPTION_KEYS"
      echo "RUNTIME_ENCRYPTION_KEYS: $RUNTIME_ENCRYPTION_KEYS"
    2. If you use the embedded MongoDB instance instead of an external instance (which is recommended in production), you must forward the MongoDB service port and adapt the MONGO_URI, GIT_MONGO_URI, and RUNTIME_MONGO_URI parts of the previous step.
      #Get ADS mongo service name
      export ADS_MONGO_SERVICE=$(kubectl get services --namespace $ADS_RELEASE_NAMESPACE -l 'app.kubernetes.io/name=ads-mongo' --output=jsonpath='{.items[0].metadata.name}')
      MONGO_URI="${MONGO_URI/@$ADS_MONGO_SERVICE/@localhost}"
      GIT_MONGO_URI="${GIT_MONGO_URI/@$ADS_MONGO_SERVICE/@localhost}"
      RUNTIME_MONGO_URI="${RUNTIME_MONGO_URI/@$ADS_MONGO_SERVICE/@localhost}"
      kubectl port-forward --namespace $ADS_RELEASE_NAMESPACE svc/$ADS_MONGO_SERVICE 27017:27017 &
    3. Create directories for the files that you want to back up.
      mkdir adsbackup
    4. Back up the collections to these directories by using the mongodump tool.
      for col_name in Credentials JobHistory Library MachineLearningProvider MachineLearningProviderPerSolution \
                      Permission \ credentials-service.ChangeLog credentials-service.DbVersion \ run-service.ChangeLog run-service.DbVersion \
                      rest-api.ChangeLog rest-api.DbVersion ; do
          mongodump --uri "$MONGO_URI" --tlsInsecure --out=adsbackup/ads-db --collection="$col_name"
      done
      
      for col_name in Branch ImportReport Repo Resource Revision StagingResource Tag UserStaging \
                      fs.chunks fs.files git-service.ChangeLog git-service.DbVersion  ; do
          mongodump --uri "$GIT_MONGO_URI" --tlsInsecure --out=adsbackup/ads-git-db --collection="$col_name"
      done
      
      mongodump --uri "$RUNTIME_MONGO_URI" --tlsInsecure --out=adsbackup/ads-runtime
  8. Back up Business Automation Insights data.
    Business Automation Insights stores data in two different places. In addition, you are responsible for putting in place back-up and restore processes for the Kafka server, which is configured through IBM Automation foundation. The Advanced configuration page gives you access to the IBM Automation foundation documentation.