Backing up your environments

It is important to back up your data so that you can resume work as quickly and effectively as possible.

1. Make copies of the Cloud Pak custom resource (CR) files that are used in the primary and secondary environments.
   The custom resource (CR) file for a secondary environment has a different hostname to the primary environment.
2. Make copies of the security definitions that are used to protect the configuration data in the primary and secondary environments.
3. Make copies of the persistent volumes (PV) and persistent volume claims (PVC) in the primary and secondary environments.
   • PV and PVC definition files.
   • The files stored on the persistent volume (PV).
4. Backup all databases by using database commands.
5. If you use Automation Decision Services, back up the secrets and your MongoDB collections.

   By default, the database connection is encapsulated in a secret that is named ibm-dba-ads-mongo-secret.

   The secret ibm-dba-ads-designer-secret encapsulates the encryptionKeys key, which is used to cipher and decipher secrets in Decision Designer.

   The secret ibm-dba-ads-runtime-secret encapsulates the encryptionKeys key, which is used to cipher and decipher secrets in the decision runtime.

   Make a copy of the secrets. To retrieve the secrets, run the following commands:

   kubectl --namespace $ADS_RELEASE_NAMESPACE get secret ibm-dba-ads-designer-secret -o yaml

   kubectl --namespace $ADS_RELEASE_NAMESPACE get secret ibm-dba-ads-runtime-secret -o yaml

   Adapt the name of the secrets if you changed them in the CR with the ads_configuration.decision_designer.admin_secret_name and ads_configuration.decision_runtime.admin_secret_name parameters.

   Decision Designer stores its data in multiple collections in three different databases. The Repo collection contains the Decision Designer solutions. The solutions must be synchronized with a remote Git repository to be backed up and restored. Also, the list of solutions must be manually re-created after a crash.

   The following Decision Designer collections can be restored:

   • Library (from the database specified by the mongoUri parameter of the secret ibm-dba-ads-mongo-secret)
   • Credentials (from the database specified by the mongoUri parameter)
   • MachineLearningProviderPerSolution (from the database specified by the mongoUri parameter)
   • credentials-service.ChangeLog
   • credentials-service.DbVersion
   • run-service.DbVersion
   • run-service.ChangeLog
   • rest-api.DbVersion
   • rest-api.ChangeLog

   The decision runtime stores its data in the ads-runtime-archive-metadata database. All collections of this database can be restored.

   To make a backup of the MongoDB service.

   1. Get the value of the secrets and the name of the MongoDB service.

      DESIGNER_ENCRYPTION_KEYS=$(kubectl get secret --namespace $ADS_RELEASE_NAMESPACE ibm-dba-ads-designer-secret -o jsonpath="{.data.encryptionKeys}" | base64 --decode)
      RUNTIME_ENCRYPTION_KEYS=$(kubectl get secret --namespace $ADS_RELEASE_NAMESPACE ibm-dba-ads-runtime-secret -o jsonpath="{.data.encryptionKeys}" | base64 --decode)
      MONGO_URI=$(kubectl get secret --namespace $ADS_RELEASE_NAMESPACE ibm-dba-ads-mongo-secret -o jsonpath="{.data.mongoUri}" | base64 --decode)
      GIT_MONGO_URI=$(kubectl get secret --namespace $ADS_RELEASE_NAMESPACE ibm-dba-ads-mongo-secret -o jsonpath="{.data.gitMongoUri}" | base64 --decode)
      RUNTIME_MONGO_URI=$(kubectl get secret --namespace $ADS_RELEASE_NAMESPACE ibm-dba-ads-mongo-secret -o jsonpath="{.data.runtimeMongoUri}" | base64 --decode)
      echo "DESIGNER_ENCRYPTION_KEYS: $DESIGNER_ENCRYPTION_KEYS"
      echo "RUNTIME_ENCRYPTION_KEYS: $RUNTIME_ENCRYPTION_KEYS"

   2. If you use the embedded MongoDB instance instead of an external instance (which is recommended in production), you must forward the MongoDB service port and adapt the MONGO_URI, GIT_MONGO_URI, and RUNTIME_MONGO_URI parts of the previous step.

      #Get ADS mongo service name
      export ADS_MONGO_SERVICE=$(kubectl get services --namespace $ADS_RELEASE_NAMESPACE -l 'app.kubernetes.io/name=ads-mongo' --output=jsonpath='{.items[0].metadata.name}')
      MONGO_URI="${MONGO_URI/@$ADS_MONGO_SERVICE/@localhost}"
      GIT_MONGO_URI="${GIT_MONGO_URI/@$ADS_MONGO_SERVICE/@localhost}"
      RUNTIME_MONGO_URI="${MONGO_RUNTIME_URI/@$ADS_MONGO_SERVICE/@localhost}"

      kubectl port-forward --namespace $ADS_RELEASE_NAMESPACE svc/$ADS_MONGO_SERVICE 27017:27017 &

   3. Create directories for the files that you want to back up.

      mkdir adsbackup

   4. Back up the collections to these directories by using the mongodump tool.

      # Get collections
         mongodump --uri "$GIT_MONGO_URI" --ssl --sslAllowInvalidCertificates --out=adsbackup --collection=Repo
         for col_name in Credentials Library MachineLearningProviderPerSolution \
                         credentials-service.ChangeLog credentials-service.DbVersion \
                         run-service.ChangeLog run-service.DbVersion \
                         rest-api.ChangeLog rest-api.DbVersion ; do
            mongodump --uri "$MONGO_URI" --ssl --sslAllowInvalidCertificates --out=adsbackup --collection="$col_name"
         done
         # not on 21.0.1:
         mongodump --uri "$RUNTIME_MONGO_URI" --ssl --sslAllowInvalidCertificates --out=adsbackup --db=ads-runtime-archive-metadata

      You can also use a Bitnami MongoDB Docker image to do the backup.

      docker run --rm --name mongo -v $(pwd)/adsbackup:/app --net="host" bitnami/mongodb:latest mongodump --uri ${MONGO_URI/@$ADS_MONGO_SERVICE/@localhost} --ssl --sslAllowInvalidCertificates --collection=Repo -o /app
      

6. Back up Business Automation Insights data.
   Business Automation Insights stores data in two different places.

   • Elasticsearch contains the time series and the summaries.

     For more information, see Taking and restoring snapshots of Elasticsearch data.

   • Flink contains the state of event processing.

     For more information, see Restarting from a checkpoint or savepoint.

   In addition, you are responsible for putting in place back-up and restore processes for the Kafka server, which is configured through IBM Automation foundation. The Advanced configuration page gives you access to the IBM Automation foundation documentation.