Preparing the management subsystem for disaster recovery using S3 backups

You can prepare a Management subsystem for disaster recovery from S3 backups by taking specific steps before and after the installation of the subsystem.

1. Prior to initial installation, configure the Management subsystem for database backups:
   See Configuring backup settings for fresh install of the Management subsystem (10.0.1.1-eus or later).
2. After installation, you must back up essential Kubernetes secrets that are used by the Management subsystem.

   The Kubernetes secrets are created during installation. You must save these in case a restore is needed after a disaster has occurred. At that time, you will use them in the setup of the restored Management subsystem:

   1. Obtain and save the Management database encryption secret.

      Use the following command to find the name of the secret in the status of the Management Subsystem. Replace <namespace> with the namespace used for the subsystem installation:

      kubectl get mgmt -n <namespace> -o yaml | grep encryption

      The output of this command shows the name of the Kubernetes secret storing the database encryption key, for example:

      encryptionSecret: management-enc-key

      In this case, the name of the Management database encryption secret is management-enc-key. Use the following command to back up this secret locally:

      kubectl get secret management-enc-key -n <namespace> -o yaml --export > management_enc_key.yaml

      The secret is stored in a file called management_enc_key.yaml, located in the present working directory.

   2. Obtain and save the Management client application credential secrets.
      1. Use the following command to find the names of the secrets in the status of the Management subsystem:

         kubectl get mgmt -n <namespace> -o yaml | grep CredentialSecret

         The output of this command shows the names of the Kubernetes Secrets used to store the various client application credentials. For example:

         atmCredentialSecret: management-atm-cred
         consumerToolkitCredentialSecret: management-ccli-cred
         consumerUICredentialSecret: management-cui-cred
         designerCredentialSecret: management-dsgr-cred
         juhuCredentialSecret: management-juhu-cred
         toolkitCredentialSecret: management-cli-cred
         uiCredentialSecret: management-ui-cred

         Here, for example, the name of the ATM Client Application Credential Secret is management-atm-cred.

      2. Next, back up all of the secrets locally. Use the following command for each listed Credential Secret, replacing <secret_name> with the secret name listed each time:

         kubectl get secret <secret_name> -n <namespace> -o yaml --export > <secret_name>.yaml

         For example, use the following command to backup the ATM Client Credential Secret:

         kubectl get secret management-atm-cred -n <namespace> -o yaml --export > management-atm-cred.yaml

      3. After you save each of the client application Credential Secrets locally, open each of the YAML files saved for each secret in turn. Remove both the ownerReferences subsection and the selfLink property. Re-save the file.

         For example, the ownerReferences and selfLink properties to be removed appear in the YAML files similar to the following:

         ownerReferences:
           - apiVersion: management.apiconnect.ibm.com/v1beta1
             blockOwnerDeletion: true
             controller: true
             kind: ManagementCluster
             name: management
             uid: 623e6b20-7eb8-46ce-94ac-6b64cd71afc4
           selfLink: /api/v1/namespaces/default/secrets/management-atm-cred

   3. Ensure that all of the YAML files that contain the various backed-up Secrets are stored persistently and safely.
      Important: If the files are lost, you cannot restore after a disaster event.
   4. Take note of the following values in the Management subsystem CR. You will need them to restore the Management subsystem.

      Table 1. Management subsystem CR settings

      Setting	Example value
      name	management
      siteName	82b290a2
      Version 10.0.1.4-ifix1-eus or later originalUID	fa0f6f49-b931-4472-b84d-0922a9a92dfd Version 10.0.1.4-ifix1-eus or later: During restore, if you do not specify originalUID, the operator automatically sets its value in the new CR to the value of metadata.uid. If the originalUID in the new CR does not match originalUID in the saved CR, the restore will fail.
      Version 10.0.1.2-eus or earlier metadata.uid	If planning to restore on Version 10.0.1.4-ifix1-eus or later: make a note of the subsystem CR metadata.uid. You must use it when performing a restore on v10.0.1.4-ifix1-eus or later, to populate the spec.originalUID value of the CR. Note: If planning to restore on Version 10.0.1.2-eus or earlier, performing a restore might not work if the subsystem CR name exceeds 15 characters.
      databaseBackup.protocol	objstore
      databaseBackup.s3provider	aws
      databaseBackup.host	s3.amazonaws.com/eu-west-1
      databaseBackup.path	my-bucket/mgmt
      databaseBackup.backupCerts	my-custom-s3-ca-cert 10.0.1.2-eus or later
      databaseBackup.backups3URIStyle	my-s3-uri-style 10.0.1.2-eus or later
      databaseBackup.schedule	0 3 * * *

      • The values name and siteName form the name of the database cluster name eg. management-82b290a2-postgres and is needed to synchronize the backups from your old Management subsystem to your new one.

        The siteName is either be a randomly generated alphanumeric ID, or a custom name that was specified by the administrator in the Management Subsystem CR helper YAML file at deploy time. Use the following command to get the siteName:

        kubectl get mgmt -o yaml -n <namespace> | grep siteName

      • Take note of databaseBackupschedule. The schedule will be included during the restoration.
3. Make sure that you have a backup that can be used in case of a disaster event:
   • If scheduled backups were configured as part of Configuring backup settings for fresh install of the Management subsystem (10.0.1.1-eus or later), backups to the specified cloud storage will take place as scheduled.
   • You can also perform a manual backup at any time by creating a manual backup CR. See Generating a manual backup of the management subsystem (10.0.1.1-eus or later).