Preparing the Management subsystem for disaster recovery on OpenShift

Back up data and essential deployment information so that you can use it to recover your deployment after a disaster.

1. Save a copy of the Management database encryption secret.
   1. Run the following command to get the name of the encryption secret:

      oc get mgmt -n <APIC-namespace> -o yaml | grep encryption

      The output of this command shows the name of the secret that is storing the database encryption key, for example:

      encryptionSecret: management-enc-key

   2. Run the following command to save a local copy of the secret as a YAML file:

      oc get secret management-enc-key -n <APIC-namespace> -o yaml > management-enc-key.yaml

2. Save a copy of the Management client application credential secrets.

   Save the credential secrets (the client_id and client_secret) for client applications provided by the Management subsystem.

   1. Run the following command to get the names of the secrets:

      oc get mgmt -n <APIC-namespace> -o yaml | grep CredentialSecret

      The output of this command shows the names of the secrets used to store each of the client application credentials. For example:

      atmCredentialSecret: production-mgmt-atm-cred
      consumerToolkitCredentialSecret: production-mgmt-ccli-cred
      consumerUICredentialSecret: production-mgmt-cui-cred
      designerCredentialSecret: production-mgmt-dsgr-cred
      governanceCredentialSecret: production-mgmt-governance-cred
      juhuCredentialSecret: production-mgmt-juhu-cred
      toolkitCredentialSecret: production-mgmt-cli-cred
      uiCredentialSecret: production-mgmt-ui-cred

   2. Run the following command to save a local copy of each secret as a YAML file, replacing <secret_name> with the name of the application credential secret:

      oc get secret <secret_name> -n <APIC-namespace> -o yaml > <secret_name>.yaml

   3. Edit each of the saved YAML files and remove both the ownerReferences subsection and the selfLink property, and then save the updated file.

      The information to remove looks like the following example:

      ownerReferences:
        - apiVersion: management.apiconnect.ibm.com/v1beta1
          blockOwnerDeletion: true
          controller: true
          kind: ManagementCluster
          name: management
          uid: 623e6b20-7eb8-46ce-94ac-6b64cd71afc4
        selfLink: /api/v1/namespaces/default/secrets/management-atm-cred

3. Run a backup of the Management subsystem database.
   1. Configure the Management subsystem for backups.
   2. Perform a backup of the Management subsystem.
   3. Get the name and ID of the newest Management backup and save them for reference during a recovery.

      Run the following command to list the Management backups:

      oc get mgmtb -n <APIC-namespace>

      The response looks like the following example; the newest backup is the one with the smallest AGE:

      NAME                  STATUS   ID                 CLUSTER      TYPE   CR TYPE   AGE
      management-3aa3bebf   Ready    20200929-152150F   management   full   record    10h
      management-3c4ca8df   Ready    20200929-115520F   management   full   record    20h
      management-5tbxj      Ready    20200929-224349F   management   full   create    17h
      management-f10af337   Ready    20200925-133703F   management   full   record    5d2h
      management-jtlcd      Ready    20200929-224349F   management   full   create    25h
      management-zxjtz      Ready    20200929-224349F   management   full   create    28h
      

4. Save a copy of the username and password for a Cloud Manager administrator.

   When you recover the deployment, you will need to log in to the Cloud Manager as an administrator using the Cloud Manager User Registry. Any user with Administrator permission defined in the Cloud Manager User Registry can be used. Before backing up the information, verify that you an log in to Cloud Manager using that account.

   If you want to backup the secret that contains the administrator password that was created by the API Connect operator during the original deployment, run the following command to get the secret:

   oc --kubeconfig kubeconfig -n <APIC_namespace> get secret <instance-name>-mgmt-admin-pass -o jsonpath="{.data.password}" | base64 -d

   It is possible that the password was changed at some point and the secret is now out-of-date. Be sure to verify that you can log in to Cloud Manager with the password contained in the secret. If the password is not valid, then make sure you have another Cloud Manager administrator account in the Cloud Manager User Registry, and back up that username and password instead.

5. Copy all of the information that you saved to a secure location.