Preparing the management subsystem for disaster recovery

You can prepare a Management subsystem for disaster recovery from S3 or SFTP backups by taking specific steps before and after the installation of the subsystem.

About this task

  • This task must be performed before any disaster event occurs, and also prior to the installation of a replacement Management subsystem during the recovery process. Best practice is to complete these steps immediately after initial configuration of a management subsystem during your original v10 deployment.
  • Any local backups are presumed to have been lost in the disaster scenario and are non-recoverable in this procedure.
Important: Successful disaster recovery depends on recovery of both the Management subsystem and the Developer Portal subsystem. You must complete preparation steps for both subsystems in order to achieve disaster recovery. If you have to perform a restore, you must complete the restoration of the Management Service first, and then immediately restore the Developer Portal. Therefore, the backups of the Management and Portal must be taken at the same time, to ensure that the Portal sites are consistent with Management database.


  1. Prior to initial installation, configure the Management subsystem for database backups:
  2. After installation, back up essential Kubernetes secrets that are used by the Management subsystem.

    The Kubernetes secrets are created during installation. You must save these in case a restore is needed after a disaster has occurred. At that time, you will use them in the setup of the restored Management subsystem:

    1. Obtain and save the Management database encryption secret.

      Use the following command to find the name of the secret in the status of the Management Subsystem. Replace <namespace> with the namespace used for the subsystem installation:

      kubectl get mgmt -n <namespace> -o yaml | grep encryption

      The output of this command shows the name of the Kubernetes secret storing the database encryption key, for example:

      encryptionSecret: management-enc-key

      In this case, the name of the Management database encryption secret is management-enc-key. Use the following command to back up this secret locally:

      kubectl get secret management-enc-key -n <namespace> -o yaml > management_enc_key.yaml

      The secret is stored in a file called management_enc_key.yaml, located in the present working directory.

    2. Obtain and save the Management client application credential secrets.
      1. Use the following command to find the names of the secrets in the status of the Management subsystem:
        kubectl get mgmt -n <namespace> -o yaml | grep CredentialSecret

        The output of this command shows the names of the Kubernetes Secrets used to store the various client application credentials. For example:

        atmCredentialSecret: management-atm-cred
        consumerToolkitCredentialSecret: management-ccli-cred
        consumerUICredentialSecret: management-cui-cred
        designerCredentialSecret: management-dsgr-cred
        juhuCredentialSecret: management-juhu-cred
        toolkitCredentialSecret: management-cli-cred
        uiCredentialSecret: management-ui-cred

        Here, for example, the name of the ATM Client Application Credential Secret is management-atm-cred.

      2. Next, back up all of the secrets locally. Use the following command for each listed Credential Secret, replacing <secret_name> with the secret name listed each time:
        kubectl get secret <secret_name> -n <namespace> -o yaml > <secret_name>.yaml

        For example, use the following command to backup the ATM Client Credential Secret:

        kubectl get secret management-atm-cred -n <namespace> -o yaml > management-atm-cred.yaml
      3. After you save each of the client application Credential Secrets locally, open each of the YAML files saved for each secret in turn. Remove both the ownerReferences subsection and the selfLink property. Re-save the file.

        For example, the ownerReferences and selfLink properties to be removed appear in the YAML files similar to the following:

          - apiVersion:
            blockOwnerDeletion: true
            controller: true
            kind: ManagementCluster
            name: management
            uid: 623e6b20-7eb8-46ce-94ac-6b64cd71afc4
          selfLink: /api/v1/namespaces/default/secrets/management-atm-cred
    3. Ensure that all of the YAML files that contain the various backed-up Secrets are stored persistently and safely.
      Important: If the files are lost, you cannot restore after a disaster event.
    4. Take note of the following values in the Management subsystem CR. You will need them to restore the Management subsystem.
      Table 1. Management subsystem CR settings
      Setting Used with Protocol Example value
      name S3 and SFTP management
      siteName S3 and SFTP 82b290a2
      S3 and SFTP fa0f6f49-b931-4472-b84d-0922a9a92dfd

      : During restore, if you do not specify originalUID, the operator automatically sets its value in the new CR to the value of metadata.uid. If the originalUID in the new CR does not match originalUID in the saved CR, the restore will fail.

      .databaseBackup.protocol S3 and SFTP only
      • S3: objstore
      • SFTP: sftp
      databaseBackup.s3provider S3 only aws S3 and SFTP
      • SFTP <SFTP-host-name>
      • S3:
      databaseBackup.path S3 and SFTP
      • SFTP <SFTP-path>
      • S3: my-bucket/mgmt
      databaseBackup.schedule S3 and SFTP 0 3 * * *
      databaseBackup.credentials SFTP only secret-containing-SFTP-credentials
      databaseBackup.backupCerts S3 (custom only, v10.0.2.0 or later) my-custom-s3-ca-cert
      databaseBackup.backups3URIStyle S3 (custom only, v10.0.2.0 or later) my-s3-uri-style
      • The values name and siteName form the name of the database cluster name, such as management-82b290a2-postgres and is needed to synchronize the backups from your old Management subsystem to your new one.

        The siteName is either be a randomly generated alphanumeric ID, or a custom name that was specified by the administrator in the Management Subsystem CR helper YAML file at deploy time. Use the following command to get the siteName:

        kubectl get mgmt -o yaml -n <namespace> | grep siteName
      • The and databaseBackup.path settings must be identical to what was configured on your old Management subsystem. This is the location of your old Management subsystem backups from where we will recover from.
      • Take note of databaseBackupschedule. The schedule will be included during the restoration.
  3. Make sure that you have a backup that can be used in case of a disaster event:

What to do next

You should now complete the preparation steps for the Developer Portal subsystem; see Preparing the Developer Portal for disaster recovery.