Configuring S3 backup settings for a fresh install of the Management subsystem on OpenShift or Cloud Pak for Integration

Configure backups for the API Connect Management subsystem that you will deploy in your OpenShift or Cloud Pak for Integration environment.

Before you begin

If the APIConnectCluster instance is already created, do not use these instructions. Instead, add the Management backup configuration by following the instructions in Reconfiguring or adding backup settings after installation of the management subsystem on OpenShift and Cloud Pak for Integration.

About this task

When creating an instance of API Connect, use the advanced configuration settings to configure database backups for the Management subsystem. Beginning with v10.0.2, the Management subsystem supports custom S3 solutions such as MinIO.

Important: For S3 backups of the management subsystem, do not use retention features provided by Cloud-based S3 storage providers. Use of such features can result in periodic deletion of archived backups, which can cause backup corruptions that can cause database restores to fail.
Warning:
When configuring a new management subsystem for S3 backup, the content of the S3 bucket path must be empty. If you want to configure a management subsystem to use the content of an S3 bucket path already used by another management subsystem, you have two options:
  1. Shutdown the postgres database on the existing management subsystem that points to the S3 bucket path before you start the new management subsystem.
  2. In a disaster recovery scenario where you want to restore from the backup of an existing installation: Create a new S3 path and use S3 tools to copy the backup content from the existing management subsystem S3 bucket path, to the new bucket path.

Procedure

  1. Create a backup secret.

    The backup secret is a Kubernetes secret that contains your username and password for your S3 backup database.

    For examples of how to generate the appropriate access key and secret for two of them, see:

    The secret can be created with the following command: 
    oc create secret generic mgmt-backup-secret --from-literal=key='<YOUR ACCESS KEY>' --from-literal=keysecret='<YOUR ACCESS KEY SECRET>' -n <namespace_of_management_subsystem>
  2. Configure the Management backup settings.

    Create the APIConnectCluster installation CR and add the Management backup configuration using one of the following methods:

    • OpenShift web console:
      1. Navigate to Advanced Configuration > ManagementSubsystem > Advanced Configuration > Database Backups > Advanced Configuration
      2. See the on-screen instructions for how to fill in each field. See also Management subsystem settings.

        Note that there is downtime involved when changing S3 protocol backup configuration.

    • IBM Cloud Pak Platform UI:
      1. In the Platform UI for the API Connect cluster instance, select Configuration. On the Form UI tab, set Advanced Options to On.
      2. See the on-screen instructions for how to fill in values for:
        • Credentials
        • Hostname
        • Path
        • Protocol
        • Restart DB (toggle on)
        • S3 Provider
        • Backup Credentials
        • Backup URI Style
        • Schedule (The timezone for backups is that of the node on which the postgres-operator pod is scheduled.)
        • Repo Retention

        For more information, see Management subsystem settings.

    • YAML file: Use Table 1 for guidance while you define the backup settings as shown in the following example.
      spec:
  management:
    databaseBackup:
      protocol: objstore
      s3provider: ibm
      host: s3.eu-gb.cloud-object-storage.appdomain.cloud/eu-standard
      path: apic-backup
      credentials: mgmt-backup-secret
      schedule: "0 3 * * *"
      repoRetentionFull: 2
      Note: The protocol setting was introduced in v10.0.1.1. Do not specify it on v10.0.1.0 or earlier deployments
      Table 1. Backup configuration settings
      Setting Description
      protocol The type of the backup; required in 10.0.1.1-eus and greater. For s3 storage, use objstore.

      When the parameter is not set, the backup type defaults to local storage backup.

      Note: The protocol setting was introduced in v10.0.1.1. Do not specify it on v10.0.1.0 or earlier deployments
      s3Provider Name of the S3 provider to use. You must specify one of the supported values of aws, ibm, or custom. The custom option is supported beginning with 10.0.2 or later. If you use a custom S3 provider, you must also specify the backupCerts.
      Note: The public certificate on the S3 storage provider must be signed by a known certificate authority that is trusted by API Connect. Use of an untrusted authority can cause the following error during backup upload: x509: certificate signed by unknown authority.
      backupCerts The name of a custom certificate that contains your upstream custom S3 CA certificate. Supported beginning with 10.0.2.

      Used only when s3Provider is set to custom. Most of the custom S3 providers are created based on custom CA certificates.

      This field accepts name of the Kubernetes secret containing your upstream custom S3 CA certificate. The key of the secret must be ca.crt and the value is the base64-encoded value of the CA certificate.

      If the server certificate that is presented by your S3/object-store endpoint is signed by a well known CA and includes any intermediate certificates in the chain, then you do not need to provide the CA certificate using this feature because the Analytics subsystem already trusts the server certificate. You can test this by configuring the Analytics backup to your S3/object-store server without providing the CA certificate.

      If you do need to provide the CA certificate then you must provide the entire chain in the ca.crt member of the secret, as follows:
      intermediate-certificate-1
intermediate-certificate-2
. . .
intermediate-certificate-n
root-certificate

      Where the first certificate in the ca.crt member (intermediate-certificate-1) is the issuer of the S3/object-store server certificate, and each subsequent certificate in the ca.crt member is the issuer of the preceding certificate. The root certificate in the ca.crt member must be self-signed.

      If there are no intermediate certificates involved, then the ca.crt member contains only the root certificate.

      The following sample bash script creates the Kubernetes secret:
      cat >customs3ca.yaml <<EOF
apiVersion: v1
data:
  ca.crt: $(base64 <path-to-ca-certificate> | tr -d '\n' )
kind: Secret
metadata:
  name: custom-server-ca
type: generic
EOF

kubectl apply -f customs3ca.yaml -n <namespace>
      backups3URIStyle Indicates whether URIs to your S3 backup should use specify a host or a path value. Supported beginning with 10.0.2.

      Valid values: host and path.

      Used only when s3Provider is set to custom.

      Some custom S3 providers require URI style to be set to path. For example, MinIO supports both host and path style setup. You can create a MinIO S3 server to only accept path style client communications.

      Important: Contact your custom S3 administrator before configuring this field. If not properly configured, an upstream custom S3 can reject connections from the client, such as the API Connect Management subsystem.
      host For objstore type backup, specify S3 endpoint with the corresponding S3 region in the format <S3endpoint>/<S3region>
      path The path to the location of the backup: For objstore backups, the name of your S3 bucket to store backup data. For example, bucket-name/folder. The use of subdirectories in the bucket name is not supported.
      Note: When your deployment includes a management subsystem in two different clusters (with active databases), the two management subsystems cannot use the same s3 bucket name in the database backup configurations. Each management subsystem must use a unique s3 bucket name.

      Ensure that bucket-name/folder is empty. If the folder was previously used for backups, the folder will not be empty, and stanza create might encounter an error.

      Explanation: Once a folder is used, archive.info and backup.info files are created. During stanza creation, the process compares the database version and database system id between the two info files and the current Postgres database cluster. Stanza creation fails if there is a mismatch. To prevent this possibility, remove all files from the folder before configuring.
      credentials For objstore type backups, the name of the Kubernetes secret containing your S3 Access Key / Key Secret
      schedule Cron like schedule for performing automatic backups. The format for the schedule is:
      • * * * * *
      • - - - - -
      • | | | | |
      • | | | | +----- day of week (0 - 6) (Sunday=0)
      • | | | +------- month (1 - 12)
      • | | +--------- day of month (1 - 31)
      • | +----------- hour (0 - 23)
      • +------------- min (0 - 59)

      The timezone for backups is that of the node on which the postgres-operator pod is scheduled.

      There is no default backup schedule set. Be sure to set your backup schedule.

      All scheduled Management subsystem backups are of type full only.
      repoRetentionFull v10.0.3.0 or later - The number of full S3 backups to retain. When the next full backup successfully completes, and the specified number of retained backups is reached, the oldest full backup is deleted. All incremental backups and archives associated with the oldest full backup also expire. Incremental backups are not counted for this setting.

      Applies to both manual backups and scheduled backups.

      Minimum value: 1

      Maximum value: 9999999

      Default: none
  3. Verify that the configuration deploys successfully.

    When you configure Management subsystem backups, the operator runs a stanza-create job. This job creates the stanza (Postgres cluster configuration) on the upstream S3 server, which is used for backup and archive procedures. The job also brings up the necessary pod.

    Check the status of the stanza-create job:
    oc get jobs -n <namespace> | grep stanza
    • On success:
      • The stanza-create job status is 1/1:
        oc get jobs | grep stanza
m1-b722e361-postgres-stanza-create          1/1           5s         127m
      • The stanza-create job shows status: "True" and type: Complete:
        oc get job m1-b722e361-postgres-stanza-create -o yaml

status:
  completionTime: "2021-04-13T18:38:50Z"
  conditions:
    - lastProbeTime: "2021-04-13T18:38:50Z"
      lastTransitionTime: "2021-04-13T18:38:50Z"
      status: "True"
      type: Complete
 startTime: "2021-04-13T18:38:45Z"
 succeeded: 1
      • The stanza-create pod is in Completed state:
        oc get pods | grep stanza
m1-b722e361-postgres-stanza-create-q4fvp                     0/1     Completed   0          127m
      • Exec into the pod:
        oc exec -it <backrest-shared-repo-pod> -- bash
      • The pgbackrest command returns the S3 contents in valid JSON: 
        pgbackrest info --output json --repo1-type s3

"backup":{"held":false}},"message":"ok"}}][pgbackrest@m1-b722e361-postgres-backrest-shared-repo-69cdfdfdd5-rs882 /
    • On failure, see Troubleshooting the stanza-create job for S3 backup configuration on OpenShift and Cloud Pak for Integration.