Backing up and restoring for EDB Postgres

The following example explains how to back up and restore EnterpriseDB (EDB) Postgres data with Rook Ceph®.

Procedure

  1. Install and configure rook-ceph. For more information, see Getting started external icon and Openshift external icon in the Rook Ceph documentation.
  2. Ensure that the rook-ceph operator is running.
    oc -n rook-ceph get pod
  3. Create an Amazon Simple Storage Service (S3) bucket.
    1. To create a CephObjectStore custom resource definition (CRD), enter the following lines into a file, such as a file named CephObjectStore.yaml.
      apiVersion: ceph.rook.io/v1
      kind: CephObjectStore
      metadata:
        name: my-store
        namespace: rook-ceph
      spec:
        metadataPool:
          failureDomain: host
          replicated:
            size: 3
        dataPool:
          failureDomain: host
          erasureCoded:
            dataChunks: 2
            codingChunks: 1
        preservePoolsOnDelete: true
        gateway:
          sslCertificateRef:
          port: 8080
          instances: 1
      
    2. Then, create the CephObjectStore CRD:
      oc create -f CephObjectStore.yaml
  4. Create the storageclass S3 bucket.
    1. Enter the following lines into a file, such as a file named rook-ceph-bucket.storageclass.yaml.
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
         name: rook-ceph-bucket
      provisioner: rook-ceph.ceph.rook.io/bucket
      reclaimPolicy: Delete
      parameters:
        objectStoreName: my-store
        objectStoreNamespace: rook-ceph
    2. Then, create the storageclass S3 bucket:
      oc create -f rook-ceph-bucket.storageclass.yaml
  5. Create the objectbucketclaim custom resource (CR).
    1. Enter the following lines into a file, such as a file named ObjectBucketClaim.yaml.
      apiVersion: objectbucket.io/v1alpha1
      kind: ObjectBucketClaim
      metadata:
        name: ceph-bucket
        namespace: netcool
      spec:
        generateBucketName: ceph-bkt
        storageClassName: rook-ceph-bucket
      Replace netcool with your namespace name.
    2. Then, create the objectbucketclaim CR:
      oc create -f ObjectBucketClaim.yaml
  6. Create a node port service to get access externally.
    1. If you create this S3 bucket on a cluster that is external to the IBM® Netcool® Operations Insight® installation, create a node port to show the S3 bucket. Enter the following lines into a file, such as a file named rook-ceph-rgw-my-store-external.yaml.
      apiVersion: v1
      kind: Service
      metadata:
        name: rook-ceph-rgw-my-store-external
        namespace: rook-ceph
        labels:
          app: rook-ceph-rgw
          rook_cluster: rook-ceph
          rook_object_store: my-store
      spec:
        ports:
        - name: rgw
          nodePort: 32252
          port: 8080
          protocol: TCP
          targetPort: 8080
        selector:
          app: rook-ceph-rgw
          rook_cluster: rook-ceph
          rook_object_store: my-store
        sessionAffinity: None
        type: NodePort
      
    2. Then, create the node port:
      oc create -f rook-ceph-rgw-my-store-external.yaml
  7. Use a file system tool, such as s5cmd, to test S3 bucket access. Search s5cmd releases to choose a release, or download and install the release in the following example:
    wget https://github.com/peak/s5cmd/releases/download/v2.0.0/s5cmd_2.0.0_Linux-64bit.tar.gz
    tar -xzf s5cmd_2.0.0_Linux-64bit.tar.gz
    sudo mv s5cmd /usr/local/bin
  8. Configure your file system tool to access the S3 bucket. This example uses s5cmd.
    1. Enter the following commands:
      export AWS_HOST=$(oc -n netcool get cm ceph-bucket -o jsonpath='{.data.BUCKET_HOST}')
      export PORT=$(oc -n netcool get cm ceph-bucket -o jsonpath='{.data.BUCKET_PORT}')
      export BUCKET_NAME=$(oc -n netcool get cm ceph-bucket -o jsonpath='{.data.BUCKET_NAME}')
      export AWS_ACCESS_KEY_ID=$(oc -n netcool get secret ceph-bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode)
      export AWS_SECRET_ACCESS_KEY=$(oc -n netcool get secret ceph-bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode)
      Replace netcool with your namespace name.
    2. Then, create the S3 credential file:
      mkdir ~/.aws
      cat > ~/.aws/credentials << EOF
      [default]
      aws_access_key_id = ${AWS_ACCESS_KEY_ID}
      aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}
      EOF
  9. Test the S3 bucket.
    1. Enter the following command:
      echo "Hello Rook" > /tmp/rookObj
      s5cmd --endpoint-url http://$AWS_HOST:$PORT cp /tmp/rookObj s3://$BUCKET_NAME
    2. List, download, and read the previously created file. If you can't upload the file, list the file, download the file, or concatenate the file, the S3 bucket might not be configured correctly.
      s5cmd --endpoint-url http://$AWS_HOST:$PORT ls s3://$BUCKET_NAME/*
    3. Download the previously written file:
      s5cmd --endpoint-url http://$AWS_HOST:$PORT cp s3://$BUCKET_NAME/rookObj /tmp/rookObj-download
    4. Read the local file that you downloaded from the S3 bucket:
      cat /tmp/rookObj-download
    5. Delete the file:
      s5cmd --endpoint-url http://$AWS_HOST:$PORT rm s3://$BUCKET_NAME/rookObj
  10. If you use a node port service, test the S3 bucket with these commands instead:
    1. Enter the following command:
      echo "Hello Rook" > /tmp/rookObj
      s5cmd --endpoint-url http://k8s-worker-name-here:32252 cp /tmp/rookObj s3://$BUCKET_NAME
    2. List, download, and read the previously created file. If you can't upload the file, list the file, download the file, or concatenate the file, the S3 bucket might not be configured correctly.
      s5cmd --endpoint-url http://k8s-worker-name-here:32252 ls s3://$BUCKET_NAME/*
    3. Download the previously written file:
      s5cmd --endpoint-url http://k8s-worker-name-here:32252 cp s3://$BUCKET_NAME/rookObj /tmp/rookObj-download
    4. Read the local file that you downloaded from the S3 bucket:
      cat /tmp/rookObj-download
    5. Delete the file:
      s5cmd --endpoint-url http://k8s-worker-name-here:32252 rm s3://$BUCKET_NAME/rookObj
  11. Use the S3 bucket in your Netcool Operations Insight CR.
    1. If you are taking backups for the first time, run the following script:
      apiVersion: noi.ibm.com/v1beta1
      kind: NOI
      metadata:
        name: evtmanager
      spec:
        version: 1.6.11
        license:
          accept: true
      ...
      ...
      ...
        postgresql:
          backups:
            enabled: true
            data: # the following settings refer to the actual data in the database, NOT the WAL files
              encryption: "default" # use the bucket default encryption, options are default, AES256, or aws:kms; the bucket must support the encryption mode; if unsure, use "default"
              compression: "none" # options are none, gzip, bzip2, or snappy, different implications for speed and size
              jobs: 1 # the number of jobs to use when backuping up the postgres data, bandwidth implications
            destinationPath: "s3://your-bucket-here-a39077bd1abd" # the s3 bucket name; the value of the BUCKET_NAME variable exported above
            endpointURL: "http://your-cluster-worker-name-here0:32252" # the endpoint URL; if using the quickstart guide, the url of one of the workers of the cluster that has the s3 bucket hosted on it, followed by the external port defined in the node port service
            retentionPolicy: "12m"
            serverName: "evtmanager-noi-postgres-cluster"
            s3Credentials:
              secretName: "ceph-bucket"
              keyNameAccessKeyID: "AWS_ACCESS_KEY_ID"
              keyNameAccessSecretKey: "AWS_SECRET_ACCESS_KEY"
              keyNameAccessSessionToken: ""
            wal:
              encryption: "default"
              compression: "none"
              walMaxParallel: 1
            endpointCA:
              enabled: false
              name: ""
            onetimeBackup:
              enabled: false
            scheduledBackup:
              enabled: true
              immediate: true
              schedule: "0 0 0 * * *"
              suspend: false
              backupOwnerReference: "none"
      • Replace evtmanager with your release name.
      • The destinationPath format is "s3://<$BUCKET_NAME>", for example: "s3://ceph-bkt-34ae25b7-4f08-40bb-ac5b-2c600c78926d"
      • The endpointURL value is your cluster worker name and port or nodePort, for example: http://worker0.o1-701948.xyz.com:32252/
      • Get the serverName by running the following command.
        oc get pod -n noi-on-ocp |grep noi-postgres
    2. Check whether the backups succeeded:
      oc get backup -n netcool
      If the backup was successful, you receive an output similar to the following example:
      <username>@21:48:22:[~/workspace/noi-on-icp]$ oc get Backup -n netcool
      NAME                                AGE    CLUSTER                           PHASE       ERROR
      evtmanager-noi-onetime-edb-backup   6d5h   evtmanager-noi-postgres-cluster   completed
    3. If the backup was not successful, enter the following command to obtain more information about the issue:
      oc describe Backup evtmanager-noi-onetime-edb-backup -n netcool
  12. Restore from the backup.
    1. Delete your existing Netcool Operations Insight CR, wait for all pods and associated resources in your netcool namespace to be deleted, and create a new Netcool Operations Insight CR for Postgres with the following script:
      apiVersion: noi.ibm.com/v1beta1
      kind: NOI
      metadata:
        name: evtmanager
      spec:
        version: 1.6.11
        license:
          accept: true
      ...
      ...
      ...
        postgresql:
          bootstrap:
            enabled: true
            clusterName: "evtmanager-noi-postgres-cluster"
            destinationPath: "s3://your-bucket-here-a39077bd1abd" # the s3 bucket name; the value of the BUCKET_NAME variable exported above
            endpointURL: "http://your-cluster-worker-name-here0:32252" # the endpoint URL; if using the quickstart guide, the url of one of the workers of the cluster that has the s3 bucket hosted on it, followed by the external port defined in the node port service
            s3Credentials:
              secretName: "ceph-bucket"
              keyNameAccessKeyID: "AWS_ACCESS_KEY_ID"
              keyNameAccessSecretKey: "AWS_SECRET_ACCESS_KEY"
              keyNameAccessSessionToken: ""
            wal:
              walMaxParallel: 1 # the number of jobs to use when bootstrapping the cluster, bandwidth implications
              encryption: "default"
              compression: "none"
          backups:
            enabled: true
            data:
              encryption: "default"
              compression: "none"
              jobs: 1
            destinationPath: "s3://your-bucket-here-a39077bd1abd" # the s3 bucket name; the value of the BUCKET_NAME variable exported above
            endpointURL: "http://your-cluster-worker-name-here0:32252" # the endpoint URL; if using the quickstart guide, the url of one of the workers of the cluster that has the s3 bucket hosted on it, followed by the external port defined in the node port service
            retentionPolicy: "12m"
            serverName: "restoredCluster" # this will be the folder that the backups from the bootstrapped cluster will go; the pre-existing backups that we took before will still be in the "evtmanager-noi-postgres-cluster" folder
            s3Credentials:
              secretName: "ceph-bucket"
              keyNameAccessKeyID: "AWS_ACCESS_KEY_ID"
              keyNameAccessSecretKey: "AWS_SECRET_ACCESS_KEY"
              keyNameAccessSessionToken: ""
            wal:
              encryption: "default"
              compression: "none"
              walMaxParallel: 1
            endpointCA:
              enabled: false
              name: ""
            onetimeBackup:
              enabled: true
            scheduledBackup:
              enabled: true
              immediate: true
              schedule: "0 0 0 * * *"
              suspend: false
              backupOwnerReference: "none"
    2. Enter the following command:
      oc get cluster -n netcool
      The bootstrapped cluster appears. After the bootstrap process completes, it is indistinguishable from a new cluster.