Backing up and restoring for EDB Postgres

The following example explains how to back up and restore EnterpriseDB (EDB) Postgres data with Rook Ceph®.

About this task

The following table describes what configuration parameters are used for backups in the postgresql section of the Netcool® Operations Insight® Custom Record definition.

Section name	Property name	Description	Default value
`postgresql.bootstrap`	`enabled`	Use this property to determine whether to bootstrap a new cluster from a preexisting backup.	`false`
`postgresql.bootstrap`	`clusterName`	This property is the name of the cluster to bootstrap from. The cluster must include an existing backup that is located in your `destinationPath` property. For example, if you had a previously running cluster that successfully took backups before, the value to provide for this property is what you originally provided in `spec.postgresql.backups.serverName`.	`evtmanager-noi-postgres-cluster`
`postgresql.bootstrap`	`destinationPath`	This property follows the `"s3://${BUCKET_NAME}"` format.	`s3://ceph-bkt-18d99a17-38ee-4798-accb-a39077bd1abd`
`postgresql.bootstrap`	`endpointURL`	This example property uses node port.	`http://worker0.detrayer.cp.xyz.com:32252`
`postgresql.bootstrap`	`# endpointURL`	If you use a local S3 bucket, the example property is `http://$AWS_HOST:$AWS_PORT`.	`http://rook-ceph-rgw-my-store.rook-ceph.svc.8080`
`postgresql.bootstrap.s3Credentials`	`secretName`	This property is the name of the secret that contains the relevant S3 credentials.	`ceph-bucket`
`postgresql.bootstrap.s3Credentials`	`keyNameAccessKeyID`	This property is the name of the key in the secret with a value that matches the access key ID.	`AWS_ACCESS_KEY_ID`
`postgresql.bootstrap.s3Credentials`	`keyNameAccessSecretKey`	This property is the name of the key in the secret with a value that matches the access secret key.	`AWS_SECRET_ACCESS_KEY`
`postgresql.bootstrap.s3Credentials`	`keyNameAccessSessionToken`	This optional property is the name of the key in the secret with a value that matches the access session token.
`postgresql.bootstrap.wal`	`walMaxParallel`	This property indicates the number of jobs to use when bootstrapping the cluster. This property has bandwidth implications.	`1`
`postgresql.bootstrap.wal`	`encryption`	Use the bucket default encryption. Options are `default`, `AES256`, or `aws:kms`.	`default`
`postgresql.bootstrap.wal`	`compression`	Options are `none`, `gzip`, `bzip2`, or `snappy`. Each option has implications for speed and size.	`none`
`postgresql.backups`	`enabled`	Enable or disable backups.	`false`
`postgresql.backups`	`data`	The following settings refer to the actual data in the database and not the Write-Ahead Logging (WAL) files: Use `default` to use the bucket default encryption. Options are `default`, `AES256`, or `aws:kms`. The bucket must support the encryption mode. If you're unsure, use `default`. The `compression` options are `none`, `gzip`, `bzip2`, or `snappy`. Each option has different implications for speed and size. The `jobs` property indicates the number of jobs to use when backing up the Postgres data. This property has bandwidth implications. Use the `encryption` property to use encryption.	`default`
`postgresql.backups`	`destinationPath`	This property is the S3 bucket name. If you use the Backing up and restoring for EDB Postgres guide, it is the value of the `BUCKET_NAME` variable.	`"s3://ceph-bkt-18d99a17-38ee-4798-accb-a39077bd1abd"`
`postgresql.backups`	`endpointURL`	This property is the endpoint URL. If you use the Backing up and restoring for EDB Postgres guide, it is the URL of one of the workers of the cluster that has the S3 bucket hosted on it followed by the external port that is defined in the node port service.	`"http://worker0.destrayer.cp.xyz.com:32252"`
`postgresql.backups`	`# endpointURL`	This example uses an S3 bucket that is co-located on the same cluster as the Netcool Operations Insight installation.	`"http://rook-ceph-rgw-my-store.rook-ceph.svc:8080" # "http://$AWS_HOST:$AWS_PORT"`
`postgresql.backups`	`retentionPolicy`	This property indicates how long to store backups.	`12m`
`postgresql.backups`	`serverName`	This property is the folder name where the backups from the cluster go. If you are bootstrapping from an existing backup, the value that is provided must be distinct from the value that is provided for the `spec.postgresql.bootstrap.clusterName` property. A common convention is to use `restoredCluster` if you are bootstrapping a new cluster from a backup. The new backups from that bootstrapped cluster go into the `restoredCluster` directory in your S3 bucket. If you are not bootstrapping a cluster from an existing backup, meaning if you are creating a new Netcool Operations Insight installation or upgrading to version 1.6.13for the first time, use the `evtmanager-noi-postgres-cluster` value.	`restoredCluster`
`postgresql.backups.s3credentials`	`secretName`	This property is the name of the secret that contains the relevant S3 credentials key.	`ceph-bucket`
`postgresql.backups.s3credentials`	`keyNameAccessKeyID`	This property is the name of the key in the secret that has a value that matches the access ID key.	`AWS_ACCESS_KEY_ID`
`postgresql.backups.s3credentials`	`keyNameAccessSecretKey`	This property is the name of the key in the secret with a value of the access secret key.	`AWS_SECRET_ACCESS_KEY`
`postgresql.backups.s3credentials`	`keyNameAccessSessionToken`	This property is optional and matches the name of the key in the secret with a value of the access session token.
`postgresql.backups.wal`	`encryption`	Use the bucket default encryption with this property. Options are `default`, `AES256`, or `aws:kms`. The bucket must support the encryption mode. If you are unsure, use `default`.	`default`
`postgresql.backups.wal`	`compression`	Options are `none`, `gzip`, `bzip2`, or `snappy`. Each option has implications for speed and size.	`none`
`postgresql.backups.wal`	`walMaxParallel`	Set this property to the number of jobs to use when backing up the WAL. This property has bandwidth implications.	`1`
`postgresql.backups.endpointCA`	`enabled`	Set this property to `true` to use a custom certificate authority (CA) certificate.	`false`
`postgresql.backups.endpointCA`	`name`	This property is the name of the custom CA certificate secret. The secret must include a key that is named `cacert` and that has a value of the Base64-encoded CA certificate.
`postgresql.backups.onetimeBackup`	`enabled`	Enable the taking of a one-time backup. You need to manually clean these properties up when you uninstall.	`true`
`postgresql.backups.scheduledBackup`	`enabled`	Set this property to determine whether to enable taking scheduled backups.	`false`
`postgresql.backups.scheduledBackup`	`immediate`	Set this property to determine whether to start taking backups immediately.	`true`
`postgresql.backups.scheduledBackup`	`schedule`	This property indicates the schedule for backups. It uses the same syntax as Kubernetes job schedules.	`"0 0 0 * * *"`
`postgresql.backups.scheduledBackup`	`suspend`	Set this property to determine whether to suspend the taking of scheduled backups. Set this property to `true` if you want to pause the taking of backups.	`false`
`postgresql.backups.scheduledBackup`	`backupOwnerReference`	Use this property for the `OwnerReference` value for the derivative backup custom resources (CRs) that are created from this `ScheduledBackup` CR. If the property is set to `none`, no `OwnerReference` is used, and you need to manually delete the derivative backup CRs when you uninstall Netcool Operations Insight. If the property is set to `self`, the `OwnerReference` for the `Backup` CRs is the `ScheduledBackup` CR. When one `ScheduledBackup` CR is deleted, all derivative `Backup` CRs are automatically deleted. If the property is set to `cluster`, the `OwnerReference` for the `Backup` CRs is the `Cluster` CR. If the `Cluster` CR is deleted, such as when the Netcool Operations Insight CR is deleted, the `Backup` CRs are automatically deleted.	`none`

Procedure

Install and configure rook-ceph. For more information, see Getting started and Openshift in the Rook Ceph documentation.
Ensure that the rook-ceph operator is running.
```
oc -n rook-ceph get pod
```

Create an Amazon Simple Storage Service (S3) bucket.

To create a CephObjectStore custom resource definition (CRD), enter the following lines into a file, such as a file named CephObjectStore.yaml.

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: my-store
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    erasureCoded:
      dataChunks: 2
      codingChunks: 1
  preservePoolsOnDelete: true
  gateway:
    sslCertificateRef:
    port: 8080
    instances: 1

Then, create the CephObjectStore CRD:
```
oc create -f CephObjectStore.yaml
```

Create the storageclass S3 bucket.

Enter the following lines into a file, such as a file named rook-ceph-bucket.storageclass.yaml.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-bucket
provisioner: rook-ceph.ceph.rook.io/bucket
reclaimPolicy: Delete
parameters:
  objectStoreName: my-store
  objectStoreNamespace: rook-ceph

Then, create the storageclass S3 bucket:

oc create -f rook-ceph-bucket.storageclass.yaml

Create the objectbucketclaim custom resource (CR).

Enter the following lines into a file, such as a file named ObjectBucketClaim.yaml.

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: ceph-bucket
  namespace: netcool
spec:
  generateBucketName: ceph-bkt
  storageClassName: rook-ceph-bucket

Replace netcool with your namespace name.

Then, create the objectbucketclaim CR:
```
oc create -f ObjectBucketClaim.yaml
```

Create a node port service to get access externally.

If you create this S3 bucket on a cluster that is external to the IBM® Netcool Operations Insight installation, create a node port to show the S3 bucket. Enter the following lines into a file, such as a file named rook-ceph-rgw-my-store-external.yaml.

apiVersion: v1
kind: Service
metadata:
  name: rook-ceph-rgw-my-store-external
  namespace: rook-ceph
  labels:
    app: rook-ceph-rgw
    rook_cluster: rook-ceph
    rook_object_store: my-store
spec:
  ports:
  - name: rgw
    nodePort: 32252
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: rook-ceph-rgw
    rook_cluster: rook-ceph
    rook_object_store: my-store
  sessionAffinity: None
  type: NodePort

Then, create the node port:

oc create -f rook-ceph-rgw-my-store-external.yaml

Use a file system tool, such as s5cmd, to test S3 bucket access. Search s5cmd releases to choose a release, or download and install the release in the following example:
```
wget https://github.com/peak/s5cmd/releases/download/v2.0.0/s5cmd_2.0.0_Linux-64bit.tar.gz
tar -xzf s5cmd_2.0.0_Linux-64bit.tar.gz
sudo mv s5cmd /usr/local/bin
```

Configure your file system tool to access the S3 bucket. This example uses s5cmd.

Enter the following commands:

export AWS_HOST=$(oc -n netcool get cm ceph-bucket -o jsonpath='{.data.BUCKET_HOST}')
export PORT=$(oc -n netcool get cm ceph-bucket -o jsonpath='{.data.BUCKET_PORT}')
export BUCKET_NAME=$(oc -n netcool get cm ceph-bucket -o jsonpath='{.data.BUCKET_NAME}')
export AWS_ACCESS_KEY_ID=$(oc -n netcool get secret ceph-bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode)
export AWS_SECRET_ACCESS_KEY=$(oc -n netcool get secret ceph-bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode)

Replace netcool with your namespace name.

Then, create the S3 credential file:

mkdir ~/.aws
cat > ~/.aws/credentials << EOF
[default]
aws_access_key_id = ${AWS_ACCESS_KEY_ID}
aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}
EOF

Test the S3 bucket.

Enter the following command:

echo "Hello Rook" > /tmp/rookObj
s5cmd --endpoint-url http://$AWS_HOST:$PORT cp /tmp/rookObj s3://$BUCKET_NAME

List, download, and read the previously created file. If you can't upload the file, list the file, download the file, or concatenate the file, the S3 bucket might not be configured correctly.
```
s5cmd --endpoint-url http://$AWS_HOST:$PORT ls s3://$BUCKET_NAME/*
```

Download the previously written file:

s5cmd --endpoint-url http://$AWS_HOST:$PORT cp s3://$BUCKET_NAME/rookObj /tmp/rookObj-download

Read the local file that you downloaded from the S3 bucket:
```
cat /tmp/rookObj-download
```

Delete the file:

s5cmd --endpoint-url http://$AWS_HOST:$PORT rm s3://$BUCKET_NAME/rookObj

If you use a node port service, test the S3 bucket with these commands instead:

Enter the following command:

echo "Hello Rook" > /tmp/rookObj
s5cmd --endpoint-url http://k8s-worker-name-here:32252 cp /tmp/rookObj s3://$BUCKET_NAME

List, download, and read the previously created file. If you can't upload the file, list the file, download the file, or concatenate the file, the S3 bucket might not be configured correctly.
```
s5cmd --endpoint-url http://k8s-worker-name-here:32252 ls s3://$BUCKET_NAME/*
```

Download the previously written file:

s5cmd --endpoint-url http://k8s-worker-name-here:32252 cp s3://$BUCKET_NAME/rookObj /tmp/rookObj-download

Read the local file that you downloaded from the S3 bucket:
```
cat /tmp/rookObj-download
```

Delete the file:

s5cmd --endpoint-url http://k8s-worker-name-here:32252 rm s3://$BUCKET_NAME/rookObj

Use the S3 bucket in your Netcool Operations Insight CR.

If you are taking backups for the first time, run the following script:

apiVersion: noi.ibm.com/v1beta1
kind: NOI
metadata:
  name: evtmanager
spec:
  version: 1.6.13
  license:
    accept: true
...
...
...
  postgresql:
    backups:
      enabled: true
      data: # the following settings refer to the actual data in the database, NOT the WAL files
        encryption: "default" # use the bucket default encryption, options are default, AES256, or aws:kms; the bucket must support the encryption mode; if unsure, use "default"
        compression: "none" # options are none, gzip, bzip2, or snappy, different implications for speed and size
        jobs: 1 # the number of jobs to use when backuping up the postgres data, bandwidth implications
      destinationPath: "s3://your-bucket-here-a39077bd1abd" # the s3 bucket name; the value of the BUCKET_NAME variable exported above
      endpointURL: "http://your-cluster-worker-name-here0:32252" # the endpoint URL; if using the quickstart guide, the url of one of the workers of the cluster that has the s3 bucket hosted on it, followed by the external port defined in the node port service
      retentionPolicy: "12m"
      serverName: "evtmanager-noi-postgres-cluster"
      s3Credentials:
        secretName: "ceph-bucket"
        keyNameAccessKeyID: "AWS_ACCESS_KEY_ID"
        keyNameAccessSecretKey: "AWS_SECRET_ACCESS_KEY"
        keyNameAccessSessionToken: ""
      wal:
        encryption: "default"
        compression: "none"
        walMaxParallel: 1
      endpointCA:
        enabled: false
        name: ""
      onetimeBackup:
        enabled: false
      scheduledBackup:
        enabled: true
        immediate: true
        schedule: "0 0 0 * * *"
        suspend: false
        backupOwnerReference: "none"

Replace evtmanager with your release name.
The destinationPath format is "s3://<$BUCKET_NAME>", for example: "s3://ceph-bkt-34ae25b7-4f08-40bb-ac5b-2c600c78926d"
The endpointURL value is your cluster worker name and port or nodePort, for example: http://worker0.o1-701948.xyz.com:32252/
Get the serverName by running the following command.
```
oc get pod -n noi-on-ocp |grep noi-postgres
```

Check whether the backups succeeded:

oc get backup -n netcool

If the backup was successful, you receive an output similar to the following example:

<username>@21:48:22:[~/workspace/noi-on-icp]$ oc get Backup -n netcool
NAME                                AGE    CLUSTER                           PHASE       ERROR
evtmanager-noi-onetime-edb-backup   6d5h   evtmanager-noi-postgres-cluster   completed

If the backup was not successful, enter the following command to obtain more information about the issue:
```
oc describe Backup evtmanager-noi-onetime-edb-backup -n netcool
```

Restore from the backup.

Delete your existing Netcool Operations Insight CR, wait for all pods and associated resources in your netcool namespace to be deleted, and create a new Netcool Operations Insight CR for Postgres with the following script:

apiVersion: noi.ibm.com/v1beta1
kind: NOI
metadata:
  name: evtmanager
spec:
  version: 1.6.13
  license:
    accept: true
...
...
...
  postgresql:
    bootstrap:
      enabled: true
      clusterName: "evtmanager-noi-postgres-cluster"
      destinationPath: "s3://your-bucket-here-a39077bd1abd" # the s3 bucket name; the value of the BUCKET_NAME variable exported above
      endpointURL: "http://your-cluster-worker-name-here0:32252" # the endpoint URL; if using the quickstart guide, the url of one of the workers of the cluster that has the s3 bucket hosted on it, followed by the external port defined in the node port service
      s3Credentials:
        secretName: "ceph-bucket"
        keyNameAccessKeyID: "AWS_ACCESS_KEY_ID"
        keyNameAccessSecretKey: "AWS_SECRET_ACCESS_KEY"
        keyNameAccessSessionToken: ""
      wal:
        walMaxParallel: 1 # the number of jobs to use when bootstrapping the cluster, bandwidth implications
        encryption: "default"
        compression: "none"
    backups:
      enabled: true
      data:
        encryption: "default"
        compression: "none"
        jobs: 1
      destinationPath: "s3://your-bucket-here-a39077bd1abd" # the s3 bucket name; the value of the BUCKET_NAME variable exported above
      endpointURL: "http://your-cluster-worker-name-here0:32252" # the endpoint URL; if using the quickstart guide, the url of one of the workers of the cluster that has the s3 bucket hosted on it, followed by the external port defined in the node port service
      retentionPolicy: "12m"
      serverName: "restoredCluster" # this will be the folder that the backups from the bootstrapped cluster will go; the pre-existing backups that we took before will still be in the "evtmanager-noi-postgres-cluster" folder
      s3Credentials:
        secretName: "ceph-bucket"
        keyNameAccessKeyID: "AWS_ACCESS_KEY_ID"
        keyNameAccessSecretKey: "AWS_SECRET_ACCESS_KEY"
        keyNameAccessSessionToken: ""
      wal:
        encryption: "default"
        compression: "none"
        walMaxParallel: 1
      endpointCA:
        enabled: false
        name: ""
      onetimeBackup:
        enabled: true
      scheduledBackup:
        enabled: true
        immediate: true
        schedule: "0 0 0 * * *"
        suspend: false
        backupOwnerReference: "none"

Enter the following command:
```
oc get cluster -n netcool
```
The bootstrapped cluster appears. After the bootstrap process completes, it is indistinguishable from a new cluster.