Resizing a PVC for Postgres on OpenShift by moving content

Resize a Persistent Volume Claim (PVC) for Postgres on OpenShift by creating a new, larger PVC, and then moving content between the PVCs.

Before you begin

This task only applies to the PVC used by Postgres.

About this task

Use this procedure when the storage class does not support volume expansion.

Procedure

Create a new temporary PVC. Ensure that the requested storage size is more than the original PVC. For example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: temp-pvc
spec:
  storageClassName: rook-ceph-block
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Determine the primary Postgres pod.

oc get pods --selector role=master | grep -v bootstrapselector=role=master

Gracefully shutdown Postgres pods.
1. Exec into the Postgres primary pod.
```
oc exec -it <postgres-pod> -- bash
```
2. Pause the Patroni process. Note: you can skip this step if using the n1 (non-production) profile.
```
patronictl pause
```
3. Stop the Postgres process gracefully.
```
pg_ctl stop -D /pgdata/<cluster-name>
```
  For example:
  - cluster-name is usually the name of the cluster. If primary pod name is m1-1b12bf81-postgres-58dfcc5f97-fsvvt, cluster-name is m1-1b12bf81-postgres.
  - There will be a folder named cluster-name located under /pgdata/.
  - In this example, the command is:
```
pg_ctl stop -D /pgdata/m1-1b12bf81-postgres
```

Determine the PVC linked to the primary Postgres pod, and obtain the claimName of the PVC.

oc get pod <postgres-pod> -o json | jq '.spec.volumes[] | select ( has ("persistentVolumeClaim"))'

Scale down the Postgres deployment.

oc get deploy | grep postgres | grep -v bouncer | grep -v bootstrap | grep -v backrest | grep -v operator

If you are using n3 profile then you find 3 deployments. Scale them down:

oc scale deploy <deployment-name> --replicas=0

Start a BusyBox pod with the source and destination PVC volumes mounted, to swap the content between PVCs. For example:

apiVersion: v1
kind: Pod
metadata:
  name: swap-pvc
spec:
  volumes:
    - name: volume-source
      persistentVolumeClaim:
        claimName: m1-mgmt-35817114-postgres-wal
    - name: volume-destination
      persistentVolumeClaim:
        claimName: temp-pvc
  containers:
    - name: debug
      image: busybox
      command: ['sleep', '3600']
      volumeMounts:
        - name: volume-source
          mountPath: /data-source
        - name: volume-destination
          mountPath: /data-destination

Run the following commands to exec to the BusyBox pod, and copy data from the source to the destination:
```
oc exec -it <busybox-pod-name> -- sh
```
```
cp -r data-source/ data-destination/
```
Copy content to the temporary PVC. For example, using the mountPath values shown in Step 6:
```
cp -r data-source/ data-destination/
```
Delete the BusyBox pod.
```
oc delete pod <busybox-pod-name>
```
Delete the original PVC.
```
oc delete pvc <claimName>
```

Create an original PVC with the same name but with increased size. For example:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: m1-mgmt-35817114-postgres-wal
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: rook-ceph-block
  volumeMode: Filesystem

Run the following commands to start the BusyBox pod again and copy the content from the destination to the source PVC.
```
oc exec -it <busybox-pod> -- sh
```
```
cp -r data-destination/ data-source/
```
Delete the BusyBox pod.
```
oc delete pod <busybox-pod>
```
Delete the temporary PVC that you created for swapping the content.
```
oc delete pvc temp-pvc
```

Scale up the Postgres deployment.

oc get deploy | grep postgres | grep -v bouncer | grep -v bootstrap | grep -v backrest | grep -v operator

If you are using the n3 profile then you find 3 deployments. Scale them up:

oc scale deploy <deployment-name> --replicas=1

If using the n3 profile, resume the Patroni process. If using the n1 profile, skip this step.
1. Determine the new primary Postgres pod.
```
oc get pods --selector role=master | grep -v bootstrapselector=role=master
```
2. Exec into the new Postgres primary pod.
```
oc exec -it <postgres-pod> -- bash
```
3. Resume Patroni.
```
patronictl resume
```