Resizing a PVC for Postgres on OpenShift by moving content

Resize a Persistent Volume Claim (PVC) for Postgres on OpenShift by creating a new, larger PVC, and then moving content between the PVCs.

Before you begin

This task only applies to the PVC used by Postgres.

About this task

Use this procedure when the storage class does not support volume expansion.

Procedure

  1. Create a new temporary PVC. Ensure that the requested storage size is more than the original PVC. For example:
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: temp-pvc
    spec:
      storageClassName: rook-ceph-block
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
    
  2. Determine the primary Postgres pod.
    oc get pods --selector role=master | grep -v bootstrapselector=role=master
  3. Gracefully shutdown Postgres pods.
    1. Exec into the Postgres primary pod.
      oc exec -it <postgres-pod> -- bash
    2. Pause the Patroni process. Note: you can skip this step if using the n1 (non-production) profile.
      patronictl pause
    3. Stop the Postgres process gracefully.
      pg_ctl stop -D /pgdata/<cluster-name>

      For example:

      • cluster-name is usually the name of the cluster. If primary pod name is m1-1b12bf81-postgres-58dfcc5f97-fsvvt, cluster-name is m1-1b12bf81-postgres.
      • There will be a folder named cluster-name located under /pgdata/.
      • In this example, the command is:
        pg_ctl stop -D /pgdata/m1-1b12bf81-postgres
  4. Determine the PVC linked to the primary Postgres pod, and obtain the claimName of the PVC.
    oc get pod <postgres-pod> -o json | jq '.spec.volumes[] | select ( has ("persistentVolumeClaim"))'
  5. Scale down the Postgres deployment.
    oc get deploy | grep postgres | grep -v bouncer | grep -v bootstrap | grep -v backrest | grep -v operator

    If you are using n3 profile then you find 3 deployments. Scale them down:

    oc scale deploy <deployment-name> --replicas=0
  6. Start a BusyBox pod with the source and destination PVC volumes mounted, to swap the content between PVCs. For example:
    apiVersion: v1
    kind: Pod
    metadata:
      name: swap-pvc
    spec:
      volumes:
        - name: volume-source
          persistentVolumeClaim:
            claimName: m1-mgmt-35817114-postgres-wal
        - name: volume-destination
          persistentVolumeClaim:
            claimName: temp-pvc
      containers:
        - name: debug
          image: busybox
          command: ['sleep', '3600']
          volumeMounts:
            - name: volume-source
              mountPath: /data-source
            - name: volume-destination
              mountPath: /data-destination 
    
  7. Run the following commands to exec to the BusyBox pod, and copy data from the source to the destination:
    oc exec -it <busybox-pod-name> -- sh
    cp -r data-source/ data-destination/
  8. Copy content to the temporary PVC. For example, using the mountPath values shown in Step 6:
    cp -r data-source/ data-destination/
  9. Delete the BusyBox pod.
    oc delete pod <busybox-pod-name>
  10. Delete the original PVC.
    oc delete pvc <claimName>
  11. Create an original PVC with the same name but with increased size. For example:
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: m1-mgmt-35817114-postgres-wal
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
      storageClassName: rook-ceph-block
      volumeMode: Filesystem   
    
  12. Run the following commands to start the BusyBox pod again and copy the content from the destination to the source PVC.
    oc exec -it <busybox-pod> -- sh
    
    cp -r data-destination/ data-source/
  13. Delete the BusyBox pod.
    oc delete pod <busybox-pod>
  14. Delete the temporary PVC that you created for swapping the content.
    oc delete pvc temp-pvc
  15. Scale up the Postgres deployment.
    oc get deploy | grep postgres | grep -v bouncer | grep -v bootstrap | grep -v backrest | grep -v operator

    If you are using the n3 profile then you find 3 deployments. Scale them up:

    oc scale deploy <deployment-name> --replicas=1
  16. If using the n3 profile, resume the Patroni process. If using the n1 profile, skip this step.
    1. Determine the new primary Postgres pod.
      oc get pods --selector role=master | grep -v bootstrapselector=role=master
    2. Exec into the new Postgres primary pod.
      oc exec -it <postgres-pod> -- bash
    3. Resume Patroni.
      patronictl resume