A StatefulSet application stuck after failover

Troubleshoot a StatefulSet application stuck after failover.

Problem

While relocating to a preferred cluster, DRPlacementControl is stuck reporting PROGRESSION as MovingToSecondary.

Previously, before Kubernetes v1.23, the Kubernetes control plane never cleaned up the PVCs created for StatefulSets. This activity was left to the cluster administrator or a software operator managing the StatefulSets. Due to this, the PVCs of the StatefulSets were left untouched when their Pods are deleted. This prevents Ramen from relocating an application to its preferred cluster.

Resolution
  1. If the workload uses StatefulSets, and relocation is stuck with PROGRESSION as MovingToSecondary, then run:
    oc get pvc -n <namespace>
  2. For each bounded PVC for that namespace that belongs to the StatefulSet, run:
    oc delete pvc <pvcname> -n namespace

    Once all PVCs are deleted, Volume Replication Group (VRG) transitions to secondary, and then gets deleted.

  3. Run the following command:
    oc get drpc -n <namespace> -o wide
    The PROGRESSION reports Completed and the relocation is complete.
    Note: Changing to Completed can take anywhere between a few seconds up to a few minutes.
Result

The workload is relocated to the preferred cluster

BZ reference: [2118270]