A StatefulSet application stuck after failover
Troubleshoot a StatefulSet application stuck after failover.
- Problem
-
While relocating to a preferred cluster, DRPlacementControl is stuck reporting PROGRESSION as MovingToSecondary.
Previously, before Kubernetes v1.23, the Kubernetes control plane never cleaned up the PVCs created for StatefulSets. This activity was left to the cluster administrator or a software operator managing the StatefulSets. Due to this, the PVCs of the StatefulSets were left untouched when their Pods are deleted. This prevents Ramen from relocating an application to its preferred cluster.
- Resolution
-
- If the workload uses StatefulSets, and relocation is stuck with
PROGRESSION as MovingToSecondary, then
run:
oc get pvc -n <namespace>
- For each bounded PVC for that namespace that belongs to the StatefulSet,
run:
oc delete pvc <pvcname> -n namespace
Once all PVCs are deleted, Volume Replication Group (VRG) transitions to secondary, and then gets deleted.
- Run the following
command:
oc get drpc -n <namespace> -o wide
The PROGRESSION reports Completed and the relocation is complete.Note: Changing to Completed can take anywhere between a few seconds up to a few minutes.
- If the workload uses StatefulSets, and relocation is stuck with
PROGRESSION as MovingToSecondary, then
run:
- Result
-
The workload is relocated to the preferred cluster
BZ reference: [2118270]