StatefulSet pods with local storage are stuck in Pending state
StatefulSet pods using local storage remain bound to failed nodes.
Problem
If you are using local storage and a node in the cluster goes down, then StatefulSet pods remain
bound to that node. These pods are unable to restart on another node and are stuck in
Pending state because they have a Persistent Volume on the node that is down.
Resolution
The persistent volumes and persistent volume claims for these pods must be removed to allow the pods to be reassigned to other nodes. There is a script here https://www.ibm.com/support/pages/node/6245762 that can be used to do this.
Run the script with the following command:
./cleanPVCandPV.sh
#4856 Needs triage/work