A namespace is stuck in the Terminating state
Symptom
A namespace is stuck in the Terminating state
Cause
If a Kubernetes API extension is not available, the resources that are managed by the extension cannot be deleted. Failure to delete the API extension causes namespace deletion to fail.
Resolving the problem
Before you begin
kubectl
must be installed.
Obtain the API descriptions that are not deleted
Complete the following steps to get a description of the APIs that are not deleted:
-
View the namespaces that are stuck in a Terminating state:
kubectl get namespaces
-
Find the resources that are not deleted:
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <terminating-namespace>
-
If the previous command returns the following error message:
unable to retrieve the complete list of server APIs: <api-resource>/<version>: the server is currently unable to handle the request
, continue to run the following command with the information you received:kubectl get APIService <version>.<api-resource>
For example, run the following command for an API service that is named
custom.metrics.k8s.io/v1beta1
:kubectl get APIService v1beta1.custom.metrics.k8s.io
-
Get a description of the API service to continue to debug your API service. Run the following command:
kubectl describe APIService <version>.<api-resource>
-
Make sure that the issue is resolved. Run the following command to verify that your namespace can be deleted:
kubectl get namespace
Manually delete a terminating namespace
If the issue is not resolved, you can manually delete your namespace that is stuck in the Terminating state.
-
View the namespaces that are stuck in the Terminating state:
kubectl get namespaces
-
Select a terminating namespace and view the contents of the namespace to find out the
finalizer
:kubectl get namespace <terminating-namespace> -o yaml
Your YAML contents might resemble the following output:
apiVersion: v1 kind: Namespace metadata: creationTimestamp: 2018-11-19T18:48:30Z deletionTimestamp: 2018-11-19T18:59:36Z name: <terminating-namespace> resourceVersion: "1385077" selfLink: /api/v1/namespaces/<terminating-namespace> uid: b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5 spec: finalizers: - kubernetes status: phase: Terminating
-
Create a temporary JSON file:
kubectl get namespace <terminating-namespace> -o json >tmp.json
-
Edit your
tmp.json
file. Remove thekubernetes
value from thefinalizers
field and save the file.Your
tmp.json
file might resemble the following output:{ "apiVersion": "v1", "kind": "Namespace", "metadata": { "creationTimestamp": "2018-11-19T18:48:30Z", "deletionTimestamp": "2018-11-19T18:59:36Z", "name": "<terminating-namespace>", "resourceVersion": "1385077", "selfLink": "/api/v1/namespaces/<terminating-namespace>", "uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5" }, "spec": { "finalizers": [] }, "status": { "phase": "Terminating" } }
-
To set a temporary proxy IP and port, run the following command. Be sure to keep your terminal window open until you delete the stuck namespace:
kubectl proxy
Your proxy IP and port might resemble the following output:
Starting to serve on 127.0.0.1:8001
-
From a new terminal window, make an API call with your temporary proxy IP and port:
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/<terminating-namespace>/finalize
Your output might resemble the following content:
{ "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "<terminating-namespace>", "selfLink": "/api/v1/namespaces/<terminating-namespace>/finalize", "uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5", "resourceVersion": "1602981", "creationTimestamp": "2018-11-19T18:48:30Z", "deletionTimestamp": "2018-11-19T18:59:36Z" }, "spec": { }, "status": { "phase": "Terminating" } }
Note: The
finalizer
parameter is removed. -
Verify that the terminating namespace is removed:
kubectl get namespaces
-
Continue to follow the steps for other namespaces that are stuck in the Terminating state.