Troubleshooting
Problem
Understanding the pod pending status
Resolving The Problem
What';s Happening
The applications in the cluster are down. When you do a basic health check, all the application pods are in a pending status. However, the worker nodes and cluster are in the ready state.
To get the pods status, you can use the following command:
kubectl get pods
Why it';s Happening
If the pods are stuck in Pending, it means that it cannot be scheduled onto a node. Generally this situation occurs because there is insufficient resources of one type or another that is prevent scheduling.
To understand the reason why the pods are in the pending status, use the following command:
kubectl get events
The command output might show a message similar to the following text:
"No nodes are available that match all of the predicates: Insufficient cpu (2), Insufficient memory (2)".
This information means that your pod does not fit on the nodes because of insufficient CPU and because of insufficent memory.
To see how many resources are actually used you can use the following command:
kubectl describe nodes
The results of the previous command will provide you with this information:
- The amount of CPU/memory that is available per node
- The amount of CPU member that is being used by all pods and containers
- The amount of CPU/member that is still free and used on each node.
See the following screen shot as an example of the output from the command:
When you define the resources for your cluster it is a common mistake to not consider the resources that are used by the system components. These additional containers are running in addition to the ones that are specified in the configuration. The system components are provisioned, by default, with Kubernetes and run in the "kube-system" namespace, which is not shown in the default namespace.
If you have very small nodes (only 2 CPUs per node), and there is a set of system-services always running on each node, these system-services also consume resources and might consume around 1 CPU per node.
How to Fix
Use the following recommendations to resolve the issue:
- Reduce the CPU requests until it works
- Remove other pods to free up the CPUs
- Add a new worker node to increase the available CPUs
For a production cluster, it is recommended that you avoid small nodes. When you create a cluster, use larger nodes with 4 or 8 CPUs minimum.
Document Location
Worldwide
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSJTBP","label":"IBM Cloud Kubernetes Service and Red Hat OpenShift on IBM Cloud"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB21","label":"Public Cloud Platform"}}]
Was this topic helpful?
Document Information
Modified date:
01 August 2019
UID
ibm1KB0011054