Slow interaction between kubelet and Docker cause PLEG issues

Docker delays its response to the kubelet when you create a pod.

Symptoms

Pods might fail to create because there is a timeout for kubelet to call the Docker engine.

Causes

There is a overload of pods on the host node. You might receive the following error message:

skipping pod synchronization - [PLEG is not healthy: pleg was last seen active 6m55.488150776s ago; threshold is 3m0s]

Reduce the possibility of occurrence

Update the kubelet service file.

1. Edit the kubelet service configuration file. Run the following command:

   vi /etc/systemd/system/kubelet.service
   

2. Add the --housekeeping-interval option to increase the interval time. The default value is 10s. Add the following option:

   --housekeeping-interval=30s
   

   Your kubelet service configuration file might resemble the following output:

   [Unit]
   Description=Kubelet Service
   Documentation=https://github.com/kubernetes/kubernetes
   
   [Service]
   EnvironmentFile=-/etc/environment
   ExecStart=/opt/kubernetes/hyperkube kubelet \
    --cadvisor-port=0 \
    --docker-disable-shared-pid \
    --bootstrap-kubeconfig=/etc/cfc/kubelet/kubelet-bootstrap-config \
    --kubeconfig=/etc/cfc/kubelet/kubelet-config \
    --cert-dir=/etc/cfc/kubelet \
    --config=/etc/cfc/kubelet/kubelet-service-config \
    --dynamic-config-dir=/etc/cfc/kubelet/kubelet-dynamic-config \
    --network-plugin=cni \
    --hostname-override=9.111.255.33 \
    --node-ip=9.111.255.33 \
    --pod-infra-container-image=hyc-cloud-private-edge-docker-local.artifactory.swg-devops.com/ibmcom-amd64/pause:3.1 \
    --node-labels=node-role.kubernetes.io/worker=true, \
    --register-with-taints= \
    --keep-terminated-pod-volumes=false \  
    --housekeeping-interval=30s
   
   Restart=always
   RestartSec=10
   
   [Install]
   WantedBy=multi-user.target
   

3. Restart the kubelet service on the host node with following commands:

   systemctl daemon-reload
   systemctl restart kubelet
   

4. You must specify a resource request and limit for your application. For more details, see Setting resource quota.