Kubelet fails to start

The kubelet fails to start. Learn about the scenario, the symptom, and the cause.

Scenario 1: kubelet fails to start due to unauthorized certificates

Symptoms

Kubelet fails to start and displays a message similar to the following output:

hyperkube[1554]: E0814 05:07:21.428053    1554 bootstrap.go:195] Part of the existing bootstrap client certificate is expired:      2018-08-14 03:46:00 +0000 UTC
hyperkube[1554]: F0814 05:07:21.438534    1554 server.go:262] failed to run Kubelet: cannot create certificate signing request: Unauthorized

Cause

The client certificate that was used by kubelet has expired. Kubelet failed to renew the certificate. You need to generate a new bootstrap token to join again and get new certificate.

Resolving the problem

Complete the following task to resolve the problem:

  1. Download the kubeadm binary and log in to master node with the following command:

    KUBEVERSION=$(kubectl version --short | grep Version | tail -1 | awk -F '[" "+]' '{print $3}')
curl -L -o /usr/local/bin/kubeadm https://storage.googleapis.com/kubernetes-release/release/$KUBEVERSION/bin/linux/amd64/kubeadm
chmod +x /usr/local/bin/kubeadm

  2. Run the following command to generate a new bootstrap token:

    kubeadm --kubeconfig=/etc/cfc/conf/admin.kubeconfig token create --ttl 24h0m0s

    The output is similar to the following example:

    k5ojt0.ko1wov52mdvnqbg6

    Note: Save this command output for later use.

  3. Get the new certificate:

    • Log in to the node that failed.

    • Edit the kubelet configuration. From /etc/cfc/kubelet/kubelet-bootstrap-config, replace users.user.token with the value that you saved from step two, k5ojt0.ko1wov52mdvnqbg6.

    • Restart the kubelet.

Scenario 2: kubelet fails to start due to invalid kernel flag

Symptoms

When you upgrade from 3.1.0 to 3.1.1, kubelet fails to start and displays a message similar to the following output:

hyperkube[804]: F1023 17:02:19.964867     804 kubelet.go:1333] Failed to start ContainerManager [Invalid kernel flag: vm/overcommit_memory, expected value: 1, actual value: 0, Invalid kernel flag: kernel/panic, expected value: 10, actual value: 0, Invalid kernel flag: kernel/panic_on_oops, expected value: 1, actual value: 0]

Cause

In IBM Cloud Private 3.1.1, by default protectKernelDefaults is set to true in /etc/cfc/kubelet/kubelet-service-config to meet the CIS requirement.

Resolving the problem

To resolve this issue when you upgrade, set protectKernelDefaults to false either before or after installation.