IBM Support

Containers fail to start on RHEL 7.x ICP node

Troubleshooting


Problem

In IBM Cloud Private (ICP) 2.1.0.2 and 2.1.0.3 environment, the containers may fail to start on RHEL 7.x ICP node with old Linux kernel, e.g.: 3.10.0-514.el7.x86_64.

Symptom

If the containers fail to start on RHEL 7.x ICP node with old Linux kernel, you could see below errors in system log file (/var/log/messages).

...............

kubelet.ns-k8s-node001.root.log.ERROR.20180214-113740.15702:1593018:E0320 04:59:09.572336 15702 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to start sandbox container for pod "osp-xxx-com-ljqm19-54bf7678b8-bvz9s": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:258: applying cgroup configuration
for process caused \"mkdir /sys/fs/cgroup/memory/kubepods/burstable/podf1bd9e87-1ef2-11e8-afd3-fa163ecf2dce/8710c146b3c8b52f5da62e222273703b1e3d54a6a6270a0ea7ce1b194f1b5053: no space left on device\""

...............

Or you could see below kernel errors:

...............

[43073.004575] SLUB: Unable to allocate memory on node -1 (gfp=0x8020)
[43073.022211]   cache: ip6_dst_cache(0:18479a49f76b66461d8fedd17a6eba407f08cc8960bd0246d39df8eb21f20b4f), object size: 448, buffer size: 448, default order: 2, min order: 0
[43073.061420]   node 0: slabs: 74, objs: 2610, free: 0
[43073.077875]   node 1: slabs: 88, objs: 3141, free: 0

...............

[{"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBS6K","label":"IBM Cloud Private"},"Component":"Kubernetes","Platform":[{"code":"PF016","label":"Linux"}],"Version":"2.1.0.2;2.1.0.3","Edition":"","Line of Business":{"code":"LOB77","label":"Automation Platform"}}]

Log InLog in to view more of this document

This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.

Document Information

Modified date:
27 August 2018

UID

ibm10716349

Page Feedback