Troubleshooting
Problem
In some environments, you may notice the pods icp-mongodb-x are frequently restarted.
This can have consequences on performances and reliability of the cluster.
Looking at the log of the container, we can see it shows messages like this just before the crash:
2019-10-01T13:44:22.522+0200 F - [conn3264] Failed to mlock: Unknown error
Cause
As indicated by the message "Failed to mlock",
the problem occurs because the MEMLOCK limit used by the container is to low.
You can easily check it by execing into the container and running the ulimit -l command:
kubectl exec -it icp-mongodb-0 -c icp-mongodb -- /bin/sh -c 'ulimit -l'
64
When you see frequent restart of the mongodb pods, the container logs shows "Failed to mlock" and the memlock limit
of the mongodb container shows the value 64, then this is most likely the cause of the problem.
Diagnosing The Problem
The ulimit values of the containers are inherited from parent processes.
All the containers get their ulimit from docker, that on its own, unless you configured it to do in another way, takes the values from the system.
So all start from the ulimit values defined on the node at system level.
You can anyway avoid to alter the "max locked memory" at system level by changing the value only for docker.
This can be done by setting the parameter LimitMEMLOCK=infinity into the service file of the parent process.
Resolving The Problem
Usually the file where you need to put this parameter is the docker.service file.
Anyway, for some old version of docker you may need instead to put the above parameter into the containerd.service file.
How to know which file you need to use ?
Check whether in your node, you see a process called "/usr/sbin/containerd" by doing:
ps -ef | grep /usr/sbin/containerd
If nothing is returned (/usr/sbin/containerd is not present), then you need to make the changes to docker.service file
If instead you have that process running, then you need to make the changes into containerd.service file.
How to fix the issue when you identified the proper place where making the changes ?
a) run systemctl status containerd (or docker) and look at the path for containerd.service (or docker.service)
b) Edit the configuration file and add:
LimitMEMLOCK=infinity
Save and exit.
c) Restart kubelet and docker services following this sequence :
1) stop kubelet
2) stop docker
3) re-start containerd
4) start docker
5) start kubelet
2) stop docker
3) re-start containerd
4) start docker
5) start kubelet
d) When system pods are all running, exec into the mongodb container and confirm you see unlimited for memlock:
kubectl exec -it icp-mongodb-0 -c icp-mongodb -- /bin/sh -c 'ulimit -l'
Document Location
Worldwide
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSBS6K","label":"IBM Cloud Private"},"Component":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Product Synonym
IBM Cloud Private
Was this topic helpful?
Document Information
Modified date:
29 November 2019
UID
ibm11117497