Ingress controller reported: epoll_create() failed (24: Too many open files)

Ingress controller does not work with a POWER machine, which has 160 cores. Ingress controller might fail when it is running on a node with lots of cores.

Causes

Ingress controller might be running on a node that has too many cores. The maximum number of open file descriptors is calculated with the following formula: *RLIMIT_NOFILE/worker-processes) - 1024. To resolve, you can either decrease the value of the worker processes, or increase the value of the RLIMIT_NOFILE of the container.

Solution one: Edit the configMap of nginx-ingress-controller with a decreased value of worker-processes.

1. To edit the configmap of nginx-ingress-controller, run the following command:

   kubectl -n kube-system edit cm nginx-ingress-controller
   

2. Add worker-processes: "2" to the configMap, as it is in the following example. Note: The value might not be 2, depending on your sysctl configuration.

   # Edit the following object. Lines beginning with a '#' are ignored,
   # and an empty file will abort the edit. If an error occurs while saving this file will be
   # reopened with the relevant failures.
   #
   apiVersion: v1
   data:
    body-size: "0"
    disable-access-log: "true"
    worker-processes: "2"
   

Solution two: Set docker LimitNOFILE to a larger value to increase the value of RLIMIT_NOFILE of your ingress controller container.

Note: The maximum number of open file descriptors is calculated with the formula RLIMIT_NOFILE / worker-processes) - 1024. You can increase RLIMIT_NOFILE by changing the Docker LimitNOFILE to a larger value.

1. Run the following command to list your pods that run ingress:

   $ kubectl -n kube-system get pods -owide | grep nginx-ingress
   nginx-ingress-lb-amd64-sknw6                              1/1       Running   0          35m       9.111.255.21    9.111.255.21
   

2. Log on to the node where the ingress controller is running and edit the file /lib/systemd/system/docker.service and update LimitNOFILE to a large value. If LimitNOFILE is set to infinity, it means that the value is 65536 or 1048576 (2^20), based on your operating system.

   [Service]
   LimitNOFILE=<your value>
   

   LimitNOFILE cannot be larger than fs.file-max. If you need to update fs.file-max to a larger value, run the following command:

   sysctl -w fs.file-max=<your value>
   

3. Restart Docker.

   systemctl daemon-reload
   systemctl restart docker