Ingress controller reported: epoll_create() failed (24: Too many open files)

Ingress controller does not work with a POWER machine, which has 160 cores. Ingress controller might fail when it is running on a node with lots of cores.


Ingress controller might be running on a node that has too many cores. The maximum number of open file descriptors is calculated with the following formula: *RLIMIT_NOFILE/worker-processes) - 1024. To resolve, you can either decrease the value of the worker processes, or increase the value of the RLIMIT_NOFILE of the container.

Solution one: Edit the configMap of nginx-ingress-controller with a decreased value of worker-processes.

  1. To edit the configmap of nginx-ingress-controller, run the following command:

    kubectl -n kube-system edit cm nginx-ingress-controller
  2. Add worker-processes: "2" to the configMap, as it is in the following example. Note: The value might not be 2, depending on your sysctl configuration.

    # Edit the following object. Lines beginning with a '#' are ignored,
    # and an empty file will abort the edit. If an error occurs while saving this file will be
    # reopened with the relevant failures.
    apiVersion: v1
     body-size: "0"
     disable-access-log: "true"
     worker-processes: "2"

Solution two: Set docker LimitNOFILE to a larger value to increase the value of RLIMIT_NOFILE of your ingress controller container.

Note: The maximum number of open file descriptors is calculated with the formula RLIMIT_NOFILE / worker-processes) - 1024. You can increase RLIMIT_NOFILE by changing the Docker LimitNOFILE to a larger value.

  1. Run the following command to list your pods that run ingress:

    $ kubectl -n kube-system get pods -owide | grep nginx-ingress
    nginx-ingress-lb-amd64-sknw6                              1/1       Running   0          35m
  2. Log on to the node where the ingress controller is running and edit the file /lib/systemd/system/docker.service and update LimitNOFILE to a large value. If LimitNOFILE is set to infinity, it means that the value is 65536 or 1048576 (2^20), based on your operating system.

    LimitNOFILE=<your value>

    LimitNOFILE cannot be larger than fs.file-max. If you need to update fs.file-max to a larger value, run the following command:

    sysctl -w fs.file-max=<your value>
  3. Restart Docker.

    systemctl daemon-reload
    systemctl restart docker