java.lang.OutOfMemoryError: Failed to create a thread error

Why am I getting this and how can I correct it?

Symptoms

I was trying to run a script/starting the product services, but the environment was unable to create Java threads for it due to OutOfMemoryError exception.

Causes

The java.lang.OutOfMemoryError: Failed to create a thread message occurs when the system does not have enough resources to create a new thread. There are three possible causes for this message:
  • Inadequate user/application resources.
  • The system has run out of native memory to use for the new thread. Threads require a native memory for internal JVM structures, a Java stack, and a native stack.
  • There are too many threads already running and the system has run out of internal resources to create new threads.
Sample error stack:
java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11
        at java.lang.Thread.startImpl(Native Method)
        at java.lang.Thread.start(Thread.java:891)
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:738)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:668)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:396)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:353)
        at java.lang.Thread.run(Thread.java:738)
Note: The native, or system heap, is different from Java heap. The Java heap contains the instances of Java objects and is maintained by Garbage Collection. The maximum size of the Java heap is pre-allocated during JVM startup as one contiguous area, even if the minimum heap size setting is lower. Meanwhile the native heap is allocated by using the underlying malloc and free mechanisms of the operating system, and is used for the underlying implementation of Java objects. We can increase or decrease the maximum native heap available by altering the size of the Java heap. This relationship between the heaps occurs, because the process address space that is not used by the Java heap is available for the native heap usage.

Resolving the problem

Refer to the following (in order) to correct this error:
  1. Linux has a maximum allowed process per user limit, we can check this using the ulimit -u command. If this value is low (default is 1024), then either make it unlimited or raise it to a high value, say 131072. This section is also highlighted as the max user processes section in ulimit -u output. Use the following command to set it to unlimited: ulimit -u unlimited
  2. Increase the amount of native memory available by lowering the size of the Java heap by using the -Xmx option. The process address space that is not used by the Java heap is available for the native heap usage. Java heap is set in $TOP/bin/conf/service_mem_settings.ini file for the 6 services and as custom_java_options in $TOP/bin/conf/env_settings.ini file for the back end scripts.
  3. Check space requirements by using the df -k command.
  4. Lower the number of threads being used.