Handling Multiple Connections

Resolve Out of Memory Issues

Each non-multiplex nSession to Universal Messaging creates at least three Java threads:

• A connection read thread (UM-Connection-Reader) - responsible for reading events received from the Universal Messaging server.
• A connection write thread (fProcessThreadedQueue) - responsible for sending events to the Universal Messaging server.
• One or more event dispatch threads (UM-Event-Processing-Pool) - responsible for processing events that are read from the connection. These threads are available only when threading is enabled on the session. Threading is enabled by default and a thread is created when the session is started. For more information about threading, see the Universal Messaging Java API reference documentation for nSession.enableThreading(boolean).

The Java virtual machine (JVM) allocates a certain amount of memory to each thread stack. For this reason, when many threads are created for many sessions, the client application might run out of memory.

You can perform the following actions to reduce memory consumption:

• Reduce the amount of memory allocated for a thread stack by using the -Xss command-line option of the JVM. For more information about the option, see the documentation of your JVM vendor.
• Decrease the number of event dispatch threads used for every session by configuring the com.softwareag.um.client.useSharedThreadPool and com.softwareag.um.client.SharedThreadPoolMaxSize client system properties.

  Normally, the Universal Messaging Java client uses a dedicated thread pool of event dispatch threads for every session. Instead, you can configure the client to use a single, shared thread pool for this type of threads for all client sessions by using the following properties.

  • com.softwareag.um.client.useSharedThreadPool - enables usage of a shared thread pool for processing incoming nConsumeEvent events for each client session. Values are true or false (default). By default, each client session creates and uses a separate thread pool for processing received nConsumeEvent events. When this system property is set to true, all client sessions will use a shared thread pool. Using a shared thread pool for all client sessions may have a performance impact depending on the size of the shared thread pool and the number of client sessions.
    Note: When each client session uses a separate thread pool, closing the session interrupts any event listener threads that are blocked during processing. When all client sessions share a thread pool, closing a session does not interrupt any event listener threads that are blocked during processing. Thus, the blocked event listener does not release a shared thread after closing the client session. This behavior impacts the processing of incoming events, because the shared pool will not have available threads. IBM recommends that you enable the shared pool for all client sessions only if the event listener of the client application ensures that the processing of the events is fast and graceful.
  • com.softwareag.um.client.SharedThreadPoolMaxSize - specifies the maximum size of the shared thread pool. The default is 5 threads.

Resolve Socket Exceptions

The connection protocol used for each session to Universal Messaging impacts the total number of sockets that are created. An nSession using the nsp or nsps protocol, that is, the Universal Messaging socket protocol or SSL socket protocol, uses a single socket for communication with the Universal Messaging server. An nSession using the nhp or nhps protocol, that is, the Universal Messaging HTTP or HTTPS protocol, uses two sockets for communication with the server.

An application creating many sessions might fail with a "java.net.SocketException: Too many open files" error if it creates so many sockets that it reaches the maximum allowed count of open files, or sockets, on the operating system.

To resolve this issue, you might need to perform any or a combination of the following actions:

• Increase the limit of maximum allowed files or sockets.
• Consider if your application requires that many sessions.
• Check if the application creates sessions but fails to close them after they are no longer needed.
• Check for TCP/IP port exhaustion.

  This issue might occur even when the application closes sessions when they are no longer needed. If the application creates and then closes a lot of sessions in a short time, it might cause TCP/IP port exhaustion because operating systems usually keep sockets in a waiting status (TIME WAIT) for some time after sockets are released by the application. In this case, you might need to do the following:

  • Expand the range of ports available for use by your application.
  • Reduce the time the operating system keeps those sockets in a waiting status after they are released.

Use NSP or NSPS Protocol

When handling many inbound connections that do not require the HTTP or HTTPS protocol, IBM recommends that you use the Universal Messaging Socket Protocol (nsp) or the Universal Messaging SSL Protocol (nsps).

For more information about what native communication protocols that Universal Messaging supports, see Native Communication Protocols.

Adjust Interface Attributes

The Universal Messaging server does not allocate individual threads for each incoming connection. Instead, the server uses a thread pool named Accept Threads to accept the connection and a separate thread pool named Select Threads to monitor data to be read or written to the socket.

When a Universal Messaging server is handling many incoming connections at the same time, you might need to adjust several attributes of the server interface to which the client connects. You configure the attibutes on the Comms > Interfaces tab for a server in the Enterprise Manager as follows:

• Accept Threads. Increase the Accept Threads value, otherwise some of the connections will be queued in a backlog queue waiting to be accepted.
• Backlog. Increase the capacity of the Backlog queue, otherwise when the capacity of the backlog queue is reached, additional connection requests will be denied.
  Note: On Linux, you might need to set the kernel parameter net.core.somaxconn. For Linux 2.2 and greater, the parameter specifies the maximum queue length for completely established sockets waiting to be accepted.

  For additional operating-system-specific tuning, see Tuning the Linux Operating System.

• Auth Time. Increase the time to authenticate incoming connection requests if connection authentication is slow and causes the server to close the connection.
• Select Threads. Keep the Select Threads value low even if the interface is handling a lot of connections. The number of select threads should not typically exceed the number of cores available.

  Select Threads transfers reading or writing of network data to separate read or write thread pools. You configure the maximum count of threads for the read and write thread pools in the ReadThreadPoolMaxSize and WriteThreadPoolMaxSize properties on the Config > Thread Pool Config panel in the Enterprise Manager. Read pool threads are used for processing incoming client requests, while write pool threads are involved in writing data to client connections.

For more information about the attributes of an interface, see Basic Attributes for an Interface.

Resolve "java.lang.OutOfMemoryError Direct buffer memory" Errors

Universal Messaging uses in-memory buffers to handle network traffic. By default, these network buffers use the Java direct (off-heap) memory. To configure the Java direct memory for a realm in the Enterprise Manager, you go to Config > Connection Config > UseDirectBuffering.

If the connections handle a lot of traffic or if there are too many connections, the server might return the error "java.lang.OutOfMemoryError Direct buffer memory". In this case, you should increase the amount of direct memory allocated for the Universal Messaging server process by using the -XX:MaxDirectMemorySize=<size> Java command-line argument.

For information about -XX:MaxDirectMemorySize, see the Oracle documentation on Java command-line options.

For more information about how to troubleshoot the "java.lang.OutOfMemoryError Direct buffer memory" error and configure direct memory, see Troubleshooting.

Handle a Slow Network when Clients are Across WAN

In case of a slower network when clients are across WAN, IBM recommends that you increase the socket read and write buffer size based on bandwidth-delay product.

Be aware that increasing the read and write buffer size impacts the direct memory allocated by the server. You might need to adjust your direct memory accordingly.