JSOR environment settings (Linux® only)

Use these environment variables to set and size resources for connection and data transfer by using Java™ Sockets over Remote Direct Memory Access (JSOR). Start of changes for service refresh 2 fix pack 10 These environment variables also apply to JSOR NIO client/server scenarios. End of changes for service refresh 2 fix pack 10

Note: The RDMA implementation is deprecated and will likely be removed in a future release of IBM® SDK, Java Technology Edition, Version 8. A possible alternative is the open source Libfabric library.

You can improve your Java application network performance by tuning these variables as follows:

If your application sends small messages, increase the IBM_JAVA_RDMA_INLINE value to reduce the latency of the messages.
If your application sends large messages, increase the IBM_JAVA_RDMA_SBUF_SIZE and IBM_JAVA_RDMA_RBUF_SIZE values to significantly reduce network latencies.
Increasing the IBM_JAVA_RDMA_SQ_SIZE and IBM_JAVA_RDMA_RQ_SIZE values offers only marginal improvement in network performance.
Use the IBM_JAVA_RDMA_POLLING_TIME variable to limit the amount of CPU time spent polling for completion events.
Tuning IBM_JAVA_RDMA_UNACK_CQ_EVENTS allows completion queue events to be received in batch mode, which reduces processing time.
Use the IBM_JAVA_RDMA_COMP_VECTOR variable to spread the completion workload over multiple CPU cores.

Environment variables

IBM_JAVA_RDMA_COMP_VECTOR

This variable is used for tuning completion vectors when creating a new socket completion queue. The default value of this variable is completion vector 0. You can spread completion workload among CPU cores on a system by associating different completion vectors with each connection. The actual mapping of a completion vector to a particular CPU core is decided by the smp_affinity factor that is set at the system level for the corresponding interrupt request (IRQ) number.

This variable accepts comma-separate values, specified either individually or in a range. For example:

To specify individual completion vectors, use 2,3,4,5
To specify completion vectors in a range, use 2 - 5

When a new connection is being set up, the next available completion vector is used. On reaching the end of a specified list, the indexer resets to the beginning of the list. This process ensures that all the completion vectors are distributed evenly between the connections that are created.

IBM_JAVA_RDMA_SBUF_SIZE

IBM_JAVA_RDMA_RBUF_SIZE

These variables specify the RDMA socket send (IBM_JAVA_RDMA_SBUF_SIZE) and receive (IBM_JAVA_RDMA_RBUF_SIZE) buffer sizes, in bytes. If the environment variables are not set, a default size of 128 KB is used for send and receive buffers.

The minimum buffer size is 4096 bytes (4 KB). There is no limit on the upper value of the send buffer size; this limit is determined by the amount of pinned memory that is available on the machine.

The specified value is memory-aligned to 4 KB pages.

If the machine is short of lockable physical memory, the specified send or receive buffer registration might fail, resulting in the failure of RDMA socket creation during the accept or connect stage. If this situation occurs, the API throws an IOException exception that indicates that an accept or connect operation failed.

IBM_JAVA_RDMA_SQ_SIZE

IBM_JAVA_RDMA_RQ_SIZE

These variables specify the RDMA send (IBM_JAVA_RDMA_SQ_SIZE) and receive (IBM_JAVA_RDMA_RQ_SIZE) queue sizes, as integers.

When a send or receive operation is in progress, subsequent send or receive requests are queued until the first request is complete. The number of pending requests is limited by the size of the send or receive queue.

The default queue size and limits are the same for both send and receive queues. In line with RSocket settings, the default queue size is 384. The minimum queue size is 16. The maximum queue size is limited by the hardware. For example:

For Mellanox MT26428, MT26448, and MT4099 InfiniBand or RoCE adapters, the send and receive queue sizes are limited to 16351.
For Mellanox MT25208 InfiniBand adapters, the send and receive queue sizes are limited to 16384.

If a write operation tries to overfill the send queue, or a read operation tries to take data from an empty receive queue, the associated send or receive operations are blocked until a send slot is vacant or a receive slot is filled.

IBM_JAVA_RDMA_INLINE

This variable specifies the size of message, in bytes, that can be posted inline to a send queue. An inline message is one that is directly read by the low-level driver instead of the RDMA device. Inline buffers do not require memory registration and can be reused immediately after the send operation returns.

The default value is 64 bytes. The minimum value is 64 bytes. The maximum value is 768 bytes

IBM_JAVA_RDMA_POLLING_TIME

This variable specifies the amount of time in microseconds to poll for completion events before the run time is set to blocking mode. Altering this setting affects the application's CPU usage counters.

The default value is 10 microseconds. The minimum value is zero. Although there is no limit on the maximum value you can set, a higher value results in increased CPU usage due to polling.

When a connection request is accepted from the passive side or established from the active side, an RDMA socket is populated with the necessary RDMA resources required for communication. These resources include send and receive buffers, and queue pairs for posting work requests. Appropriate sizes for these buffers and queues are set immediately after an RDMA socket is allocated. The following rules apply:

On the passive side, when an incoming connection request is accepted, the sizes that are used for all child sockets are inherited from the server socket.
When creating a new server socket or initiating a new connection on the client side, the sizes are set according to the following order of priority:
1. Sizes are specified in the environment variables.
2. Sizes are taken from global preferences set by using the socket options property. For more information, see -Dcom.ibm.net.rdma.socketOptions (Linux only).
3. Sizes are based on hardcoded defaults.

IBM_JAVA_RDMA_UNACK_CQ_EVENTS

This variable is used to acknowledge completion events in batch, instead of individually. This variable accepts an integer value greater than or equal to 1. The default value is 1. The upper limit of this variable is set to the maximum possible completion events in a queue. However, for optimum performance, set this value to the size of a completion queue, which is equal to the sum of send and receive queue pair sizes. Each acknowledgment involves acquiring and releasing a shared lock. By enabling batch processing, the overhead of many lock and unlock cycles is minimized.