Behavior of ConnectionMultiplicity alongside with MinOpenConnection and MaxOpenConnection

Cause

A brief about the three properties:

com.ibm.CORBA.ConnectionMultiplicity -

The value of the ConnectionMultiplicity defines the number of concurrent TCP connections between the server and client ORBs. By default this value is set to 1, i.e. there will be only one connection between the server and client ORB and all the requests between the client and server ORB will be multiplexed onto the same connection. This could lead to a performance bottleneck in J2EE deployments where there are a large number of concurrent requests between client & server ORB.

Note that the ConnectionMultiplicity is a client side property and will not have any effect for a server ORB, unless the server ORB also acts as a client to another ORB.

com.ibm.CORBA.MinOpenConnections, com.ibm.CORBA.MaxOpenConnections -

These values specify a minimum and maximum number of open connections (active connections required by the server/client in the application) to be cached by the ORB. By default they are set to 100 and 240 respectively. In the case of WebSphere eXtreme Scale these properties are set to 1024 by default.

ORB keeps a cache of socket connections (incoming and outgoing) to avoid creation of new socket for every request. The size of this connection cache (connection table) is determined by these three properties. Once the entries in connection table exceeds the MinOpenConnections value the connections which are not in use maybe removed from the connection table. However, if all of the connections are in use, then none of the connections will be removed, and the size of the connection table can even grow beyond MaxOpenConnections . After exceeding MaxOpenConnections, connection cleanup is triggered for every new connection. This behavior has performance implications because even though the connection will not be removed, the clean-up logic will be triggered on the event of every connection being added to the connection table. Hence, it is recommended to set these properties appropriately.

Though a higher value of the MaxOpenConnections may cause connections to linger around in the JAVA heap without being used, it would avoid COMM_FAILUREs triggered due to the clean-up activity.

Once the connection is removed from the table, the reader thread associated with the connection is killed and COMM_FAILURE exception is cascaded to all the threads waiting on the connection. In cases where MinOpenConnections and MaxOpenConnections are set to lower values than the ConnectionMultiplicity, there is a higher probability of connections being removed from the connection table . Connections are removed when they are not in use, and also under two special scenarios (see Retry Mechancism for ORB below). The lower the ratio between MinOpenConnections and MaxOpenConnections to ConnectionMultiplicity, the higher the frequency of COMM_FAILURES. Even if ConnectionMultiplicity, MinOpenConnections and MaxOpenConnections are set to the same value COMM_FAILURES can be expected.


Retry Mechanism for ORB

Usually requests are triggered as such - stub calls _request() to get a Request object and then calls _invoke() on it. Any transient exception thrown from the invoke path (with Completed Status:No) is retried by ORB on a new connection.

ORB is not able to retry requests under these two scenarios:

Scenario 1:
1. Client ORB has sent all the request fragments and is waiting for response from Server ORB
2. The server ORB has not received any of the request and closes the connection
3. Client ORB throws COMM_FAILURE to application

Scenario 2:
1. Client sends the request to Server
2. Server receives and process the request (the processing takes a bit longer in the failing case)
3. Meanwhile client marks the connection as free and the connection is purged 4. Server completes the execution but can't send the response back to client as the connection is closed by client

Resolving the problem:

Modify ORB properties (MinOpenConnections/MaxOpenConnections/ConnectionMultiplicity) accordingly.

However, if it is feasible for the Application code to modify EJB logic such that duplicate/redundant requests have no effect then the application can implement the retry logic as shown in the below code-snippet -

Throwable cause = e.getCause();
if (cause instanceof org.omg.CORBA.COMM_FAILURE &&
((org.omg.CORBA.COMM_FAILURE)cause).minor ==0x4942F305) {
//Retry request again
}

Recommendation:

The values for ORB properties :
MinOpenConnections/MaxOpenConnections/ConnectionMultiplicity are primarily based on the Application set-up which can be set based on the profiling the application during the peak load hours.

For a client-server setup, an optimal setting for the ConnectionMultiplicity would be approximately 10% of the number of concurrent client threads if the number of concurrent client threads is more than 100, otherwise use 15% of the number of concurrent client threads . For example, if the client expects about 200 requests, then the CM value can be set to 20. If the client expects about 60 requests, then the CM value can be set to 9.

**Note that these values are derived from internal testing, and the optimum values may vary in a real-world application. These values can be started as a base in order to identify the optimum setting.

The number of concurrent threads can be identified by capturing a javacore at the client side during peak load hours.

Ideally, values of MaxOpenConnections should be equal to the average number of active requests handled by a server at a given point in time and MinOpenConnections should be the minimum number of active requests handled by the server.

In an ideal scenario, the value of these three properties will have the following relation:
MinOpenConnections <= Active Connections < MaxOpenConnections