Recommendations for tuning maxTcpConnsPerNodeConn parameter

Starting with IBM Storage Scale 5.1.1, you can establish multiple TCP connections between nodes (with the same daemon IP address being used on each end) to optimize the use of network bandwidth. IBM Storage Scale 5.1.5 introduces the Multi-Rail Over TCP (MROT) feature that enables the concurrent use of multiple subnets to communicate with a specified destination, and now allows the concurrent use of multiple physical network interfaces without requiring bonding to be configured. The number of connections is controlled through the maxTcpConnsPerNodeConn parameter, which can be changed by using the mmchconfig command. Valid values are 1-16, with the default being 2.

The value that is assigned to maxTcpConnsPerNodeConn must be defined after you consider the following factors:

  • The overall bandwidth of the cluster network.
  • The number of nodes in the cluster.
  • With MROT, the number of physical network interfaces used for communication.
  • The value that is configured for the maxReceiverThreads parameter.
  • Memory resource implications of setting a higher value for the maxTcpConnsPerNodeConn parameter.

Tuning based on the overall bandwidth of the cluster network

The bandwidth that one TCP connection can achieve depends on a number of factors such as tuning, CPU, memory, and network adapter performance. However, a good starting point estimation is to assume that the bandwidth per each TCP connection can be as much as 25 Gbps. You can then set a larger value for the maxTcpConnsPerNodeConn parameter on faster networks so that the full network bandwidth can be used.

With MROT, if multiple network interfaces are used for communication, you can set a larger value for the maxTcpConnsPerNodeConn parameter, which can be greater than 8.
Note: The clusters that support RDMA for data transfers might not need to change the default value of maxTcpConnsPerNodeConn to improve bandwidth, as verbsRDMA is used instead of TCP for more bandwidth sensitive operations.

Tuning based on the number of nodes in the cluster

As the number of nodes in a cluster increase, you should decrease the value of the maxTcpConnsPerNodeConn parameter because each node requires a number of connections to other nodes, and the additional connections consume more network bandwidth. That is, large clusters with many client connections to NSD server nodes might not need to set maxTcpConnsPerNodeConn higher than 1, unless there are other traffic patterns such as the use of HDFS transparency, which can benefit from this tuning.

The following table provides guidelines for the value of maxTcpConnsPerNodeConn in proportion to the number of nodes in the cluster.

Table 1. Recommendations for selecting value for the maxTcpConnsPerNodeConn parameter
Number of nodes in the cluster Network bandwidth Value of maxTcpConnsPerNodeConn
Less than 100 100 Gbps 2 - 8
100 - 1000 100 Gbps 1 - 4
1000 - 2000 100 Gbps 1 - 2
More than 2000 100 Gbps 1

For more information about the maxTcpConnsPerNodeConn parameter, see the mmchconfig command.

Implications on receiver threads

Configuring maxTcpConnsPerNodeConn has a potential impact on the maxReceiverThreads parameter because the additional network connections might require more receiver threads. Each receiver thread can typically monitor up to 128 TCP connections, but optimal performance can be achieved when each receiver thread monitors fewer connections. The value of maxReceiverThreads should be selected after considering the value of the maxTcpConnsPerNodeConn parameter and the number of nodes in your cluster. Some large clusters need to increase the value of maxReceiverThreads based on the number of TCP connections that will be needed to other nodes in both local clusters and remote clusters that are joined. The total number of TCP connections that are required is calculated by using the following formula: (maxTcpConnsPerNodeConn*(number of nodes -1)).

The maximum number of receiver threads that are created on any node is defined to be the minimum of the number of logical CPUs on the node and the value of the maxReceiverThreads parameter. You can specify a value in the 1-128 range for the maxReceiverThreads parameter, with the default value being 32. For more information about how to configure maxReceiverThreads, see mmchconfig command.

Implications on memory resources

Setting a higher value for this parameter can require more memory for allocations such as kernel socket buffers, and it can put more pressure on memory-related resources in IBM Storage Scale.