SMC-R link groups
A Shared Memory Communications over RDMA (SMC-R) link group is a logical grouping of SMC-R links between two communicating peers, as shown in Figure 1. An SMC-R link group is formed when the initial SMC-R link is established between two peers.
All SMC-R links in an SMC-R link group must be equal links. SMC-R links are considered to be equal when all of the following conditions are true:
- The links provide access to the same RDMA memory buffers at the remote peer virtual servers.
- The links have the same VLAN ID, or they do not use a VLAN ID.
- The links have the same TCP server and TCP client roles or relationship.
A peer that is acting as the TCP connection server has different responsibilities for establishing and maintaining SMC-R communications than a peer that is acting as the TCP connection client. Unique SMC-R link groups are established between two peers when the peers act as both servers and clients for TCP connections.
When the initial SMC-R link is established and a second IBM® 10 GbE RoCE Express® interface is available, Communications Server establishes an equal SMC-R link between the peers. The 10 GbE RoCE Express interfaces are shown as RNICs in Figure 1.
Adding a second SMC-R link to the SMC-R link group provides the following benefits:
- High availability
To maintain high availability, you need two SMC-R links between SMC-R peers. If a failure occurs with one SMC-R link, TCP connections that are using the failing SMC-R link are switched to the other active link in the link group and disruptions to application workloads are avoided. For more information, see SMC-R high availability considerations.
- Workload balancing
TCP connections are distributed across the SMC-R links in a link group, increasing bandwidth and avoiding bottlenecks.
Rule: Workload balancing within an SMC-R link group occurs only when both the local and the remote peers have two 10 GbE RoCE Express interfaces, and thus two SMC-R links are established in the link group.
Because SMC-R links within a link group are considered equal, TCP connections can be assigned to any SMC-R link within the group. Furthermore, the client and the server can choose to assign the TCP connection to different SMC-R links within the group, and can move the TCP connections from one SMC-R link to another within the group. For example, in Figure 1, client traffic might flow over one SMC-R link (between RNICs A and C) and server traffic might flow over the other SMC-R link (between RNICs B and D). The peers do not have to exchange knowledge of which physical 10 GbE RoCE Express interface is being used for data transmission, and the recipient is only aware that data was placed into the RDMA memory buffer.
An SMC-R link group remains active for up to 10 minutes after the last TCP connection that is using the link group is stopped.