Network requirements for SMC-R

z/OS® Communications Server supports connectivity to multiple, distinct layer 2 networks through unique physical LANs. Each unique physical network is identified by existing Ethernet standards that are based on the physical layer 2 broadcast domain. You can define a physical network ID (PNetID) for each physical network. For more information, see SMC-R physical network considerations.

For hosts to communicate by using SMC-R, they must connect directly to the same Ethernet layer 2 LAN network. If VLANs are in use, each host must also have access to the same VLAN. For more information, see SMC-R VLANID usage.

There are restrictions on the physical distances that can be used to route RDMA frames. To understand these distance specifications and limitations, see IBM z Systems™ Planning for Fiber Optic Links.

RoCE network high availability

Because RoCE connections do not use IP routing and the RDMA connections to remote hosts are direct point-to-point connections that use reliable connected queue pairs (RC QPs), there is no concept of an alternative IP route to the peer. SMC-R connectivity is possible with a single 10 GbE RoCE Express® feature, but a loss in that single feature means that the associated TCP connections and workloads are disrupted. Therefore, redundant 10 GbE RoCE Express features on both the local and remote hosts are required to achieve network high availability with SMC-R. If your TCP workloads require high availability, redundant 10 GbE RoCE Express features and redundant Ethernet switches are required. The SMC-R protocol actively uses both features, rather than using one feature with the other in standby mode. For more information, see SMC-R high availability considerations.

IBM® 10 GbE RoCE Express features also have redundant internal PCIe support structures, or PCIe internal paths, as described in Redundancy levels. To avoid another single point of failure, install each 10 GbE RoCE Express feature that is managed by the same operating system with a unique internal path. For more information about how to install a 10 GbE RoCE Express feature to achieve full redundancy, see your IBM service representative.

RoCE bandwidth

The 10 GbE RoCE Express features provide 10 GbE ports. When redundant features are defined, SMC-R link groups can be formed by using both features, resulting in a 20 GbE logical pipe to each physical network. z/OS Communications Server uses only two features within a link group at any particular time.