RDMA over Converged Ethernet

Ubuntu Server supports RoCE features as shown in Table 1. Note that the mapping of ports to function keys depend on the adapter hardware.

The RoCE support requires PCI Express support.

You can use a PCI function as a base for MacVTab or OpenVSwitch similarly to an OSA adapter, see Using an HSCI interface as a base device for MacVTap or OpenVSwitch.

More information

For more information about RoCE Express, see Networking with RoCE Express, SC34-7745. You can find this publication and further information about using RoCE Express with Linux on IBM Z and IBM LinuxONE on IBM Documentation at ibm.com/docs/en/linux-on-systems?topic=configuration-roce-express.

Using a RoCE device for SMC-R

SMC-R requires RoCE devices that are associated with network devices of TCP networks through a PNET ID, for example through statements in the IOCDS.

The following figure illustrates how a RoCE device and a Ethernet device are associated by a matching PNET ID. A communication peer has a similarly associated pair of an RoCE device and Ethernet device. With this setup, the TCP connection can switch over to an SMC-R connection over the SMC protocol. The communication peer can but need not be on the same CPC.

Using SMC-R link groups

Once established, failed SMC-R links do not fall back to the TCP connection. To protect against link failure, SMC-R creates link groups for you. Link groups use multiple RoCE devices with the same PNET ID. A similar association of an Ethernet device with multiple RoCE devices on the communication peer then results in multiple, independent SMC-R links within a link group.

The SMC-R connection survives failures of individual RoCE devices if at least one device remains operational on each side.

Use the smcr command to explore SMC-R links, link groups, and devices (see smcr - Display information about SMC-R).