Setting up the SMC support

Edit online

Red Hat Enterprise Linux 8.6 LPAR mode z/VM guest KVM guest

SMC traffic requires a mapping of the OSA or HiperSockets network interfaces to the RoCE adapters or ISM devices.

SMC-D version 1

Physical network (PNET) IDs provide this mapping. If a network interface and a RoCE or ISM device have the same PNET ID, they are connected to the same physical network and can be used together for SMC.

LPAR and z/VM®
For Linux® in LPAR mode and for Linux on z/VM you can assign PNET IDs to OSA, HiperSockets, RoCE, and ISM devices through the IOCDS.
Figure 1 illustrates how the IOCDS assigns the PNETID NET1 to an ISM device and a network interface for an Ethernet device. In Linux, the matching PNETID associates the ISM device with an Ethernet device.
Figure 1. PNET ID and ISM device association

The PNET on the Ethernet device and in the SMC definition must be the same

As a fallback, you can also use a software PNET table that maps network interfaces to RoCE adapters. For details, see the KVM information that follows.

KVM
For SMC on Linux on KVM, you need a software PNET table that maps network interfaces to RoCE adapters or ISM devices. Use the smc_pnet command to create a physical network (PNET) table with this mapping.
Note: z/OS® does not support the RoCE Express adapter as an Ethernet device, and therefore uses OSA adapters for the initial handshake for SMC-R connections. Linux has no such constraint.

SMC-D version 2

Available as of z15, SMC-D version 2 allows for communication peers on different subnets, as illustrated in Figure 2. The communication peers must have the same system enterprise ID (SEID). For communication with Linux, instances of z/OS must have the SEID set. All operating system instances on the same CPC have the same SEID.

Figure 2. SMC-D version 2 provides SMC over multiple IP subnets

The graphic shows two subnets, NET1 and NET2, connected through an IP router.

ISM devices must have the same channel ID, virtual or physical, to be useable for SMC-D version 2 communication between two peers.

For the initial handshake of the connection, an ISM device must either have no PNET ID set, or, as a compatibility feature with version 1, must have the same PNET ID as any network device on the operating system.
Figure 3. A connection handshake requires an eligible ISM device

The graphic shows one subnet and an extra OSA adapter in another subnet, a handshake is performed between the two.

Network device settings

On the network device that is associated with the IB (RoCE) device you want to use for SMC traffic, check the settings with the ethtool command and ensure that pause settings are turned on.

For example, if enP2s13 is the network device that is associated with the desired IB device port:
# ethtool -a enP2s13
Pause parameters for enP2s13:
Autonegotiate: off
RX: on
TX: on

Sysctl settings

SMC requires contiguous memory. The minimum is 16 KB, and the maximum is 512 MB. The SMC implementation selects a value as follows:
  • Some socket applications define the socket send- and receive buffer sizes with a setsockopt call, whose upper limits are defined in net.core.wmem_max and net.core.rmem_max.
  • If setsockopt SO_SNDBUF is not used, the socket send buffer size is taken from the value of net.ipv4.tcp_wmem.
  • If setsockopt SO_RCVBUF is not used, the socket receive buffer is taken from the value of net.ipv4.tcp_rmem, rounded to the next higher power of 2.

Make an existing application use SMC

Use the preload library to make the unmodified socket application use SMC. Existing TCP/IP applications can benefit from the SMC protocol without recompiling, if they are invoked with the SMC preload library libsmc_preload.so. See the smc-tools package for the smc_run script, which makes an existing TCP/IP socket program use SMC.

Converting an application to use SMC

Alternatively, if you need to, you can convert an application. To convert an application from TCP/IP to SMC sockets, change the socket() function call from AF_INET to AF_SMC with protocol 0 and from AF_INET6 to AF_SMC with protocol 1. For example, change:
sd = socket(AF_INET, SOCK_STREAM, 0);
to:
sd = socket(AF_SMC, SOCK_STREAM, 0);
and
sd = socket(AF_INET6,SOCK_STREAM, 0);
to:
sd = socket(AF_SMC, SOCK_STREAM, 1);
Use the sockets.h header file from the glibc-header package. For more programming information, see the af_smc(7) man page.