Setting up the SMC support

SMC traffic requires a mapping of the OSA or HiperSockets network interfaces to the RoCE adapters or ISM devices.

Physical network (PNET) IDs provide this mapping. If a network interface and a RoCE or ISM device have the same PNET ID, they are connected to the same physical network and can be used together for SMC.

LPAR and z/VM®: For Linux® in LPAR mode and for Linux on z/VM you can assign PNET IDs to OSA, HiperSockets, RoCE, and ISM devices through the IOCDS.
Figure 1 illustrates how the IOCDS assigns the PNETID NET1 to an ISM device and a network interface for an Ethernet device. In Linux, the matching PNETID associates the ISM device with an Ethernet device.
Figure 1. PNET ID and ISM device association

As a fallback, you can also use a software PNET table that maps network interfaces to RoCE adapters. For details, see the KVM information that follows.
KVM: For SMC on Linux on KVM, you need a software PNET table that maps network interfaces to RoCE adapters or ISM devices. Use the smc_pnet command to create a physical network (PNET) table with this mapping.

Note: z/OS® does not support the RoCE Express adapter as an Ethernet device, and therefore uses OSA adapters for the initial handshake for SMC-R connections. Linux has no such constraint.

Network device settings

On the network device associated with the IB (RoCE) device you want to use for SMC traffic, check the settings with the ethtool command and ensure that pause settings are turned on.

For example, if enP2s13 is the network device associated with the desired IB device port:

# ethtool -a enP2s13
Pause parameters for enP2s13:
Autonegotiate: off
RX: on
TX: on

Sysctl settings

The TCP port must not be shared by IPv4 and IPv6 connections. Use the following sysctl call to ensure that an IPv6 socket binds only to IPv6 addresses:

# sysctl -w net.ipv6.bindv6only=1

SMC requires contiguous memory. The minimum is 16 KB, and the maximum is 512 MB. The SMC implementation selects a value as follows:

Some socket applications define the socket send- and receive buffer sizes with a setsockopt call, whose upper limits are defined innet.core.wmem_max and net.core.rmem_max.
If setsockopt SO_SNDBUF is not used, the socket send buffer size is taken from the value of net.ipv4.tcp_wmem.
If setsockopt SO_RCVBUF is not used, the socket receive buffer is taken from the value of net.ipv4.tcp_rmem, rounded to the next higher power of 2.

Make an existing application use SMC

Use the preload library to make the unmodified socket application use SMC. Existing TCP/IP applications can benefit from the SMC protocol without recompiling, if they are invoked with the SMC preload library ld_pre_smc.so. See smc_run - Run a TCP socket program with the SMC protocol using a preloaded library, which makes an existing TCP/IP socket program use SMC.

Converting an application to use SMC

Alternatively, if you need to, you can convert an application.To convert an application from TCP/IP to SMC sockets, change the socket() function call from AF_INET to AF_SMC with protocol 0 and from AF_INET6 to AF_SMC with protocol 1. For example, change:

sd = socket(AF_INET, SOCK_STREAM, 0);

to:

sd = socket(AF_SMC, SOCK_STREAM, 0);

and

sd = socket(AF_INET6,SOCK_STREAM, 0);

to:

sd = socket(AF_SMC, SOCK_STREAM, 1);

Use the sockets.h header file from the glibc-header package. For more programming information, see the af_smc(7) man page.