Setting up the SMC support

SMC traffic requires a mapping of the OSA or HiperSockets network interfaces to the RoCE adapters or ISM devices.

Physical network (PNET) IDs provide this mapping. If a network interface and an RoCE or ISM device have the same PNET ID, they are connected to the same physical network and can be used together for SMC. Assign PNET IDs to OSA, HiperSockets, RoCE, and ISM devices through the IOCDS. For more information about IOCDS, see z/OS HCD User's Guide, SC34-2669.

Figure 1 illustrates how the IOCDS assigns the PNETID NET1 to an ISM device and a network interface for an Ethernet device. In Linux, the matching PNETID associates the ISM device with an Ethernet device.

Figure 1. PNET ID and ISM device association
The PNET on the Ethernet device and in the SMC definition must be the same

As a fallback solution, SMC-R supports a software PNET table that maps network interfaces to RoCE adapters. Use the smc_pnet command to create a physical network (PNET) table with this mapping. SMC-D does not support such a software table.

Note: z/OS® does not support the Mellanox card as an Ethernet device, and therefore uses OSA adapters for the initial handshake for SMC-R connections. Linux has no such constraint.

Network device settings

On the network device that is associated with the RoCE device you want to use for SMC traffic, check the settings with the ethtool command and ensure that pause settings are turned on.

For example, if enP2s13 is the network device that is associated with the wanted device port:
# ethtool -a enP2s13
Pause parameters for enP2s13:
Autonegotiate: off
RX: on
TX: on

Sysctl settings

SMC requires contiguous memory. The minimum is 16 KB, and the maximum is 512 MB. The SMC implementation selects a value as follows:
  • Some socket applications define the socket send- and receive buffer sizes with a setsockopt call, whose upper limits are defined in net.core.wmem_max and net.core.rmem_max.
  • If setsockopt SO_SNDBUF is not used, the socket send buffer size is taken from the value of net.ipv4.tcp_wmem.
  • If setsockopt SO_RCVBUF is not used, the socket receive buffer is taken from the value of net.ipv4.tcp_rmem, rounded to the next higher power of 2.

Make an existing application use SMC

Use the preload library to make the unmodified socket application use SMC. Existing TCP/IP applications can benefit from the SMC protocol without recompiling, if they are invoked with the SMC preload library ld_pre_smc.so. See the smc-tools package for the smc_run script, which makes an existing TCP/IP socket program use SMC.

Converting an application to use SMC

Alternatively, if you need to, you can convert an application. To convert an application from TCP/IP to SMC sockets, change the socket() function call from AF_INET to AF_SMC with protocol 0 and from AF_INET6 to AF_SMC with protocol 1. For example, change:
sd = socket(AF_INET, SOCK_STREAM, 0);
to:
sd = socket(AF_SMC, SOCK_STREAM, 0);
and
sd = socket(AF_INET6,SOCK_STREAM, 0);
to:
sd = socket(AF_SMC, SOCK_STREAM, 1);
Use the sockets.h header file from the glibc-header package. For more programming information, see the af_smc(7) man page.