Using a PCI function for SMC-R connections

Start your application with the smc_run command to enable it for SMC-R connections.

The smc_run command uses the libsmc-preload.so preload library for the application. This library makes existing TCP/IP socket programs use SMC if both communication peers support it. After an initial handshaking through a TCP/IP connection, the PCI function is used in RDMA mode for all further traffic.

Before you begin

Both the smc_run command and the libsmc-preload.so preload library are included in the smc-tools package.

For SMC-Rv2, your network adapter must be RoCE Express®2, RoCE Express3 or Network Express. For RoCE Express, the smc_info command must list v2 for the SMC-R Features and for RoCE as seen in the following example.

# smcr info
Kernel Capabilities
SMC Version: 2.0
SMC Hostname: t8345009.host.name
SMC-R Features: v1 v2

Hardware Capabilities
SEID: IBM-SYSZ-ISMSEID000000002E488561
ISM:  v1 v2
RoCE: v1 v2

For SMC-R version 1, the communication partners must be in the same subnet. As of RoCE Express2, RoCE Express3 or Network Express using SMC-R version 2, communication partners can be in different IP subnets. For SMC-Rv2, SMC traffic is regulated through enterprise IDs (EIDs) (see Setting up connections with version 2 of SMC-D or SMC-R).

About this task

To establish an SMC-R connection you need two interfaces: a TCP/IP interface and an RDMA interface. The two interfaces must be associated through a matching PNET ID. PNET IDs label interfaces as belonging to a particular physical network, see Managing PNET IDs.

Because network adapter PCI functions on Linux® on IBM® Z can provide both interfaces you have a choice.

Use a TCP/IP interface from an OSA adapter and associate it with PCI function for the RDMA interface.
Use the PCI function for both interfaces.
For this choice, assigning the appropriate PNET ID merely labels the PCI function as belonging to a physical network. You do not need a PNET ID to associate the TCP/IP and RDMA interface of the same PCI function.

If a connection is opened while no functional SMC-R link is available, TCP/IP is used. For established connections, SMC-R links do not revert to TCP/IP. Similarly, a version 2 SMC-R connection requires both communication partners to support version 2. If one partner does not support version 2, the connection falls back to version 1.

Configure for link failover by assigning the same PNET ID to multiple PCI functions that can reach the intended peer. For such setups, the TCP/IP link must be based on an OSA adapter. Typical high availability setups use two bonded TCP/IP interfaces, each from a different OSA adapter, and two redundant PCI functions f, all associated with the same PNET ID (see Path redundancy for SMC-R connections).

Procedure

Confirm that your communication network, the network interfaces on your Linux instance, and the intended communication peer are all set up correctly for an SMC-R connection.
Issue an smc_chk command to confirm the setup.
Resolve any issues until the smc_chk command confirms the setup.
```
# smc_chk -C 192.168.5.47 -p 23
  Live test (SMC-D and SMC-R)
     Success, using SMC-R
```
Start your application with the smc_run command according to the following syntax.
```
# smc_run <smc_run_parameters> <program> <program_parameters>
```
For more information about the smc_run command, see the man page or issue smc_run -h.

Example

This example specifies two smc_run parameters, -r to set a receive buffer to 16 KB, and -t to set a transfer buffer to 512 KB. The command starts a program iperf3 with program-specific parameters -s and -p.

# smc_run -r 16k -t 512k iperf3 -s -p 12345