Scenario: Setting up Linux for SMC-R

To an existing OSA-based network connection, add a connection through a RoCE Express adapter and configure the network interfaces for SMC-R traffic.

# smcr info 
Kernel Capabilities 
SMC Version:      2.0 
SMC Hostname:     a3545024.lnxne.boe               
SMC-D Features:   v1 v2 
SMC-R Features:   v1 v2 

Hardware Capabilities 
SEID:             n/a 
ISM:              n/a 
RoCE:             v1 v2

About this task

The scenario assumes that a Linux instance runs in a z/VM® guest virtual machine with z/VM user ID A3545024. The z/VM administrator already enabled z/VM for PCI usage and made the PCI function with FID 110e available to the Linux instance. In the IOCDS, the hardware definition assigns PNET ID NET10 to the PCI function. The PCI function connects to an external LAN.

Through a z/VM VSWITCH and an OSA-Express adapter, the Linux instance has a TCP/IP connection to the same external LAN as the PCI function. The OSA interface on Linux is assumed to be encbdf0.

The goal is to add another connection for SMC-R traffic through the PCI function of the RoCE Express adapter.

Perform the following steps from Linux.

1. List the available PCI functions.

   # vmcp 'query pcifunction' 
   PCIF 0000110E FREE                          DISABLED RoCE Express 2

   The DISABLED in the output indicates that the PCI function is not yet attached to the z/VM guest virtual machine.
2. Attach the PCI function.

   # vmcp 'attach pcifunction 0000110E to *'
   PCI FUNCTION 0000110E ATTACHED TO A3545024 0000110E

   Repeating the query command now shows the PCI function as attached to A3545024.

   # vmcp 'query pcifunction' 
   PCIF 0000110E ATTACHED TO A3545024 0000110E ENABLED  RoCE Express 2

3. Display information for the PCI function.

   # smc_rnics  
       FID  Power  PCI_ID        PCHID  Type           PPrt  PNET_ID           Net-Dev 
   ----------------------------------------------------------------------------------------- 
      110e  1      110e:00:00.0  020c   RoCE_Express2  0     NET10             eno4366

   The command output shows NET10 as the PNET ID that the hardware configuration assigns to the PCI function.
4. Create a link with an IP address for the PCI function.
   This link is required for the SMC-R setup but it must not be used for any IP traffic. The following ip commands set up a suitable but transient link for our example. For a persistent setup, use the network manager of your distribution.

   # ip link add dev mylink link eno4366
   # ip addr add 192.0.2.1/24 dev mylink
   # ip link set mylink up
   # ip route flush scope link dev mylink

   The final command removes all possible routes for this link and so prevents any traffic. The network manager of your distribution might interpret this link configuration as erroneous and take corrective actions. To preserve the configuration, remove the interface from the scope of the network manager. For example, to stop NetworkManger from changing the link configuration, issue the following nmcli command:

   # nmcli device set eno4366 managed no

5. Assign the PNET ID of the PCI function to the network interface that the z/VM VSWITCH provides to the guest.
   Because this interface is a virtual network component, the PNET ID cannot be set in the hardware configuration. Therefore, use the smc_pnet command to create an entry in a PNET table.

   # smc_pnet --add NET10 --interface encbdf0
   

6. Confirm your setup with the smc_chk command.
   • If a peer with PNET ID NET10 is already set up for an SMC-R connection, you can test the connection to the peer.
     The following command confirms SMC-R connectivity to a peer, for example, an SMC-R capable service on a z/OS® instance with IP address 192.168.5.47 and port 23.

     # smc_chk -C 192.168.5.47 -p 23
       Live test (SMC-D and SMC-R)
         Success, using SMC-R
     

   • Alternatively, you can confirm the local setup through a loopback connection to a local server application.
     In the following example, the command starts a local server on port 37373.

     # smc_chk -S &
     Server started on port 37373

     The following command confirms that the server can be reached through an SMC-R connection.

     # smc_chk -C 127.0.0.1 -p 37373
     Test with target IP 127.0.0.1 and port 37373
       Live test (SMC-D and SMC-R)
          Success, using SMC-R

     The following command stops the server.

     # killall smc_chk