Working with the RoCE support

Edit online

Ubuntu 22.04 LTS LPAR mode z/VM guest KVM guest

Because the 10 GBE RoCE Express feature hardware physically consists of a Mellanox adapter, you must ensure that the following prerequisites are fulfilled before you can work with it.

Procedure

  1. Ensure that PCIe support is enabled and the required PCI cards are active on your system.
  2. Use the appropriate Mellanox device driver:
    • If you want to use TCP/IP, you need the mlx4_core module and mlx4_en or mlx5_core module. If it is not compiled into the kernel or already loaded, load it using for example, modprobe.
    • If you also want to use RDMA with InfiniBand (that is, using reliable datagram sockets, RDS), you need the mlx4_ib or mlx5_ib module. If it is not compiled into kernel or already loaded, load it using for example, modprobe. To use RDS, you also need the rds module and the rds_rdma module, see Documentation/networking/rds.txt in the Linux® source tree and the rds and rds-rdma man pages.
  3. Activate the network interface.
    You need to know the network interface name, which you can find under:
    • /sys/bus/pci/drivers/mlx4_core/<pci_slot>/net/<interface> for RoCE Express.
    • /sys/bus/pci/drivers/mlx5_core/<pci_slot>/net/<interface> for RoCE Express 2.
    Use the ip command or equivalent to activate an interface. See the dev_port sysfs attribute of the interface name to ensure that you are working with the correct port. Note that the numbering of network device ports start with 0, but the numbering of InfiniBand device ports start with 1. For example:
    # cat /sys/class/infiniband/mlx4_0/ports/
1/ 2/