Working with the RoCE support

Edit online

Red Hat Enterprise Linux 9.2 LPAR mode z/VM guest KVM guest

Because the RoCE Express feature hardware physically consists of a Mellanox adapter, you must ensure that the following prerequisites are fulfilled before you can work with it.

Procedure

  1. Ensure that PCIe support is enabled and the required PCI cards are active on your system.
  2. Use the appropriate Mellanox device driver:
    • To use TCP/IP, you need the mlx4_core and mlx4_en or mlx5_core module. If it is not already loaded, load it using for example, modprobe.
    • To also use RDMA with InfiniBand you need the mlx4_ib or mlx5_ib module. You can use SMC sockets or reliable datagram sockets (RDS).
      • For SMC, the SMC protocol support must be in place.
      • For RDS, you need the rds module and the rds_rdma module, see Documentation/networking/rds.txt in the Linux® source tree and the rds and rds-rdma man pages.
      Load any modules that are already loaded, for example, with modprobe.
  3. Activate the network interfaces.
    You need to know the network interface name, which you can find under:
    • /sys/bus/pci/drivers/mlx4_core/<pci_slot>/net/<interface> for RoCE Express.
    • /sys/bus/pci/drivers/mlx5_core/<pci_slot>/net/<interface> for RoCE Express2.
    Use the ip command or equivalent to activate an interface. See the dev_port sysfs attribute of the interface name to ensure that you are working with the correct port. Note that the numbering of network device ports start with 0, but the numbering of InfiniBand device ports start with 1. For example:
    # cat /sys/class/infiniband/mlx4_0/ports/
1/ 2/