Because the 10 GBE
RoCE Express feature hardware physically consists of a Mellanox adapter,
you must ensure that the following prerequisites are fulfilled before
you can work with it.
Procedure
- Ensure that PCIe support is enabled and the PCI card is
active on your system.
- Use the appropriate Mellanox device driver:
- To use TCP/IP, you need the
mlx4_core and
mlx4_en or mlx5_core module.
- To also use RDMA with InfiniBand, you further need the
mlx4_ib or mlx5_ib module. You can use SMC sockets or reliable
datagram sockets (RDS).
- For SMC, the SMC protocol
support must be in place.
- For RDS, you need the
rds module and the rds_rdma module, see
Documentation/networking/rds.txt in the Linux® source tree and the rds and
rds-rdma man pages.
Load any modules that are not compiled into the kernel or already loaded, for example, with
modprobe.
-
Activate the network interface.
You need to know the network interface name, which you can find under:
- /sys/bus/pci/drivers/mlx4_core/<pci_slot>/net/<interface> for Mellanox
ConnectX-3.
- /sys/bus/pci/drivers/mlx5_core/<pci_slot>/net/<interface> for Mellanox
ConnectX-4.
Use the
ip command or equivalent to activate the interface. See the
dev_port sysfs attribute of the interface name to ensure that you are working
with the correct port.
Note that the numbering of network device ports start with
0, but the numbering of InfiniBand device ports start with 1. For
example:# cat /sys/class/infiniband/mlx4_0/ports/
1/ 2/
What to do next
For further information about Mellanox, see: