RDMA over Converged Ethernet (RoCE)

This section describes support for RDMA over Converged Ethernet, or RoCE.

Host Networking

IBM Storage Scale requires the host device to be passed into the pod network namespace to detect the interface. If the device cannot be detected, you see an error message in mmfs.log.latest:

[W] VERBS RDMA parse error   verbsPort mlx5_3/1   ignored due to interface not found for port 1 of device mlx5_3 with GID fb476ffeffd3ebea. Please check if the correct inet6 address for the corresponding IP network interface is set

To pass the host-device in as a CNI interface, define a NetworkAttachmentDefinition. The next example defines a resource that is called ens5f1np1-roce-host-net:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: ens5f1np1-roce-host-net
  namespace: ibm-spectrum-scale
spec:
  config: |-
    {
      "cniVersion": "0.3.1",
      "name": "ens5f1np1-roce-host-net",
      "type": "host-device",
      "device": "ens5f1np1",
      "ipam": {
        "type": "static"
      }
    }

The resource has the following properties:

• the CNI network name is ens5f1np1-roce-host-net. Use this when annotating the nodes with scale.spectrum.ibm.com/daemon-network.
• host-device is set as the CNI type
• device is defined as ens5f1np1, this is the physical Ethernet interface on the node where core pods are deployed.
• the definition is defined in the ibm-spectrum-scale namespace.

IBM Storage Scale Container native cluster CR

If you are creating an IBM Storage Scale container native cluster that uses NVIDIA Ethernet networking, start by downloading the sample cluster CR from NVIDIA InfiniBand Cluster CR.

To enable RDMA over Converged Ethernet (RoCE), uncomment the RDMA-specific configuration from the sample CR and make the following changes:

• Change verbsPort to a value matching your specific hardware configuration. This value should match the network interface name for the Ethernet device. For example: "mlx5_3", "ens5f1np1", and so on.
• verbRdmaCm must be set to "enable".

If you have an existing cluster already that uses NVIDA InfiniBand networking, enable RDMA by adding the following verbs* tuning parameters to your existing cluster resource configuration:

apiVersion: scale.spectrum.ibm.com/v1beta1
kind: Cluster
metadata:
  name: ibm-spectrum-scale
spec:
  daemon:
    clusterProfile:
      # -------------------------------------------------------------------------------
      # RDMA specific configuration.
      # Change verbsPort to value based on your configuration. This value should match the
      # network interface name for the InfiniBand device. For example: "mlx5_2", "ibs5f0", etc.
      # -------------------------------------------------------------------------------
      verbsPorts: mlx5_3
      verbsRdma: enable
      verbsRdmaSend: "yes"
      # Only enable `verbsRdmaCm` when deploying RoCE - RDMA over Converged Ethernet
      verbsRdmaCm: enable
      ...