Enabling RoCE

IBM Storage Scale support matrix

IBM Storage Scale supports RDMA on Red Hat® Enterprise Linux® and Ubuntu. IBM Storage Scale uses the VERBS programming interface to provide RDMA support. While IBM Storage Scale uses the VERBS programming interface for RDMA support, the underlying implementation of RDMA is vendor-specific.

RDMA requirements and limitations

Mellanox requirements and limitations

• RDMA is not supported on a node when both Mellanox HCAs and Intel Omni-Path HFIs are enabled for RDMA.
• You cannot use bonded ports across different adapters for use with RoCE.
• You need to have Connect X-6 adapters or later for RoCE.

RoCE configuration methods

Network requirements

For running RoCE, you can get the best performance when the network is configured as a lossless network. Depending on the vendor, components, and the topology, the requirement for the lossless network can quickly become complex and is out of scope.

For more information, see Network configuration examples for RoCE deployment.

ofed_info -s
MLNX_OFED_LINUX-x.x.x.x.x

Make sure that the IBM Storage Scale client nodes run the same MOFED level as the NSD and IBM Storage Scale System. However, in many projects and environments, it is a challenge to maintain all nodes with the same MOFED level.

It is possible that the client nodes are running the OFED software, which is distributed by the operating system. Such configurations are simple to operate and to maintain in the client clusters. However, in cases of trouble and network glitches, such configurations can cause unexpected failures.

Network topology

The acronym host is used for an endpoint in the network. It can be an IBM Storage Scale client system or an IBM Storage Scale System and NSD system.

A host configuration depends on the number of network ports that are used or needed. The number of ports a node connects with the network depends on the expected bandwidth, the number of subnets that are needed to get access to other environments and high availability requirements.

In TCP/IP environments, scaling bandwidth with multiple ports is commonly acquired with bonding network ports, which are known as link aggregation. But bonding can have some challenging complexities. It makes the deployment complex from the network perspective.

Bonding is a commonly used technology in data centers for balancing the traffic in a network. However, bonding does not help single-socket network traffic to use more than one cable. Further more, another major shortcoming of bonding is that the selection of network path is hard to predict or cannot be controlled from the application layer. Therefore, when you scale bandwidth over multiple ports, the use of RoCE is the most appropriate option.

To use a bond or not with an IBM Storage Scale system, the generic rule is to have at least one IP interface that is connected to the network for the daemon-to-daemon communication. A recommended network topology with bonding and RoCE is shown in the following figure.

Depending on using a bond, it is considered that the LACP link aggregation needs to be set in the network. Alternatively, by using IBM Storage Scale and IBM Storage Scale System building blocks, you can also rely on higher HA layers in IBM Storage Scale such as NSD and recovery group server fail over and can consider skipping bonds in your topology to make the network set up less complex. For better performance and less complex setup, it is recommended to use a configuration without bond.

For better availability, use bonded configuration. An example for a configuration without bond is shown in the following figure.

As shown in the figure, for network topology without bond, it is essential to have one IP address per network port. On the adapter that has the mmfsd IP address that is configured, additional RoCE IP address nor alias is needed until nodes in your cluster are able to communicate to this mmfsd IP address. You can configure as many aliases as the operating system version supports. You need one IP per adapter and RoCE can also use the existing IP, which is also used for TCP/IP traffic.

Running a configuration without bonds allows to later in the GPFS configuration, one can enhance the whole configuration by fabric numbers. By using such configuration, traffic on the ISL (inter-switch links) might be avoided.

MTU consideration

RDMA was introduced on InfiniBand networks. RDMA over InfiniBand supports maximum transmission unit (MTU) sizes from 256 up to 4096 Bytes.

It is recommended to adjust the MTU to jumbo frames, which is 9000 Bytes. Adjust this setting on all adapters and all nodes, communicating in your clusters. If you need to communicate with external nodes in remote networks and you cannot be sure that the path through the network supports MTU 9000 Bytes end–to–end, make sure that MTU Path Discovery is enabled. The MTU Path Discovery is enabled by default on Red Hat Enterprise Linux.

Configuring a bond

For RoCEv2 to work properly, all interfaces need an IPv6 local link address and an IPv4 address.

The bond interfaces configuration is not needed to run RoCE. If you do not need a bonded TCP/IP interface, you can proceed with next section.

Configuring regular IP interface

For RoCEv2 to work properly, all interfaces need an Ipv6 local link address and an IPv4 address. Configure all interfaces intended to use for RoCE communication, as shown in the following example.

Configuring routing

In Figure 5, the mmfsd communication runs in the 192.168.12.0/24 network. All other Mellanox cards interfaces are highlighted in green and configured to be in the subnet 10.10.10.x/24. All interfaces are intended to use for RDMA communication, while only the IP addresses in 192.168.12.0/24 network are used for TCP/IP communication.

With current IBM Storage Scale releases, only one IP address per node is supported for communication to the mmfs daemon's network.

To scale out over multiple ports, RoCE can be used. But according to the definition of OFED standards, each RDMA interface needs to have an IP address to maintain the connection.

To use multiple interfaces, you need to configure multiple subnets. It is complex for larger environments and technically not needed. It is possible to configure all interfaces that are intended to be used for RDMA into one separate subnet.

As a limitation of the Red Hat Enterprise Linux kernel, multiple interfaces to the same subnets are complex. When you have more than one interface per subnet on a node, answering arp requests, selecting rules for outgoing IP traffic, and other complexities need to be reconsidered. To resolve such issues, you must add some special routing tables, described in 2. You need to specify routing entries for each interface and for all subnets, where you have more than one interface connected.

In example, each I/O server node has four interfaces that are connected to the same physical network. These interfaces need to get an IP address in a different IP-range(subnet) than the mmfsd but they are all in the same network. Refer to the following figure for topology overview.

For each green-marked interface, a routing entry is needed.

1. Enable routing for each RoCE interface.

   essgennetworks -N <IOnoden> --configureRouteForRoCE --roceRoutingTableId 101 --roceRoutingTableName t1 --interface enP48p1s0f0

   A sample output is as follows:

   2021-10-25T04:51:53.048486 [INFO] Starting network generation...
   2021-10-25T04:51:53.111121 [INFO] nodelist:  <IOnoden>
   2021-10-25T04:52:13.897913 [INFO] Interface(s) available on node <IOnoden>
   2021-10-25T04:52:13.897991 [INFO] Considered interface(s) of node <IOnoden> are ['enP48p1s0f0'] with RDMA Port ['mlx5_2'] for this operation
   2021-10-25T04:52:14.202172 [WARN] MTU is NOT set to 9000 for interface enP48p1s0f0 at node <IOnoden>. It's advisible to set MTU as 9000 for better performance.
   2021-10-25T04:52:14.202235 [WARN] Interface MTU NOT set to 9000 at node <IOnoden>
   2021-10-25T04:52:14.202268 [WARN] Please re-configure the interface with --mtu 9000 and make sure switch is also configured to use --mtu 9000 i.e. Jumbo frame for better performance at node <IOnoden>
   2021-10-25T04:52:14.206459 [INFO] Interface enP48p1s0f0 have the IPv4 Address assigned at node <IOnoden>
   2021-10-25T04:52:14.210723 [INFO] Interface enP48p1s0f0 have the IPv6 Address assigned at node <IOnoden>
   2021-10-25T04:52:14.225432 [INFO] RoCE routing validation passes.
   2021-10-25T04:52:14.225488 [INFO] Setting UP routing for interface RDMA for Interface ['enP48p1s0f0'] at node <IOnoden>
   2021-10-25T04:52:14.228225 [INFO] Successfully added routing id and it's table name inside /etc/iproute2/rt_tables at node <IOnoden>
   2021-10-25T04:52:14.672155 [INFO] Successfully configured route for interface enP48p1s0f0 at node <IOnoden>
   2021-10-25T04:52:14.672219 [INFO} Routing configuration for subnet has configured successfully.

2. Check the routing table entry.

   ip r

   10.111.222.100/30 dev enP1p8s0f3 proto kernel scope link src 10.111.222.101 metric 101 linkdown
   192.168.2.0/24 dev bond0 proto kernel scope link src 192.168.2.51 metric 300
   192.168.4.0/24 dev enP48p1s0f0 proto kernel scope link src 192.168.4.51 metric 104
   192.168.15.0/24 dev enP1p8s0f0 proto kernel scope link src 192.168.15.51 metric 100

   Repeat these steps to configure multiple RoCE interfaces for all interfaces .

Configuring the sysctl file

In addition to the interface scripts, you need system-wide settings that are managed by sysctl. Depending on the IBM Storage Scale System version, a customized sysctl setting is available by a tuned profile, which is named as scale. You need to edit the /etc/tuned/scale/tuned.conf file. For any other client node in your cluster, you can deploy the same sysctl configuration file.

1. Ensure that the following sysctl settings are applied:

   net.ipv6.conf.all.disable_ipv6=0
   net.ipv6.conf.default.disable_ipv6=0

   Note: The IPv6 configuration can be enabled by the default IBM Storage Scale System deployment process.
2. Validate the verbsPorts configuration.

   mmlsconfig
   [<MSn>-ce,<IOnode1>-ce,<IOnode2>-ce]
   verbsRdma enable
   verbsRdmaCm enable
   verbsRdmaSend yes
   [<MSn>-ce,<IOnode2>-ce]
   verbsPorts mlx5_bond_0
   [<IOnode1>-ce]
   verbsPorts mlx5_2

3. You can run the mmdiag --network command and check for the verbsRDMA connection.