Managing nodes that use RDMA-capable Ethernet ports

Edit online

The system supports node-to-node connections that use Ethernet protocols that support remote direct memory access (RDMA) technology, such as RDMA over Converged Ethernet (RoCE) or iWARP. To use these protocols, the system requires that an RDMA-capable adapter is installed on each node and dedicated RDMA-capable Ethernet ports are only configured for node-to-node communication.

RDMA technologies, like RoCE and iWARP, enable the RDMA-capable adapter to transfer data directly between nodes, bypassing CPU and caches, making transfers faster. RDMA technologies provide faster connection and processing time than traditional iSCSI connections.

The following prerequisites apply to all RDMA-capable Ethernet ports that are used between nodes:
  • All installation of the node hardware is complete.
  • The 25 Gbps Ethernet adapter that supports RDMA technology is installed on each node. If you are using RDMA-technology for node-to-node communications, ensure that the RDMA-capable adapters use the same technology, such as RoCE or iWARP. These RDMA-capable adapters must be installed in the same slots across all the nodes of the system. These installation requirements ensure that port identifiers are the same across all nodes in the system.
    Note: The 100-Gbps Ethernet port does not support node-to-node communication.
  • Ethernet cables between each node are connected correctly.
  • The protocol technology on the source and destination adapters is the same.
  • The local and remote IP addresses can be reached.
  • Each IP address for RDMA-capable Ethernet ports and their associated subnet masks are unique on each node.
  • Router must not be placed between nodes that use RDMA-capable Ethernet ports for node-to-node communication.
  • The negotiated speeds on the local and remote adapters are the same.
  • The local and remote port virtual LAN identifiers are the same.
  • A minimum of two dedicated RDMA-capable Ethernet ports are required for node-to-node communications to ensure best performance and reliability. These ports must be configured for inter-node traffic only and must not be used for host attachment, virtualization of Ethernet-attached external storage, or IP replication traffic.
  • A maximum of four RDMA-capable Ethernet ports per node are allowed for node-to-node communications.

The basic node tasks, such as adding an node or removing a node, are the same for both Fibre Channel and RDMA-based connections between nodes, but you can need to complete management actions on the RDMA-capable Ethernet ports beforeto completing node-level management tasks.

Before completing managing tasks that are related to RDMA-capable Ethernet ports on a node, use the following best practices to manage these ports:
  • If you already have a system that is configured to use RDMA-capable Ethernet ports, you must ensure that one redundant path is available before adding, removing, or updating settings for RDMA-capable Ethernet ports.
  • Only add, remove, or update settings on one RDMA-capable Ethernet port at a time. Wait 15 seconds between these changes before updating other RDMA-capable Ethernet ports.
  • If you are using virtual LAN to create physical separation of networks, ensure that you follow these additional guidelines when completing managing related tasks:
    • VLAN IDs cannot be update or added independently of other settings on a RDMA-capable Ethernet port, such as IP address.
    • Before adding or updating VLAN ID information to RDMA-capable Ethernet ports, you must configure VLAN support on the all the Ethernet switches in your network. For example, on each switch, set VLAN to "Trunk" mode and specify the VLAN ID for the RDMA-capable Ethernet ports that will be in the same VLAN.