Migrating from Fibre Channel connections to RDMA over Ethernet connections between nodes
The system supports node-to-node connections that use Ethernet protocols that support remote direct memory access (RDMA) technology, such as RDMA over Converged Ethernet (RoCE) or iWARP. To use these protocols, the system requires that an RDMA-capable adapter is installed on each node and dedicated RDMA-capable Ethernet ports are only configured for node-to-node communication. If your system currently uses Fibre Channel ports, you can migrate to RDMA-capable Ethernet ports for node-to-node communications.
RDMA technologies, like RoCE and iWARP, enable the RDMA-capable adapter to transfer data directly between nodes, bypassing CPU and caches, making transfers faster. RDMA technologies provide faster connection and processing time than traditional iSCSI connections.
Using the service assistant and management GUIs
- On all Fibre Channel-connected nodes, upgrade the software to the 8.2.1 release.
- After all nodes are upgraded, log in to the service assistant GUI and
complete these steps:
- Using a supported browser, enter
https://ip_address/service
, where the ip_address is the management IP address of the system. - On the Home page, select a node that is not the configuration node of the system. If you do not know whether the node is the configuration node, select a node and review the information on Node Details tab.
- After you have selected the node, select Power off from the Actions menu. Click Go.
- Using a supported browser, enter
- After the node completes powering off, you need to install the 25 Gbps Ethernet adapter on the node. Use installation instructions for the 25 Gbps Ethernet adapter to complete this step.
- Configure the IP addresses for the RDMA-capable Ethernet ports on the
adapter by completing these steps in the service assistant GUI:
- In the service assistant GUI, select the node that was powered off and select Change Node IP.
- On the Change Node IP panel, select the port to define IP addresses and click Modify.
- Enter the IP address, subnet mask, gateway, and VLAN ID for the selected RDMA-capable Ethernet
ports. Note: The IP address and subnet mask for each port must be unique and cannot be used anywhere else on the system. However, the VLAN ID for RDMA-capable Ethernet ports on all nodes must be the same, or connections between the nodes using these IP addresses will fail.
- Click Save.
- Repeat these steps for all RDMA-capable Ethernet ports that are associated with the node.
- After updating all the RDMA-capable Ethernet ports add the node into the system by completing
these steps in the management GUI:
- Select .
- Select Add Node from the System Actions menu.
- After the node is added to the system, verify that the status of the node is
online on the System - Overview page. If the node is
listed as offline or if errors occur, select to view error messages or select to view port connectivity for the node to view potential connection problems. You can
also use the ping command to troubleshoot connectivity issues between the nodes.
This page displays connection statuses between nodes. If a connection issue is present, error data
information is provided to help determine what is causing the problem:
- Discovered
- Indicates that an Ethernet port on the selected node is configured, but a connection cannot be
established. This status indicates a potential problem that needs to be resolved. The
Error Data column indicates the reason for the
Discovered status. The following values are possible in the Error
Data column:
- Protocol mismatch
- Indicates that the protocol on the source and destination adapters is not the same. This error occurs when one node in the system does not have the 25 Gbps Ethernet adapter installed.
- Unreachable
- Indicates that the local and remote IP addresses cannot be reached. This error can occur if one of the nodes in the system is offline. Select to view errors and run any necessary fix procedures to return the node to an online status.
- Duplicate IP addresses
- Indicates that one or more IP addresses are being used in the network. Each node IP address must be unique. To fix this error, you can use the Service Assistant interface to change the node IP address.
- Degraded
- Indicates that the negotiated speed on both the local and remote adapters is not the same. Degraded status occurs when one or both adapters are configured at lower speed rather than the maximum speed that the adapters support. To fix this issue, ensure that adapters on both nodes are configured at the maximum speed.
- VLAN ID Mismatch
- Indicates that the local and remote port virtual LAN identifiers are not the same. To fix this error, ensure that the local and remote nodes belong to the same VLAN. If you are using VLAN in your network, you must configure VLAN on switches by setting VLAN to "Trunk" mode and specifying the VLAN ID on the switch before you configure IP addresses and other settings for the RDMA-capable Ethernet ports on the nodes in the system.
- Repeat steps 2 to 4 on each node until all the nodes are configured with RDMA-capable Ethernet ports.
- Power off each node that remains in the system one at a time by completing these steps:
- Using a supported browser, enter https://ip_address/service where the ip_address is the management IP address of the system.
- On the Home page, select a node that is not the configuration node of the system. If you do not know whether the node is the configuration node, select a node and review the information on Node Details tab.
- After you have selected the node, select Power off from the Actions menu. Click Go.
- For all the Fibre Channel adapters that are on the nodes, use the appropriate remove procedure that corresponds with the type of Fibre Channel adapters that you are migrating from.
- Ensure that hosts can now connect to the nodes.
Using the command-line interface
- On all Fibre Channel connected nodes, upgrade the software to the 8.2.1 release.
- On a node that is not the current configuration node of the system, enter
the following command:
satask stopnode -poweroff
- After the node completes powering off, you need to install the 25 Gbps Ethernet adapter on the node. Use installation instructions for the 25 Gbps Ethernet adapter to complete this step.
- To define the IP address for RDMA-capable Ethernet ports in the
command-line, complete these steps:
- To define port IP addresses for RDMA-capable Ethernet ports, enter the following command on each
port for each node within the
system:
where ip_address is the IP address for the port that is identified by its port identifier (port_number) and node name (panel_name). Ensure that all ports are in the same VLAN by specifying the same vlanid for each IP address on the RDMA-capable Ethernet port.satask chnodeip -ip ip_address -mask mask -gw gateway -port_id port_number -vlan vlanid panel_name
Note:- The IP address and subnet mask for each port must be unique and cannot be used anywhere else on the system. However, the VLAN ID for RDMA-capable Ethernet port, on all nodes must be the same, or connections between the nodes that use these IP addresses will fail.
- The 100-Gbps Ethernet port does not support node-to-node communication.
- To add the node to the system, complete these steps:
- Ensure that the node is listed as a candidate node by entering the following
command:
In the results that display, verify that the id parameter displays the WWNN for the node. If the node is not detected, verify cabling to the node.svcinfo lsnodecandidate
- Enter the following command to determine the I/O group where the node must be added:
lsiogrp
- Record the name or ID of the first I/O group that has a node count of zero. You need the name or
ID for the next step. Note: You must do this step for the first node that is added. You do not do this step for the second node of the pair because it uses the same I/O group number.
- Enter this command to add the node to the system:
Where WWNN is the WWNN of the node, iogrp_name is the name of the I/O group that you want to add the node to, and new_name_arg is the name that you want to assign to the node. If you do not specify a new node name, a default name is assigned.addnode -wwnodename WWNN -iogrp iogrp_name -name new_name_arg
- Ensure that the node is listed as a candidate node by entering the following
command:
- Verify connectivity between the nodes that use RDMA-capable Ethernet ports, by entering this
command:
In the results that display, a value of Status:Connected indicates a successful connection. A value of Status:Discovered indicates that port on the selected node is configured, but a connection cannot be established. If connection errors occur, possible reasons for these errors are displayed in the error_data parameter. For descriptions of these error states, see the management GUI instructions. You can also use the ping command to troubleshoot connectivity issues between the nodes.sainfo lsnodeipconnectivity
- To define port IP addresses for RDMA-capable Ethernet ports, enter the following command on each
port for each node within the
system:
- Repeat steps 2 to 4 on each node until all the nodes are configured with RDMA-capable Ethernet ports.
- Power off each node that remains in the system one at a time by entering the following command:
Each node needs to be completely powered off before issuing his command on another node.satask stopnode -poweroff