Migrating from Fibre Channel connections to RDMA over Ethernet connections between nodes

Edit online

The system supports node-to-node connections that use Ethernet protocols that support remote direct memory access (RDMA) technology, such as RDMA over Converged Ethernet (RoCE) or iWARP. To use these protocols, the system requires that an RDMA-capable adapter is installed on each node and dedicated RDMA-capable Ethernet ports are only configured for node-to-node communication. If your system currently uses Fibre Channel ports, you can migrate to RDMA-capable Ethernet ports for node-to-node communications.

RDMA technologies, like RoCE and iWARP, enable the RDMA-capable adapter to transfer data directly between nodes, bypassing CPU and caches, making transfers faster. RDMA technologies provide faster connection and processing time than traditional iSCSI connections.

The following prerequisites apply to all RDMA-capable Ethernet ports that are used between nodes:

All installation of the node hardware is complete.
The 25 Gbps Ethernet adapter that supports RDMA technology is installed on each node. If you are using RDMA-technology for node-to-node communications, ensure that the RDMA-capable adapters use the same technology, such as RoCE or iWARP. These RDMA-capable adapters must be installed in the same slots across all the nodes of the system. These installation requirements ensure that port identifiers are the same across all nodes in the system.
Note: The 100-Gbps Ethernet port does not support node-to-node communication.
Ethernet cables between each node are connected correctly.
The protocol technology on the source and destination adapters is the same.
The local and remote IP addresses can be reached.
Each IP address for RDMA-capable Ethernet ports and their associated subnet masks are unique on each node.
Router must not be placed between nodes that use RDMA-capable Ethernet ports for node-to-node communication.
The negotiated speeds on the local and remote adapters are the same.
The local and remote port virtual LAN identifiers are the same.
A minimum of two dedicated RDMA-capable Ethernet ports are required for node-to-node communications to ensure best performance and reliability. These ports must be configured for inter-node traffic only and must not be used for host attachment, virtualization of Ethernet-attached external storage, or IP replication traffic.
A maximum of four RDMA-capable Ethernet ports per node are allowed for node-to-node communications.

Using the service assistant and management GUIs

To migrate from Fibre Channel ports to RDMA-capable Ethernet ports for node-to-node communications, complete these steps:

On all Fibre Channel-connected nodes, upgrade the software to the 8.2.1 release.
After all nodes are upgraded, log in to the service assistant GUI and complete these steps:
1. Using a supported browser, enter https://ip_address/service, where the ip_address is the management IP address of the system.
2. On the Home page, select a node that is not the configuration node of the system. If you do not know whether the node is the configuration node, select a node and review the information on Node Details tab.
3. After you have selected the node, select Power off from the Actions menu. Click Go.
After the node completes powering off, you need to install the 25 Gbps Ethernet adapter on the node. Use installation instructions for the 25 Gbps Ethernet adapter to complete this step.
Configure the IP addresses for the RDMA-capable Ethernet ports on the adapter by completing these steps in the service assistant GUI:
1. In the service assistant GUI, select the node that was powered off and select Change Node IP.
2. On the Change Node IP panel, select the port to define IP addresses and click Modify.
3. Enter the IP address, subnet mask, gateway, and VLAN ID for the selected RDMA-capable Ethernet ports.
  Note: The IP address and subnet mask for each port must be unique and cannot be used anywhere else on the system. However, the VLAN ID for RDMA-capable Ethernet ports on all nodes must be the same, or connections between the nodes using these IP addresses will fail.
4. Click Save.
5. Repeat these steps for all RDMA-capable Ethernet ports that are associated with the node.
After updating all the RDMA-capable Ethernet ports add the node into the system by completing these steps in the management GUI:
1. Select Monitoring > System.
2. Select Add Node from the System Actions menu.
3. After the node is added to the system, verify that the status of the node is online on the System - Overview page. If the node is listed as offline or if errors occur, select Monitoring > Events to view error messages or select Settings > Network > Ethernet Connectivity to view port connectivity for the node to view potential connection problems. You can also use the ping command to troubleshoot connectivity issues between the nodes. This page displays connection statuses between nodes. If a connection issue is present, error data information is provided to help determine what is causing the problem:
  
  Discovered
  
  Indicates that an Ethernet port on the selected node is configured, but a connection cannot be established. This status indicates a potential problem that needs to be resolved. The Error Data column indicates the reason for the Discovered status. The following values are possible in the Error Data column:
  
  Protocol mismatch
  
  Indicates that the protocol on the source and destination adapters is not the same. This error occurs when one node in the system does not have the 25 Gbps Ethernet adapter installed.
  
  Unreachable
  
  Indicates that the local and remote IP addresses cannot be reached. This error can occur if one of the nodes in the system is offline. Select Monitoring > Events to view errors and run any necessary fix procedures to return the node to an online status.
  
  Duplicate IP addresses
  
  Indicates that one or more IP addresses are being used in the network. Each node IP address must be unique. To fix this error, you can use the Service Assistant interface to change the node IP address.
  
  Degraded
  
  Indicates that the negotiated speed on both the local and remote adapters is not the same. Degraded status occurs when one or both adapters are configured at lower speed rather than the maximum speed that the adapters support. To fix this issue, ensure that adapters on both nodes are configured at the maximum speed.
  
  VLAN ID Mismatch
  
  Indicates that the local and remote port virtual LAN identifiers are not the same. To fix this error, ensure that the local and remote nodes belong to the same VLAN. If you are using VLAN in your network, you must configure VLAN on switches by setting VLAN to "Trunk" mode and specifying the VLAN ID on the switch before you configure IP addresses and other settings for the RDMA-capable Ethernet ports on the nodes in the system.
Repeat steps 2 to 4 on each node until all the nodes are configured with RDMA-capable Ethernet ports.
Power off each node that remains in the system one at a time by completing these steps:
1. Using a supported browser, enter https://ip_address/service where the ip_address is the management IP address of the system.
2. On the Home page, select a node that is not the configuration node of the system. If you do not know whether the node is the configuration node, select a node and review the information on Node Details tab.
3. After you have selected the node, select Power off from the Actions menu. Click Go.
A node needs to be completely powered off before powering off the next node.
For all the Fibre Channel adapters that are on the nodes, use the appropriate remove procedure that corresponds with the type of Fibre Channel adapters that you are migrating from.
Ensure that hosts can now connect to the nodes.

Using the command-line interface

On all Fibre Channel connected nodes, upgrade the software to the 8.2.1 release.
On a node that is not the current configuration node of the system, enter the following command:
```
satask stopnode -poweroff
```
After the node completes powering off, you need to install the 25 Gbps Ethernet adapter on the node. Use installation instructions for the 25 Gbps Ethernet adapter to complete this step.
To define the IP address for RDMA-capable Ethernet ports in the command-line, complete these steps:
1. To define port IP addresses for RDMA-capable Ethernet ports, enter the following command on each port for each node within the system:
```
satask chnodeip  -ip ip_address -mask mask -gw gateway -port_id port_number -vlan vlanid panel_name
```
  where ip_address is the IP address for the port that is identified by its port identifier (port_number) and node name (panel_name). Ensure that all ports are in the same VLAN by specifying the same vlanid for each IP address on the RDMA-capable Ethernet port.
  Note:
  - The IP address and subnet mask for each port must be unique and cannot be used anywhere else on the system. However, the VLAN ID for RDMA-capable Ethernet port, on all nodes must be the same, or connections between the nodes that use these IP addresses will fail.
  - The 100-Gbps Ethernet port does not support node-to-node communication.
2. To add the node to the system, complete these steps:
  1. Ensure that the node is listed as a candidate node by entering the following command:
```
svcinfo lsnodecandidate
```
    In the results that display, verify that the id parameter displays the WWNN for the node. If the node is not detected, verify cabling to the node.
  2. Enter the following command to determine the I/O group where the node must be added:
```
lsiogrp
```
  3. Record the name or ID of the first I/O group that has a node count of zero. You need the name or ID for the next step.
    Note: You must do this step for the first node that is added. You do not do this step for the second node of the pair because it uses the same I/O group number.
  4. Enter this command to add the node to the system:
```
addnode -wwnodename WWNN -iogrp iogrp_name -name new_name_arg
```
    Where WWNN is the WWNN of the node, iogrp_name is the name of the I/O group that you want to add the node to, and new_name_arg is the name that you want to assign to the node. If you do not specify a new node name, a default name is assigned.
3. Verify connectivity between the nodes that use RDMA-capable Ethernet ports, by entering this command:
```
sainfo lsnodeipconnectivity
```
  In the results that display, a value of Status:Connected indicates a successful connection. A value of Status:Discovered indicates that port on the selected node is configured, but a connection cannot be established. If connection errors occur, possible reasons for these errors are displayed in the error_data parameter. For descriptions of these error states, see the management GUI instructions. You can also use the ping command to troubleshoot connectivity issues between the nodes.
Repeat steps 2 to 4 on each node until all the nodes are configured with RDMA-capable Ethernet ports.
Power off each node that remains in the system one at a time by entering the following command:
```
satask stopnode -poweroff
```
Each node needs to be completely powered off before issuing his command on another node.