Setting up the IP interfaces on the switch on a RoCE network (Linux®)

On a remote direct memory access (RDMA) over Converged Ethernet (RoCE) network, you must manually set up the IP interfaces on the switch. The procedures documented in this page are no longer required as adapter port liveliness test has been enhanced and automated. Some restrictions apply. Refer to technote#0733765 for restrictions.

To set up the IP interfaces:

1. Log on to the command-line interface of the switch with the admin user ID and password.
2. Create the interface and setup its IP address using these guidelines:
   1. Create the same number of IP interfaces on a switch as the number of distinct IP subnets connected from a CF or member host. (For example, in the figure below, each switch has two distinct IP subnets connected from the CF host.)
   2. Each IP interface must be assigned an IP address that is on one of the distinct IP subnets from the CF host.
   3. No IP subnet can be used more than once on a switch.
3. For a single switch cluster, the setup is now completed. If the cluster in Figure 1 only has one switch, after completing steps 1 to 2, the switch will have the following IP addresses assigned to their IP interfaces :

    192.168.1.2, 192.168.2.2, 192.168.3.2 and 192.168.4.2.

   The final step is to verify that the newly created IP interfaces are pingable from each other and each adapter in members and CFs can ping the external pingable IP address(es) specified for that adapter in their netmon.cf.
4. For a two-switch cluster where at least 2 adapter ports are used in each CF and member, repeat steps 1 and 2 on the other switch. At the end of this step, there should be an equal number of IP interfaces on each switch with each IP address on a different IP subnet. The total number of distinct IP subnets on both switches is the same as the maximum of the total number of distinct IP subnets on a CF or member host.

   Using Figure 1 as an example, after completing steps 1 to 4, each switch will have the following IP addresses assigned to their IP interfaces:

   Switch 1 - 192.168.1.2 and 192.168.2.2
   Switch 2 - 192.168.3.2 and 192.168.4.2

5. For a two-switch cluster where at least 2 adapter ports are used in the CF but only a single adapter port is used in each member, create an additional IP interface on the switch where its newly created IP interfaces do not have the same IP subnet as the member's adapter. Assign this new IP interface an IP address that is on the same IP subnet as the member. After this step, one switch should have one extra IP interface than the other. In addition, both switches will have exactly one IP interface where its IP address is on the same IP subnet as all the members.
   Note: The final step is to verify the newly created IP interfaces are pingable from each other and each adapter in members and CFs can ping the external pingable IP addresses specified for that adapter in their netmon.cf.

Example

Below is the step-by-step instructions on how to create the IP interfaces on a cluster using two BNT switches :

1. Log in with the admin user ID.
2. Create the interface and setup its IP address and IP subnet mask.
3. If you are creating two interfaces on the switch:
   1. To create interface 1, setup the IP address and IP subnet mask. For example:

      RS G8124(config)#interface ip 1
      RS G8124(config-ip-if)#ip address 192.168.1.2
      RS G8124(config-ip-if)#ip netmask 255.255.255.0
      RS G8124(config-ip-if)#exit

   2. To create interface 2 using the same procedure. For example:

      RS G8124(config)#interface ip 2
      RS G8124(config-ip-if)#ip address 192.168.2.2
      RS G8124(config-ip-if)#ip netmask 255.255.255.0
      RS G8124(config-ip-if)#exit

4. For the change to take effect, enter the following command:

   RS G8124(config)#copy running-config startup-config 

5. To create interfaces for different IP subnets, repeat steps 2 to 4 on the second switch.
6. To validate the IP addresses, ping the address from the CF and member hosts that connect to the same switch.