essgennetworks command

Create a bonded ESS network Start of change Perform ESS bond creation and convert interface conversion for VPI adapters. End of change

Synopsis


essgennetworks { -N Node-List } [ --prefix Prefix ] [ --suffix Suffix ] 
               [ --interfaces Interfaces ] [--assignip ASSIGNIP] [--MRoT MROT]
               [ --create-bond | --delete-bond |--add-slave ] 
               [ --gateway Gateway ] [--bond Bondname ] [--miimon Miimon ]
               [ --vlan Vlan ]
               [ --mode { balance-rr,active-backup,balance-xor,broadcast,
                 802.3ad,balance-tlb,balance-alb } ] 
               [ --hash-policy {layer2+3,layer3+4 } ] [ --netmask Crid ]
               [ --iPoIB ] [--create-bridge-for-vm]
               [ --bridge-type {mgmt}] [ --query ] [--enableRDMA]
               [ --disableRDMA] [--enableRoCE] [--disableRoCE]
               [ --configureRouteForRoCE]
               [ --roceRoutingTableId ROCEROUTINGTABLEID]
               [ --roceRoutingTableName ROCEROUTINGTABLENAME]
               [ --verbsPortsFabricBond VERBSPORTSFABRICBOND]
               [ --verbsPortsFabricIface VERBSPORTSFABRICIFACE]
               [ --devices Devices ]
               [ --change { InfiniBand,Ethernet } ] [ --port { P1,P2 }] 
               [--mtu { 1500,2044,4092,9000 } ] [ --verbose ] [ -h | --help ]

Availability

Available with IBM Storage Scale System.

Description

Use essgennetworks to create a high-speed network by using nmcli. This command is applicable to any network with Ethernet and InfiniBand interfaces. This command also analyzes the /etc/hosts file in the management node and (based on the input that is provided in that file) creates a bonded connection on the high-speed interfaces that are in the target node. If new slave interfaces are added (or up) after the bond is created, these interfaces can be added to the bond.

By default, this command configures link aggregation control protocol (LACP) as follows:

Miimon is set to 100
Bonding mode is set to 802.3ad (LACP)
Xmit_hash_policy is set to layer2+3

Any other bond options contain the default value, which includes the lacp_rate (the default value is slow). For proper network operation, the Ethernet switch and InfiniBand switch settings in the networking infrastructure must match the I/O server node interface bond settings.

This command also support VPI adapter interface switching between InfiniBand and Ethernet. You can use --change option and select target interface type either InfiniBand or Ethernet and select the target port, which needs conversion to different type of protocol. There is a switch named --device, which allows user to select a particular device onto which conversion of the interface must be performed. Host must be rebooted after the conversion is over to get the new type of interface type to get reflected. Make sure that the MTS (Mellanox Tools Services) is running on node before you convert any type of interface. Conversion of interface type is only supported for ESS 3000 type of ESS servers.

Parameters

-N Node-List: Specifies a list of nodes for network operations .
--prefix Prefix: Specifies the hostname prefix.
Use an equal sign (=) between --prefix and value if the value starts with a hyphen (-).
--suffix Suffix: Specifies the hostname suffix.
Use an equal sign (=) between --suffix and value if the value starts with a hyphen (-).
--interface Interface: Specifies a list of interfaces for bonding. If the list is not provided, by default all high-speed interfaces are taken.
--assignip ASSIGNIP: Assigns IP address to provide interface in the --interface switch.
--MRoT MROT: Enables multi rail over TCP.
--create-bond: Creates a bonded interface.
--delete-bond: Deletes a bonded interface.
--add-slave: Adds the slave interfaces to an existing bond. This option is useful when more high-speed links are up or added since the bond creation.
--gateway Gateway: Specifies a gateway for the network. By default it will not configure any gateway on network interface until specified.
--bond Bondname: Specifies the name of the bond. The default is bond0.
--miimon Miimon: Provides a miimon value for bond. The default is miimon=100. Consider miimon=1000 if you are planning to use bond for RoCE.
--vlan Vlan: Sets VLAN_ID for the interface or bond.
--mode {balance-rr, active-backup, balance-xor, broadcast, 802.3ad,balance-tlb, balance-alb}: Specifies mode for the bonded interface. The default is 802.3ab (recommended).
--hash-policy {layer2+3,layer3+4}: Specifies the xmit hash policy for 802.3ad and balanced-xor. The default value is layer2+3.
--netmask Crid: Specifies the CIDR or network mask for the interface. The default is /24. The network mask should be specified in the format of CIDR.
--verbose: Provides more verbose output. The default is false.
--ipoib: Enables IPoIB (IP over InfiniBand, in case InfiniBand network present). By default, False. It will also enables RDMA for InfiniBand.
--create-bridge-for-vm: For VM communication, create a management or high-speed bridge. Depending on the type of bridge, management or High Speed IP is moved to the bridge.
--bridge-type {mgmt}: For VM communication, create a management or high-speed bridge. Depending on the type of bridge, management or High Speed IP is moved to the bridge.
--query: Queries the port type of the Mellanox interface.
--enableRDMA: Enables RDMA over the InfiniBand Network.
--disableRDMA: Disables RDMA over the InfiniBand Network.
--enableRoCE: Enables RoCE over the Ethernet Network.
--disableRoCE: Disables RoCE over the Ethernet Network.
--configureRouteForRoCE: Configures routing for RoCE over the Ethernet Network. It is used if same subnet has been used for different RoCE interfaces.
--roceRoutingTableId ROCEROUTINGTABLEID: Provides routing table ID for the RoCE over the Ethernet Network.
--roceRoutingTableName ROCEROUTINGTABLENAME: Specifies routing table Name for the RoCE over the Ethernet Network.
--verbsPortsFabricBond VERBSPORTSFABRICBOND: Specifies the name of the Mellanox verbs port fabric for Bonded interface. For Example: /1 or /2. It is automatically added to the verbs port. Multiple fabric can be applied by using comma separated value.
--verbsPortsFabricIface VERBSPORTSFABRICIFACE: Specifies the name of the Mellanox verbs port fabric non Bonded interface such as regular interfaces. For Example: /1 or /2. It is automatically added to the verbs port. Multiple fabric can be applied by using comma-separated value.
--devices DEVICES: Specifies the name of the Mellanox device name. All devices attached to node will be accessed with the query all. Provides comma-separated device names to query more than one device at a given time.
--change {InfiniBand,Ethernet}: Changes the Mellanox port type to InfiniBand or Ethernet and vice versa.
--port {P1,P2}: Specifies the port number of the Mellanox VPI card.
--mtu {1500,2044,4092,9000}: Provides the MTU of the bond network. For Ethernet, 1500 or 9000 MTU is allowed (Default is 1500). For InfiniBand, 2044 or 4092 MTU is allowed (Default is 2044).
-h | --help: Displays usage information about this script and exits.

Exit status

0: Successful completion.
nonzero: A failure has occurred.

Security

You must have root authority to run the essgennetworks script.

Examples

This example shows running essgennetworks to obtain information of high-speed interfaces IP address that is assigned to the bond for Ethernet interfaces.

# essgennetworks -G ess_x86_64 --suffix=-10g --verbose
  2018-02-28T01:33:51.764194 Starting network generation
  2018-02-28T01:33:52.432630 nodelist:    gssio1 gssio2
  2018-02-28T01:33:52.432669 suffix used for network hostname: -10g
  Interface(s) available on node gssio1-10g
  Up interface:  enP2p1s0
  Up interface:  enP2p1s0d1
  Up interface:  ib0
  Down interface:  ib1
  Up interface(s) of node gssio1-10g considered for bonding are ['enP2p1s0', 'enP2p1s0d1']
  Interface(s) available on node gssio2-10g
  Up interface:  enP2p1s0
  Up interface:  enP2p1s0d1
  Up interface:  ib0
  Down interface:  ib1
  Up interface(s) of node gssio2-10g considered for bonding are ['enP2p1s0', 'enP2p1s0d1']
  gssio1-10g: Current IP Address: IP not assigned
  gssio2-10g: Current IP Address: IP not assigned

This example shows creating a bond in I/O server nodes by using information that is provided in /etc/hosts file for Ethernet interfaces where bond name is bond0.

# essgennetworks -G ess_x86_64 --suffix=-10g --bond bond0 --create-bond
  2018-02-28T01:41:11.413341 Starting network generation
  2018-02-28T01:41:12.072727 nodelist:    gssio1 gssio2
  2018-02-28T01:41:12.072765 suffix used for network hostname: -10g
  Interface(s) available on node gssio1-10g
  Up interface:  enP2p1s0
  Up interface:  enP2p1s0d1
  Up interface:  ib0
  Down interface:  ib1
  Up interface(s) of node gssio1-10g considered for bonding are ['enP2p1s0', 'enP2p1s0d1']
  Interface(s) available on node gssio2-10g
  Up interface:  enP2p1s0
  Up interface:  enP2p1s0d1
  Up interface:  ib0
  Down interface:  ib1
  Up interface(s) of node gssio2-10g considered for bonding are ['enP2p1s0', 'enP2p1s0d1']
  gssio1-10g: Current IP Address: IP not assigned
  gssio2-10g: Current IP Address: IP not assigned
  gssio1-10g: IP Address assigned to bond: 172.31.250.17
  gssio2-10g: IP Address assigned to bond: 172.31.250.18
  Bond creation complete

This example shows running essgennetworks to obtain information of high-speed interfaces IP address that is assigned to the bond for InfiniBand interfaces.

# essgennetworks -G ess_x86_64 --suffix=-ib --ipoib
  2018-02-28T01:46:52.785492 Starting network generation
  2018-02-28T01:46:53.452440 nodelist:    gssio1 gssio2
  2018-02-28T01:46:53.452475 suffix used for network hostname: -ib
  Interface(s) available on node gssio1-ib
  Up interface:  enP2p1s0
  Up interface:  enP2p1s0d1
  Up interface:  ib0
  Down interface:  ib1
  Up interface(s) of node gssio1-ib considered for bonding are ['ib0']
  Interface(s) available on node gssio2-ib
  Up interface:  enP2p1s0
  Up interface:  enP2p1s0d1
  Up interface:  ib0
  Down interface:  ib1
  Up interface(s) of node gssio2-ib considered for bonding are ['ib0']
  gssio1-ib: Current IP Address: IP not assigned
  gssio2-ib: Current IP Address: IP not assigned

This example shows creating a bond in I/O server nodes by using information that is provided in /etc/hosts file for InfiniBand interfaces where bond name is bond1.

# essgennetworks -G ess_x86_64 --suffix=-ib --ipoib --create-bond
  2018-02-28T01:48:31.769301 Starting network generation
  2018-02-28T01:48:32.432582 nodelist:    gssio1 gssio2
  2018-02-28T01:48:32.432633 suffix used for network hostname: -ib
  Interface(s) available on node gssio1-ib
  Up interface:  enP2p1s0
  Up interface:  enP2p1s0d1
  Up interface:  ib0
  Down interface:  ib1
  Up interface(s) of node gssio1-ib considered for bonding are ['ib0']
  Interface(s) available on node gssio2-ib
  Up interface:  enP2p1s0
  Up interface:  enP2p1s0d1
  Up interface:  ib0
  Down interface:  ib1
  Up interface(s) of node gssio2-ib considered for bonding are ['ib0']
  gssio1-ib: Current IP Address: IP not assigned
  gssio2-ib: Current IP Address: IP not assigned
  gssio1-ib: IP Address assigned to bond: 172.31.250.1
  [WARN] gssio1-ib: Bond created with one slave interface
  gssio2-ib: IP Address assigned to bond: 172.31.250.2
  [WARN] gssio2-ib: Bond created with one slave interface
  Bond creation complete

This example shows how to convert the VPI adapter port type to and from Ethernet to InfiniBand and vice versa. Make sure that the MTS (Mellanox Tools Services) is running on a node before you convert any type of interface.

Querying device

essgennetworks -N localhost --query --devices all
2020-09-23T04:24:03.397420 [INFO] Starting network generation
2020-09-23T04:24:03.579361 [INFO] nodelist:  localhost
[ERROR] Mellanox Software Tools services are not running. Make sure Mellanox Software Tools running configuring VPI adapters.
Make sure you must start Mellanox Software Tools using "/bin/mst start" command before starting the configuration of the VPI adapters.

Start MST Services by using the following command:

/bin/mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
[warn] mst_pciconf is already loaded, skipping
Create devices
Unloading MST PCI module (unused) - Success

Querying device again.

essgennetworks -N localhost --query --devices all
2020-09-23T04:24:18.083995 [INFO] Starting network generation
2020-09-23T04:24:18.268935 [INFO] nodelist:  localhost
[INFO} Device          /dev/mst/mt4121_pciconf1 link type currently configured at system.
[INFO] Port 1 is set to **InfiniBand**
[INFO] Port 2 is set to InfiniBand
[INFO} Device          /dev/mst/mt4121_pciconf0 link type currently configured at system.
[INFO] Port 1 is set to InfiniBand
[INFO] Port 2 is set to InfiniBand

Change the port type of device /dev/mst/mt4121_pciconf1 for Port 1 from InfiniBand to Ethernet.

essgennetworks -N localhost --change Ethernet --devices /dev/mst/mt4121_pciconf1 --port P1
2020-09-23T03:45:52.322096 [INFO] Starting network generation
2020-09-23T03:45:52.510535 [INFO] nodelist:  localhost
[INFO} Changing /dev/mst/mt4121_pciconf1 Port P1 link type to Ethernet
[INFO] Successfully changes the Port type to Ethernet for Port P1

Reboot the node and make sure that the MST tool services started again.

essgennetworks -N localhost --query --devices all
2020-09-23T04:05:55.774019 [INFO] Starting network generation
2020-09-23T04:05:55.960088 [INFO] nodelist:  localhost
[INFO} Device          /dev/mst/mt4121_pciconf1 link type currently configured at system.
[INFO] Port 1 is set to **Ethernet**
[INFO] Port 2 is set to InfiniBand
[INFO} Device          /dev/mst/mt4121_pciconf0 link type currently configured at system.
[INFO] Port 1 is set to InfiniBand
[INFO] Port 2 is set to InfiniBand

Check whether the Port 1 is changed to Ethernet.

mlxconfig -d  /dev/mst/mt4121_pciconf1 query | grep -i link_type
         LINK_TYPE_P1                        **ETH(2)**
         LINK_TYPE_P2                        IB(1)

Location

/opt/ibm/ess/tools/bin