KVM RoCE Express Multiple Flat networks

To support RoCE Express on multiple flat networks in KVM on the compute node, you need to manually configure the hybrid management nodes and each of the compute nodes with MacVTap host_network_capabilities. The configuration guide works for both stand-alone and multi-node clusters.

1. Update /etc/neutron/plugins/ml2/ml2_conf.ini file

   Make sure flat_networks = *

   [ml2_type_flat]
   flat_networks = *
   

   If the user wants to configure multiple flat networks in stand-alone, run the following command on the management node.

   $ sudo openstack-config --set --inplace  /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_flat flat_networks '*'
   

   If the user wants to configure multiple flat networks in multi-node cluster, run the following command on one of the management nodes.

   $ sudo icic-opsmgr config openstackconfig -c <cluster-name> -op set -file /etc/neutron/plugins/ml2/ml2_conf.ini -section ml2_type_flat -property flat_networks -value '*'
   

   for example

   $ sudo icic-opsmgr config openstackconfig -c c12 -op set -file /etc/neutron/plugins/ml2/ml2_conf.ini -section ml2_type_flat -property flat_networks -value '*'
   

2. Restart the neutron services

   If the user wants to configure multiple flat networks in stand-alone, run it on the management node. If the user wants to configure multiple flat networks in multi-node clusters, run it on one of the management nodes.

   $ sudo icic-services neutron restart
   

1. Look for RoCE Express interfaces

   1.1. Run the following command to get RoCE Express network interfaces on the KVM host.

   $ sudo smc_rnics
        FID  Power  PCI_ID        PCHID  Type           PPrt  PNET_ID            Net-Dev
   ------------------------------------------------------------------------------------------
        001  1      0000:00:00.0  0010   RoCE_Express2  0     n/a                eno1
        002  1      0002:00:00.0  0020   RoCE_Express2  0     n/a                eno2
   

   Note:
   • If there is no a command like this, run sudo dnf install smc-tools . In this example, there are 2 RoCE Express network interfaces.

   1.2. Check the RoCE Express network interface or the RoCE Express bonded interface status. To see the candidates, you need to check each RoCE Express network interface's status by running:

   $ sudo ip a
   

   Note:
   • Ensure the RoCE Express interface is in UP status.

2. Update the /etc/neutron/plugins/ml2/macvtap_agent.ini file on the KVM compute node. Choose one RoCE Express network interface or RoCE Express bonded interface candidate, and make sure that it is set in the physical_interface_mappings to use the new provider-physical-network name, instead of 'macvtap0', because macvtap0 is the default provider-physical-network name for MacVTap.

   [macvtap]
   physical_interface_mappings = macvtap0:<RoCE interface1>,<new provider-physical-network>:<RoCE interface2>
   

   For example, eno1 was used for macvtap0 already, eno2 is a RoCE Express network interface candidate to be used for new provider-physical-network named macvtap1:

   [macvtap]
   physical_interface_mappings = macvtap0:eno1,macvtap1:eno2
   

3. Restart the MacVTap agent

   $ sudo systemctl restart neutron-macvtap-agent
   

4. Create new networks

   It is only supported to create new networks through CLI or API, not the UI. Creating flat network requests sent from the UI are using --provider-physical-network as macvtap0.

   You only need to run them on one management node in multi-node cluster.

   Note:
   • The parameter --provider-physical-network must use the parameter you set in the last step2 'Update /etc/neutron/plugins/ml2/macvtap_agent.ini file', for example:

   [macvtap]
   physical_interface_mappings = macvtap0:eno1,macvtap1:eno2
   

   Here is an example how to create a network:

   $ openstack network create --provider-network-type flat --provider-physical-network macvtap1 --tag macvtap 2nd_roce_flat
   
   $ openstack subnet create --allocation-pool start=172.26.57.131,end=172.26.57.150 --network 2nd_roce_flat --no-dhcp --subnet-range 172.26.0.0/17 --gateway 172.26.0.1 2nd_roce_flat_sub
   

5. Deploy the VM on the UI by selecting the new networks

In this example: One new RoCE Express adapter is added for each KVM compute node to be the second flat network. The new provider-physical-network value is macvtap1. The new RoCE Express network interface is eno2.

There is a macvtap0 provider-physical-network, already, and eno1 is the RoCE Express network interface for this new provider macvtap1.

1. Flat network setting

   $ sudo icic-opsmgr config openstackconfig -c c12 -op set -file /etc/neutron/plugins/ml2/ml2_conf.ini -section ml2_type_flat -property flat_networks -value '*'
   

   Note: c12 is the cluster name

2. Restart

   $ sudo icic-services neutron restart
   

1. Look for RoCE Express network interfaces

   1.1. Run the following command to get RoCE Express network interfaces on the KVM host.

   $ sudo smc_rnics
        FID  Power  PCI_ID        PCHID  Type           PPrt  PNET_ID            Net-Dev
   ------------------------------------------------------------------------------------------
        001  1      0000:00:00.0  0010   RoCE_Express2  0     n/a                eno1
        002  1      0002:00:00.0  0020   RoCE_Express2  0     n/a                eno2
   

   1.2. Check the RoCE Express network interface status.

   $ sudo ip a
   

   Note:
   • Ensure the RoCE Express interface is in UP status and not used before.

2. Update /etc/neutron/plugins/ml2/macvtap_agent.ini file on the KVM compute node.

   [macvtap]
   physical_interface_mappings = macvtap0:eno1,macvtap1:eno2
   

3. Restart the MacVTap agent

   $ sudo systemctl restart neutron-macvtap-agent
   

4. Create new network

   It is only supported to create new networks through CLI or API, not the UI. To create flat network requests sent from the UI are using --provider-physical-network as macvtap0.

   You only need to run them on one management node in multi-node cluster.

   openstack network create --provider-network-type flat --provider-physical-network macvtap1 --tag macvtap 2nd_roce_flat
   
   openstack subnet create --allocation-pool start=172.26.57.131,end=172.26.57.150 --network 2nd_roce_flat --no-dhcp --subnet-range 172.26.0.0/17 --gateway 172.26.0.1 2nd_roce_flat_sub
   

5. Deploy the VM on the UI by selecting the new networks