Migration to PCI network adapters

Migrate from QDIO-based network interfaces to interfaces that are provided by PCI functions.

Attention: Migrating a network setup can be risky, especially if you are migrating the setup for the only connection to a Linux® instance. You might not be able to use an inadequately migrated interface to connect to your Linux instance for corrective actions.

To mitigate against potential lockouts, set up and test an alternative connection to your Linux instance before you begin the migration. For example, use HiperSockets and a separate IP subnet for an alternative connection without any physical components. Once set up, verify that your alternative connection persists across reboots before you proceed to migrate your network setup.

You can resort to the Linux console to restore regular connectivity.
Linux as a KVM guest
Access the virtual console from the host or use virt-manager.
Linux as a z/VM® guest
Access the console through a 3270 terminal emulation.
Linux in LPAR or DPM partition mode
Access the console through the Operating System Messages applet or through the Integrated ASCII Console applet on the HMC.

Before you begin

Your migration options depend on your current environment, hardware and how your OSA-based network interfaces are accessed by your Linux instance.

Table 1. Migration options for RoCE Express and Network Express
Usage Migration option
Directly attached OSA-Express adapters Migration is possible for Linux in LPAR or DPM partition mode or for Linux as a z/VM guest. Except for z/VM guests that are part of a single system image (SSI) cluster, as SSI currently does not support PCI pass-through.
OSA-Express-based interfaces connected through a z/VM VSWITCH For Linux guests, z/VM VSWITCH uses QDIO-based interfaces with z/VM VSWITCH setups.

RoCE Express

z/VM VSWITCH does not support RoCE Express.

Network Express

z/VM VSWITCH can be connected to OSH of Network Express in hybrid mode.
OSA-Express-based interfaces connected to Linux as a KVM guest through Open vSwitch For KVM, MacVTap can be used with any PCI network adapter.

RoCE Express

Open vSwitch requires promiscuous mode and therefor can not be used with RoCE Express.

Network Express

Open vSwitch requires promiscuous mode.

Goal

Seamlessly migrate the network traffic of a Linux instance from OSA-Express to PCI network adapters.

Assumptions

NetworkManager was used to set up bonded interface, mybond0, on an instance of Red Hat® Enterprise Linux. The bonded interface provides path redundancy through two OSA-Express-based network interfaces, encbd00 and encbd10.

Two PCI network adapters are in place and PCI functions that are suitable to provide path redundancy are configured and visible in Linux. The corresponding network interfaces are eno5968 and eno6032.

Strategy

First, add the PCI based interfaces to the bonded interface. Then, remove the OSA-based interfaces.

Sample procedure

  1. List your bonded interfaces.
    # nmcli c  
    NAME                  UUID                                  TYPE      DEVICE   
    bond-mybond0          1f543303-4594-4230-9d2f-45a5eb424305  bond      mybond0  
    bond-slave-encbd00    30127166-31d8-4004-a55f-981df4d09573  ethernet  encbd00
    bond-slave-encbd10    6ae85d1b-fb20-4769-85b6-655fa1efb118  ethernet  encbd10
  2. Add the PCI based interfaces to the bonded interface.
    # nmcli con add type ethernet ifname eno5968 master mybond0        
    Connection 'bond-slave-eno5968' (b0bab623-b7b7-44ca-b01c-80fe8a97b708) successfully added. 
    # nmcli con add type ethernet ifname eno6032 master mybond0        
    Connection 'bond-slave-eno6032' (ad2fec5a-2c8b-42f6-8e16-37902d54b536) successfully added.
  3. Confirm that four interfaces are now part of the bonded interface.
    # nmcli c  
    NAME                  UUID                                  TYPE      DEVICE   
    bond-mybond0          1f543303-4594-4230-9d2f-45a5eb424305  bond      mybond0  
    bond-slave-encbd00    30127166-31d8-4004-a55f-981df4d09573  ethernet  encbd00
    bond-slave-encbd10    6ae85d1b-fb20-4769-85b6-655fa1efb118  ethernet  encbd10
    bond-slave-eno5968    b0bab623-b7b7-44ca-b01c-80fe8a97b708  ethernet  eno5968
    bond-slave-eno6032    7533fcce-6da0-4239-9297-26b5e7a79e5c  ethernet  eno6032
  4. Remove the OSA-Express-based interfaces from the bonded interface.
    # nmcli con delete bond-slave-encbd00
    Connection 'bond-slave-encbd00' (30127166-31d8-4004-a55f-981df4d09573) successfully deleted.
    # nmcli con delete bond-slave-encbd10
    Connection 'bond-slave-encbd10' (6ae85d1b-fb20-4769-85b6-655fa1efb118) successfully deleted.
    
  5. Confirm that only the PCI based interfaces remain as part of the bonded interface.
    # nmcli c  
    NAME                  UUID                                  TYPE      DEVICE   
    bond-mybond0          1f543303-4594-4230-9d2f-45a5eb424305  bond      mybond0  
    bond-slave-eno5968    b0bab623-b7b7-44ca-b01c-80fe8a97b708  ethernet  eno5968
    bond-slave-eno6032    7533fcce-6da0-4239-9297-26b5e7a79e5c  ethernet  eno6032

Result

Network traffic through mybond0 now uses the network adapter PCI functions.