Setting up PCI devices for VFIO pass-through

Red Hat Enterprise Linux 9.2 LPAR mode KVM guest

To set up a PCI device as a VFIO pass-through device you must enable the vfio_pci device driver to handle the PCI device type, and you must assign the specific device to vfio_pci.

Before you begin: The preferred method for setting up PCI devices is to configure them for automatic management with libvirt, see the information about configuring VFIO pass-through devices in KVM Virtual Server Management, SC34-2752. This management includes a dynamic host preparation. The tasks that follow describe a fallback method that applies only to PCI devices that are not managed by libvirt.

Enabling the vfio_pci device driver

For devices that are not managed by libvirt, enable the vfio_pci device driver for specific PCI card types.

Tip: You must know the device's vendor code and device code. Issue lspci -n to display this information for your PCIe devices in the format <vendor_code>:<device_code>.

Use the ids= module parameter to specify the PCIe card types.

vfio_pci module parameter syntax

Example: In this example, a PCIe device with function address 0001:000:000:0, vendor code 15b3, and device code 1003 is available on the host. The specification for the ids= module parameter makes this card type eligible for the vfio_pci device driver.

# lspci -n
0001:000:000:0 15b3:1003
# modprobe vfio_pci ids=15b3:1003

On a running host, you can use the /sys/bus/pci/drivers/vfio-pci/new_id sysfs attribute to enable the vfio_pci device driver to control a particular PCIe card type. Write the vendor code and device code, separated by a blank, to the attribute.

Example: This example, makes cards with vendor code 15b3 and device code 1003 eligible for the vfio_pci device driver.

# echo 15b3 1003 > /sys/bus/pci/drivers/vfio-pci/new_id

After setting up the vfio_pci device driver for one or more PCIe card types, all cards of these types that are freed from their default device driver are assigned to the vfio_pci device driver.

Assigning a PCI device to the vfio_pci device driver

For devices that are not managed by libvirt, write the function address of the PCIe device to the unbind attribute of its device driver.

# echo <function_address> > /sys/bus/pci/drivers/<pci_device_driver>/unbind

Tip: Issue lspci -v to find out which device driver controls the device of interest.

Example:

~]# lspci -v
0001:00:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Subsystem: Mellanox Technologies Device 048d
Physical Slot: 00000015
...
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core
# echo 0001:00:00.0 > /sys/bus/pci/drivers/mlx4_core/unbind