Managing NVMe devices

As of LinuxONE II, PCIe-attached NVMe devices are supported on IBM® LinuxONE.

Red Hat Enterprise Linux 9.2 LPAR mode z/VM guest

As of LinuxONE III, you can use NVMe devices as stand-alone dump devices for Linux® in LPAR mode or in a DPM partition, see Using the Dump Tools on Red Hat Enterprise Linux 9.2, SC34-7751.

The general information about PCIe and PCIe-attached devices applies to NVMe.

To Linux, NVMe devices are block devices that can be partitioned and hold file systems. NVMe disk controllers use name spaces to divide a physical NVMe device into multiple logical devices. Booting from an HMC might require name space ID 1 for NVMe boot devices.

Device names and nodes

NVMe device names follow this pattern: nvme<number>n<namespace>, where:
<number>
is an integer that is assigned by Linux during the boot process. The first NVMe device that is detected is assigned 0. Devices that follow are assigned consecutive numbers.
<namespace>
is an NVMe name space ID that is assigned by the NVMe disk controller.

For partitions, p<n> is appended to the device name of the whole device, where <n> denotes the <n>-th partition.

Table 1. Standard device names and nodes
Standard device names Standard device nodes Comment
Whole device:
nvme0n1
Partitions:
nvme0n1p1
nvme0n1p2
...		 Whole device:
/dev/nvme0n1
Partitions:
/dev/nvme0n1p1
/dev/nvme0n1p2
...		 First device with name space 1
Whole device:
nvme0n2
Partitions:
nvme0n2p1
nvme0n2p2
...		 Whole device:
/dev/nvme0n2
Partitions:
/dev/nvme0n2p1
/dev/nvme0n2p2
...		 First device with name space 2
Whole device:
nvme1n1
Partitions:
nvme1n1p1
nvme1n1p2
...		 Whole device:
/dev/nvme1n1
Partitions:
/dev/nvme1n1p1
/dev/nvme1n1p2
...		 Second device with name space 1
The mapping between physical storage space and standard device names does not persist across reboots. Depending on the udev rules of your distribution, udev creates other device nodes for you.
Example: node based on a WWN
/dev/disk/by-id/nvme-eui.01000000010000005cd2e4f0bc174f51

The WWN is a unique, fixed hardware property. This type of device node maps to the same NVMe device, across reboots.

Example: node based on manufacturer specifications
/dev/disk/by-id/nvme-INTEL_SSDPE2KX040T7_PHLF806200284P0IGN

The manufacturer specification is a unique, fixed hardware property that includes the hardware model and serial number. This type of device node maps to the same NVMe device, across reboots.

The device nodes that udev creates for partitions depend on the udev rules. Commonly, the nodes names match the names of the whole device, with -part1 appended for the first partition, -part2 for the second partition, and -part<x> for the <x>th partition.

Function addresses

If your LinuxONE hardware is configured to support UIDs, NVMe function addresses follow the pattern <UID>:00:00.0 and map to the same physical PCI slot of the NVMe device across reboots.

Without UID support, the pattern is <hhhh>:00:00.0, where the variable part, <hhhh>, is a 4-digit hexadecimal number. Linux sets this number to 0000 for the first PCIe device that it discovers and increments it by 1 for subsequent devices. So, according to this pattern, the function addresses for the first 3 PCIe devices are: 0000:00:00.0, 0001:00:00.0, and 0002:00:00.0. This naming scheme does not persist across reboots. Because function addresses include all PCIe devices, this means that addresses that mapped to a specific NVMe device might not only map to a different NVMe device, but to a different type of PCIe device altogether.

To find the function address for a standard device node use the ls command to display details for the device's representation as a block device in sysfs.
# ls -l /sys/block/nvme0n1
lrwxrwxrwx. 1 root root 0 Oct 23 16:46 /sys/block/nvme0n1 -> ../devices/pci0850:00/0850:00:00.0/nvme/nvme0/nvme0n1
In the example, nvme0n1 maps to an NVMe device with a PCIe function address 0850:00:00.0.
Tip: Issue ls -l /sys/block/nvme* for a complete mapping of function addresses and standard device nodes.

NVMe devices in sysfs

PCIe-attached NVMe devices have all generic PCIe sysfs attributes at /sys/bus/pci/devices/<function_address>, see PCI Express support.

You can find NVMe-specific attributes in the device representations at /sys/bus/pci/drivers/nvme/<function_address>.

NVMe devices with name space ID 1 can be prepared as boot devices. Consequently, they might be represented in /sys/firmware: