Mapping NVMe disk slot location

IBM Storage Scale Erasure Code Edition requires additional configuration for use with NVMe drives.

IBM Storage Scale Erasure Code Edition brings enclosure-like management services to direct attached storage disks, allowing users to identify and replace disks without compromising system availability or integrity. IBM Storage Scale Erasure Code Edition ships with support for NVMe disks with a U.2 form factor. The U.2 form factor allows system administrators to replace NVMe disks as if they were regular HDD or SSD drives. Drive LED control is not supported currently, but replacement operations will work with their slot location. This means that NVMe drives might be replaced, but the replacement process would not trigger any identification or replace lights on the drive. For more information on disk replacement procedure, see Physical disk procedures.

Note: For supporting NVMe drives with LED control of showing the disk state, an RPQ is required for IBM® to review. It is highly hardware-related and only some types of servers are tested. Ask your sales representative to contact IBM Storage Scale development about the RPQ or SCORE process.

To support disk replacement for NVMe drives on IBM Storage Scale Erasure Code Edition, users need to define a pseudo enclosure describing a server’s disk layout and capabilities.

Creating an Enclosure Descriptor File (EDF)

U.2 NVMe drives reside in a pseudo enclosure within their server node. This pseudo enclosure is defined by using a plain-text EDF. The EDF describes the structure and layout of the storage components within the enclosure, as well as the capabilities of these components.

The EDF also contains a structure that is known as a “bay_map”, which describes a mapping from the server’s external drive slots to hardware address. A given server node’s slot to hardware address mapping might vary depending on its vendor and its internal cabling. Therefore, this mapping is crucial to ensure that disk replacement operations select the correct disk. It is recommended to use the same server node hardware across an IBM Storage Scale Erasure Code Edition recovery group, as this ensures a uniform NVMe drive mapping and allows a single EDF to be deployed on all nodes without additional configuration. Otherwise, a separate EDF must be created on each node.

Note:
  • NVMe drives might be organized into exclusive namespaces on a single controller or shared namespaces across multiple controllers. For use with IBM Storage Scale Erasure Code Edition, NVMe drives must be configured such that there is a single namespace on each controller.
  • NVMe drive slot mapping must be done before the recovery group creation. IBM Storage Scale Erasure Code Edition supports doing the mapping and the remapping after the recovery group is created. The ecedrivemapping tool only reads data from NVMe drives. Do not write data on NVMe drives after the recovery group is created. The *.edf files are located in /usr/lpp/mmfs/data/gems directory.
  • To define NVMe drive mapping, you must first select a server and populate all NVMe-capable slots with NVMe drives. After the mapping process is completed, the extra drives can be returned to the spare inventory or to other servers. This can be done once for each collection of servers with the same disk topology. IBM Storage Scale Erasure Code Edition does not support mapping additional NVMe server slots after this initial NVMe drive mapping is completed.

Mapping NVMe disks

The creation of a properly formatted and named EDF with a correct bay_map is produced by the ecedrivemapping command.
ecedrivemapping [-h] [--mode {nvme,lmr}]
                              [--slotrange {0-MAX_SLOT} {0-MAX_SLOT}] [--report]
                              [--force] [--version {1,2}]
For example, to create an EDF that describes NVMe-capable server slots 16-18, issue the following command.
# ecedrivemapping  --mode nvme --slotrange 16 18
Note:
  • If the --slotrange argument is not specified, the slot range is immediately requested upon execution of the ecedrivemapping command.
  • If the --mode argument is not specified, the slot range is applied to NVMe disks and lmr disks.
For each NVMe block device found in /proc/partitions, the tool flashes that block device’s activity light by using a read workload. This prompts the user to enter the corresponding slot for the flashing disk.
>>> 3 Nvme drives were detected
Now blinking path /dev/nvme0n1
>>> Enter the slot number: 18
Now blinking path /dev/nvme1n1
>>> Enter the slot number: 17
Now blinking path /dev/nvme2n1
>>> Enter the slot number: 16

In this example, slots 16-18 represent all NVMe-capable drive slots on the server. The tool fails if it detects that you are trying to map more slots than the actual number of NVMe-capable drive slots. The EDF is written to /usr/lpp/mmfs/data/gems/, and it must be copied to all nodes with the same NVMe drive topology.

Verifying the Enclosure Descriptor File

Note: Ensure that you do not use any command to corrupt the data on the disk if the recovery group is already created.
The ecedrivemapping command can be used to summarize the current enclosure descriptor file as follows:
# ecedrivemapping --mode nvme –report
To check the slot to bus mapping from the report above, do the following steps:
  1. Check the slot location with:
    # tslsenclslot -ad| mmyfields -s slot SlotHandle LocationCode Devices| grep gems | awk '{print "location: "$2" device: "$3}'
location: SERIALNUMBER-SLOTX device: /dev/nvmeAn1
location: SERIALNUMBER-SLOTY device: /dev/nvmeCn1
...

    This command must print the location codes of the correctly configured slots.

    The serial number SERIALNUMBER can be checked by the following command:

    # dmidecode -s system-serial-number

    The SLOTX is the physical disk location on the server defined by mapping tool.

  2. For each disk controller, issue a dd read command to its corresponding block device:
    # dd if=/dev/nvmeAn1 of=/dev/null bs=1M count=10000 skip=1000

    Because the drives must be formatted with a single namespace, controller nvmeA corresponds to block device /dev/nvmeAn1. From the above report, if you ran this command on /dev/nvmeAn1, you must expect to see the activity light in slot SLOTX light up.