How to correctly handle/attach EBS-backed volumes and NVMe on Linux instances

Steps

Problem

On AWS instances built on the Nitro System, EBS volumes are exposed as NVMe block devices with names like /dev/nvme0n1 and /dev/nvme1n1. The block device driver can assign these names in a different order than specified in the block device mapping, which can change after an instance restarts.

This behavior can cause data loss or application failure if provisioning scripts, such as those used with Terraform, rely on absolute device names for mounting volumes or other operations. To prevent this, you should identify devices by their unique EBS volume ID (e.g., vol-01234567890abcdef) instead of their assigned device name.

Procedure

The method for mapping a device name to an EBS volume ID varies by Linux distribution.

Option 1: Amazon Linux

With Amazon Linux AMI 2017.09.01 or later, including Amazon Linux 2, you can use the ebsnvme-id command to map the NVMe device name to a volume ID and its corresponding sdf-style device name.

1. Run the ebsnvme-id command, specifying the NVMe device path.

   $ sudo /sbin/ebsnvme-id /dev/nvme1n1
      Volume ID: vol-01324f611e2463981
      /dev/sdf

2. Use this stable device name or volume ID in your provisioning scripts. For example, you can assign the device name to a variable, format it, and find its UUID for later use.

   $ DEVICE=/dev/sdf
   $ sudo mkfs -t xfs -q $DEVICE
   $ PUUID=$(sudo blkid $DEVICE | grep -E -o "[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}")

Option 2: Debian and Ubuntu

For Debian-based distributions, you must install the nvme-cli tool to identify volumes. This procedure requires a Linux kernel version of 4.2 or later.

1. Install the nvme-cli package.

   $ sudo apt install -q -y nvme-cli

2. Use the nvme id-ctrl command to get the volume ID and device name. The device name is available through a vendor-specific extension.

   $ sudo nvme id-ctrl -v /dev/nvme1n1
      NVME Identify Controller:
      vid     : 0x1d0f
      ssvid   : 0x1d0f
      sn      : vol01234567890abcdef
      mn      : Amazon Elastic Block Store
      ...
      0000: 2f 64 65 76 2f 73 64 66 20 20 20 20 20 20 20 20 "/dev/sdf..."

   Note: The nvme-cli tool presents the EBS volume ID without dashes (e.g., vol01234567890abcdef). Your scripts must account for this difference. You can strip dashes from an EBS ID variable in Bash like this: VOLUME="${EBS_ID//-}"

3. You can also use the lsblk command to list available devices and their mount points to help determine the correct device name to use.

   $ sudo lsblk
      NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
      nvme1n1     259:3    0  100G  0 disk 
      nvme0n1     259:0    0    8G  0 disk 
      ├─nvme0n1p1 259:1    0    8G  0 part /
      └─nvme0n1p128 259:2  0    1M  0 part

Option 3: Red Hat and Derivatives

For Red Hat-based distributions like CentOS and Fedora, you must also install nvme-cli.

1. Install the nvme-cli package using the appropriate command for your distribution.

   • CentOS/Scientific Linux:

     $ sudo yum install nvme-cli

   • Red Hat Enterprise Linux:

     $ sudo snap install nvme-cli --candidate

   • Older systems or other derivatives: You may need to add the REMI repository from https://rpms.remirepo.net/.
2. After installation, follow the same steps outlined in Option 2 to identify and use the EBS volume ID.

Additional Information

• Amazon EBS and NVMe on Linux Instances provides Amazon's official recommendations.
• Device naming on Linux instances contains general information about device naming.
• The nvme-cli GitHub repository is the official homepage for the tool.