Troubleshooting NVMe-oF gateways

Understand how to troubleshoot and learn how to fix the most common errors that are related to Ceph NVMe-oF gateway configuration and operational conditions.

Verifying gateway connectivity

If Ceph NVMe-oF gateway is not responding as expected, verify gateway connectivity.

Procedure

List the NVMe-oF block devices.

nvme list

For example,

[root@host01 ~]# nvme list
Node                    Generic           SN                   Model                   Namespace Usage                      Format           FW Rev
---------------------   ----------------  -------------------  ----------------------- --------- -------------------------- ---------------- --------
/home/nvme01_node01     /home/ng1n1       SPDK00000000000001   SPDK bdev Controller    1          10,49  MB /  10,49  MB      4 KiB +  0 B   23.01
...

Verify that the initiator is connected to all NVMe-oF gateways and subsystems in the gateway group.

nvme list-subsys

For example,

[root@init-nvme-vm5 ~]# nvme list-subsys
nvme-subsys5 - NQN=nqn.2016-06.io.spdk:cnode2
\
+- nvme5 tcp traddr=10.243.64.5,trsvcid=4420 live
+- nvme6 tcp traddr=10.243.64.10,trsvcid=4420 live
+- nvme7 tcp traddr=10.243.64.11,trsvcid=4420 live
+- nvme8 tcp traddr=10.243.64.12,trsvcid=4420 live
nvme-subsys1 - NQN=nqn.2016-06.io.spdk:cnode1.group1 
\
+- nvme1 tcp traddr=10.243.64.5,trsvcid=4420 live

Create a filesystem on the target of your choosing.

Use the target path that was found in step 1.

mkfs NVME_NODE_PATH

For example,

[root@host01 ~]# mkfs /home/nvme01_node01
mke2fs 1.46.5 (20-Dec-2023)
Discarding device blocks: done
Creating filesystem with 2560 4k blocks and 2560 inodes

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

Mount the NVMe node on the NVMe-oF directory.

Mount NVMe-oF.

mkdir /mnt/nvmeof

For example,

[root@host01 ~]# mkdir /mnt/nvmeof

Mount the node on within the NVMe-oF directory.

mount NVME_NODE_PATH /mnt/nvmeof

For example,

[root@host01 ~]# mount /home/nvme01_node01 /mnt/nvmeof

Using sudo commands, list mounted NVMe-oF files.
```
ls /mnt/nvmeof
```
For example,
```
$ ls /mnt/nvmeof
lost+found
```
Create a text file within the mnt/nvmeof directory.
For example,
```
$ sudo bash -c "echo Hello NVMe-oF > /mnt/nvmeof/hello.txt"
```
Verify that the text file can now be reached.
For example,
```
$ cat /mnt/nvmeof/hello.txt
Hello NVMe-oF
```

Changes to services on specification file are not properly implemented

About this task

When updating the NVMe-oF service specification file changes are stored correctly but they do not immediately reflect inside the running container’s configuration (nvme_conf). The updated spec is applied at the orchestration level, but the NVMe-oF service itself does not restart to pick up the new values. As a result, manual redeployment is required to propagate these changes to the container.

Procedure

Ensure that the specification file is properly deployed, by using the ceph orch apply command.

Manually redeploy the NVMe-oF service.

ceph orch redeploy nvmeof.POOL_NAME.GROUP_NAME

Finding a host NQN

A host NVMe Qualified Name (NQN) is a unique identifier The NQN identifies a host system by using FC-NVMe connections, allowing the storage system to recognize and interact with the host.

Before you begin

Before you begin, verify that nvme-cli is installed and running.

Procedure

Fetch the NQN from the /etc/nvme/hostnqn file.

cat /etc/nvme/hostnqn

For example,

[root@host] cat /etc/nvme/hostnqn nqn.2014-08.org.nvmexpress:uuid:9ed5b327-b9fc-4cf5-97b3-1b5d986345d1