Managing subsystems

As a system administrator, you can add, list, and delete NVMe-oF subsystems by using ceph nvmeof commands.

About this task

Note: Creating subsystems with the same NQNs within more than one gateway group causes issues with the NVMe-oF initiators.

Adding a subsystem

Use the subsystem add command to add an NVMe-oF subsystem. 
ceph nvmeof subsystem add --nqn NQN [--enable-ha] [--max_namespaces MAX_NAMESPACES] [--no-group-append] [--serial_number VALUE] [--dhchap_key KEY_VALUE] [--network-mask SUBNET]
Important: This command fails if controller‑level DHCHAP keys are defined for any host in the subsystem.
The NQN is a user-defined string with and appended group name, assuming that the --no-group-append was not added when adding the subsystem. In this example it is defined as nqn.2016-06.io.spdk:cnode1.group1.
Important: Use this flag with extreme caution. When using this flag, if an initiator is connected to two different gateway groups with the same subsystem NQNs, not all NVMe-oF devices are listed with the nvme list command.
Important: NQN and gateway name values must not contain an underscore character.

The default --server-port value is 5500.

The following parameters can optionally be added to the command:
--max-namespaces
The maximum number of namespaces per subsystem default is 256.

Input type: Number

--serial-number
If the serial number is not specified, the number is randomly generated.

Input type: String

--dhchap_key
Use this flag to add a generated DH-HMAC-CHAP key for use with bidirectional in-band authentication.

Input type: String

--network-mask
You can add a network subnet to be associated with an NVMe-oF subsystem during subsystem creation.

These network subnets control the automatic creation and removal of NVMe‑oF listeners on gateways with IP addresses in the specified subnet. The system automatically creates listeners on all NVMe‑oF gateways with IP addresses in the specified subnet.

A network subnet does not need to be associated during subnet creation. For information about creating and managing network subnets after creating a subsystem, see Managing network subnets for auto‑listeners.

For example, 
[root@host01 ~]# ceph nvmeof subsystem add --nqn nqn.2016-06.io.spdk:cnode1.group1 --serial_number Ceph1230985676540 --dhchap_key DHHC-1:00:abEcb12F100FNlZkIOkv37idpu/notvalid!!si1VQ09KhKv2g:
Success

Changing a subsystem authentication key

Use the subsystem change_key command to change the in-band authentication keys for a subsystem.
Important: Using the subsystem change_key command without adding the --dhchap_key parameter removes the existing key.
ceph nvmeof subsystem change_key --nqn NQN --dhchap_key KEY_VALUE
Important: This command fails if controller‑level DHCHAP keys are defined for any host in the subsystem.
For example, 
[root@host01 ~]# ceph nvmeof subsystem change_key --nqn nqn.2016-06.io.spdk:cnode1.group1 --dhchap_key DHHC-1:00:abEcb12F100FNlZkIOkv37idpu/notvalid!!si1VQ09KhKv2g:
Success

Managing network subnets for auto‑listeners

You can add or remove network subnets associated with an NVMe‑oF subsystem after the subsystem is created. These network subnets control the automatic creation and removal of NVMe‑oF listeners on gateways with IP addresses in the specified subnet.

Procedure

  • Use the subsystem add_network command to add a network subnet to a subsystem for auto-listeners.
    Note: A subsystem can have multiple network subnets. 
    ceph nvmeof subsystem add_network NQN --network-mask SUBNET
    For example, 
    ceph nvmeof subsystem add_network nqn.2016-06.io.spdk:cnode7.mygroup --network-mask 10.243.64.0/24
    The system automatically creates listeners on all NVMe‑oF gateways with IP addresses in the specified subnet.
  • Use the subsystem del_network command to remove a network subnet from a subsystem for auto-listeners. 
    ceph nvmeof subsystem del_network NQN --network-mask SUBNET
    For example, 
    ceph nvmeof subsystem del_network nqn.2016-06.io.spdk:cnode7.mygroup --network-mask 10.243.64.0/24
    All automatically created listeners associated with the removed subnet are deleted.

What to do next

To verify the configured network subnets for a subsystem, list the subsystem details. The configured subnets are shown in the Network Mask field. 
ceph nvmeof subsystem list

Listing subsystems

Use the subsystem list command to list NVMe-oF subsystems associated with a specific gateway. 
ceph nvmeof subsystem list --gw_group GW_GROUP --server-address SERVER_ADDRESS
For example, 
[root@host01 ~]# ceph nvmeof subsystem list
+-----------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+
|Nqn                                |Serial Number     |Model Number        |Namespace Count|Subtype|Max Namespaces|Has Dhchap Key|Allow Any Host|Created Without Key|
+-----------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+
|nqn.2016-06.io.spdk:cnode1.mygroup1|Ceph21415707139038|Ceph bdev Controller|16             |NVMe   |1024          |False         |True          |False              |
|nqn.2016-06.io.spdk:cnode2.mygroup1|Ceph99777976460780|Ceph bdev Controller|16             |NVMe   |1024          |False         |True          |False              |
|nqn.2016-06.io.spdk:cnode3.mygroup1|Ceph1340674839888 |Ceph bdev Controller|16             |NVMe   |1024          |False         |True          |False              |
|nqn.2016-06.io.spdk:cnode4.mygroup1|Ceph3747224959823 |Ceph bdev Controller|16             |NVMe   |1024          |False         |True          |False              |
|nqn.2016-06.io.spdk:cnode6         |Ceph34173147934206|Ceph bdev Controller|0              |NVMe   |4096          |False         |False         |False              |
+-----------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+

Deleting a subsystem

Use the subsystem del command to delete NVMe-oF subsystems associated with a specific gateway. 
ceph nvmeof subsystem del [--nqn NQN] [--force] --gw_group GW_GROUP --server-address SERVER_ADDRESS
Use the --force parameter to delete any existing subsystem namespaces before deleting the subsystem.
Important: The --force parameter is optional but if not set and the subsystem contains a namespace, the subsystem deletion results in failure.
For example, 
[root@host01 ~]# ceph nvmeof subsystem del --nqn nqn.2016-06.io.spdk:cnode1.group1 --force
Success