mmvdisk nvmeof command

Manages mmvdisk NVMeOF target for IBM Storage Scale RAID.

Synopsis

mmvdisk nvmeof create  --port [N] {--addr IpAddr | --ifname IfDev}
                           [--family {ipv4|ipv6}] [--type {rdma|tcp}]
                           [--svcid ServicePort]
                           [--ana-groupid N] [--ana-state State]
                           [-N Node[,Node ...] | --node-class NcName]
or 
mmvdisk nvmeof create  --subsys SubsysName
                           [--cntlid-min N] [--cntlid-max N]
                           [--allowed-hosts {any | Node[,Node ...]}]
                           [--device-path Dev[,Dev ...] [--ana-groupid N]]
                           [-N Node[,Node ...] | --node-class NcName]
or
mmvdisk nvmeof list  [-N Node[,Node ...] | --node-class NcName] [-Y]
or 
mmvdisk nvmeof config  --vblock --ifname IfDev
                         [-N Node[,Node ...] | --node-class NcName]
or
mmvdisk nvmeof delete  --subsys {SubsysName|all}
                         [--allowed-hosts {all | Node[,Node ...]} |
                          --device-path Dev[,Dev ...] |
                          --namespace N[,N ...]]
                         [-N Node[,Node ...] | --node-class NcName]
or
mmvdisk nvmeof delete  --port {N|all}
                           [-N Node[,Node ...] | --node-class NcName]
or
mmvdisk nvmeof change  --port {N|all} --ana-groupid grpid
                         --ana-state anaState
                         [-N Node[,Node ...] | --node-class NcName]
or
mmvdisk nvmeof change  --port {N|all} --add --ana-groupid grpid
                           [--ana-state anaState]
                           [-N Node[,Node ...] | --node-class NcName]
or
mmvdisk nvmeof change  --port {N|all} --delete --ana-groupid grpid
                         [-N Node[,Node ...] | --node-class NcName]
or
mmvdisk nvmeof change  --subsys {SubsysName|all}
                           [--device-path Dev[,Dev ...] |
                            --namespace N[,N ...]] --ana-groupid grpid
                           [-N Node[,Node ...] | --node-class NcName]]
or
mmvdisk nvmeof bind --port {N|all} --subsys {SubsysName|all}
                           [-N Node[,Node ...] | --node-class NcName]
or
mmvdisk nvmeof unbind  --port {N|all} --subsys {SubsysName|all}
                         [-N Node[,Node ...] | --node-class NcName]
or
mmvdisk nvmeof enable  --subsys {SubsysName|all}
                           [--device-path Dev[,Dev ...] |
                           --namespace N[,N ...]]
                           [-N Node[,Node ...] | --node-class NcName]
or
mmvdisk nvmeof disable --subsys {SubsysName|all}
                         [--device-path Dev[,Dev ...] |
                          --namespace N[,N ...]]
                         [-N Node[,Node ...] | --node-class NcName]]
or
mmvdisk nvmeof clear [--confirm] [-N Node[,Node ...] | --node-class NcName]

Availability

Available on all IBM Storage Scale editions.

Description

Use the mmvdisk nvmeof command to manage the NVMeOF target.

Use this command to do the following tasks:
  • Create an NVMe target port.

    Use the mmvdisk nvmeof create --port [N] command to create an NVMe target port. The value of N specifies the port ID, and it must be either 1 or greater. If N value is not specified, the mmvdisk command selects an available value as a port ID. A port must be associated with an IP address, which can be specified by using the --addr IPAddr or --ifname IfDev option. IfDev refers to an InfiniBand device name, such as mlx5_0/1, or a network device name, such as ib0. The IP address is retrieved by the mmvdisk based on the specified InfiniBand or network device. The default protocol is IPv4. The --family {ipv4|ipv6} option can be used to specify a different protocol. The default transfer type is RDMA. The --type {rdma|tcp} option can be used to specify a different transfer type for the NVMe target port. By default, the service port number is set to 4420. It can be changed by using the --svcid ServicePort option. When creating an NVMe target port, an ANA group with an ID of 1 is automatically created, and the group state is set to optimized. Additional ANA groups can be created by using the --ana-groupid N option, where N is the desired group ID. The state of the new ANA group can be set by using the --ana-state option.

  • Create an NVMe target subsystem.

    Use the mmvdisk nvmeof create --subsys SubsysName command to create an NVMe target subsystem, and to add a new device path or allowed hosts to the subsystem. To allow other nodes to discover and connect to the NVMe target subsystem, use the --allowed-hosts option. The specifed any value of this option allows any node to discover and connect to the NVMe target subsystem. Multiple node names can be specified as a comma-separated list. The node names must belong to the same cluster as the NVMe target subsystem. The --device-path option is used to specify the device path for the NVMe target subsystem. Multiple device paths can be specified as a comma-separated list. If a subsystem with the specified name exists, the specified device paths are added to the existing subsystem. The --ana-groupid option is a property of the device path. If you want to specify different device paths with different ANA group IDs under the same NVMe target subsystem, issue the mmvdisk nvmeof create --subsys SubsysName command multiple times with a different device path and an ANA group ID. The --cntlid-min and --cntlid-max options are used to set the minimum and maximum values of the controller identifier (CNTLID). The same SubsysName must be on each I/O node for NVMe building blocks with twin-tails, such as IBM ESS 3000/3200/3500, native support for multiple paths of NVMe. In this case, the --cntlid-min and --cntlid-max options must be set to different scopes for each node. Because a client cannot connect to the same SubsysName from different hosts with the same cntlid at the same time. For IBM Storage Scale ECE, it is recommended to include the node number in the SubsysName, for example, include {nodeNumber} variable in the SubysName, which ensures that subsystem names are not identical on different nodes.

  • List information about the NVMeOF target.

    Use the mmvdisk nvmeof list command to list the NVMeOF target information, including the NVMe target ports and subsystems. The --discover-subsystem option displays the NVMeOF discover.conf entry form that can be used by NVMe hosts to discover and connect to the NVMeOF target.

  • Configure a VDisk over the fabric target.
  • Delete an NVMe target subsystem.

    Use the mmvdisk nvmeof delete --subsys command to delete one or all NVMe target subsystems, and to remove device paths, allowed hosts, or namespaces from a subsystem. If the --allowed-hosts option is specified, allowed hosts are removed from the subsystem. If the --device-path or --namespace option is specified, a specified device path or namespace is removed from the subsystem. To delete an NVMe target subsystem successfully, ensure that the subsystem is not bound to any NVMe target port before deleting it.

  • Delete the NVMe target port.
  • Configure an ANA group ID or set ANA group state for an NVMe target port.

    Use the mmvdisk nvmeof change command to add or delete ANA groups, change ANA group states under NVMe target ports, and change ANA groups under namespaces of NVMe target subsystems.

  • Configure an ANA group ID for the NVMe target subsystem.
  • Bind or unbind the NVMe target subsystem to the NVMe target port.
    • Use the mmvdisk nvmeof bind command to bind one or all NVMe target subsystems to one or all NVMe target ports.
    • Use the mmvdisk nvmeof unbind command to unbind one or all NVMe target subsystems from one or all NVMe target ports.
  • Enable or disable the NVMe target subsystem.
    • Use the mmvdisk nvmeof enable command to enable one or all NVMe target subsytems. If you specify the --device-path or --namespace option, it enables the specified namespaces under the specified NVMe target subsystem.
    • Use the mmvdisk nvmeof disable command to disable one or all NVMe target subsytems. If you specify the --device-path or --namespace option, it disables the specified namespaces under the specified NVMe target subsystem.
  • Clear all NVMe target.
    Use the mmvdisk nvmeof clear command to clear the current NVMe target configuration. This command deletes all NVMe target subsystems and ports.
    Attention: After you run this command, data on the NVMe target subsystems is deleted permanently. Back up important data before you use this command.

Parameters

mmvdisk nvmeof create
Creates an NVMe target subsystem and a port.
mmvdisk nvmeof config
Configures a VDisk over a fabric target.
mmvdisk nvmeof list
Lists an NVMe target port and a subsystem.
mmvdisk nvmeof delete
Delete an NVMe target port or a subsystem.
mmvdisk nvmeof change
Adds or deletes an ANA group, changes ANA group state for an NVMe target port. Changes ANA group for namespace of an NVMe target subsystem.
mmvdisk nvmeof bind
Binds an NVMe target subsystem to an NVMe target port.
mmvdisk nvmeof unbind
Unbinds an NVMe target subsystem from an NVMe target port.
mmvdisk nvmeof enable
Enables an NVMe target subsystem.
mmvdisk nvmeof disable
Disables an NVMe target subsystem.
mmvdisk nvmeof clear
Clears current NVMe target configuration.
--node-class NcName
Specifies a single mmvdisk node class name to be created, deleted, changed, bound, unbound, enabled, disabled, or cleared.
-N Node[,Node...]
Specifies individual nodes to be created, deleted, changed, bound, unbound, enabled, disabled, or cleared.
--subsys SubsysName
Specifies an NVMe target subsystem name for use with the mmvdisk nvmeof create, delete, change, bind, unbind, enable, and disable commands. SubsysName supports following variable: {nodeName}, {shortName}, and {nodeNumber}. When above variable occurs in SubsysName, mmvdisk replaces it with the actual value on a target node. For example, --subsys mysubsys.{nodeNumber} means --subsys mysubsys.1, if the node number of the specified node is 1.
--port {N|all}
Specifies an NVMe target port ID for use with the mmvdisk nvmeof create, delete, and change commands. N must be greater than 0. The option all is used to refer to all existing port IDs. When the port ID is not specified by using the mmvdisk nvmeof create command, the mmvdisk automatically selects a suitable port ID to create the NVMe target port.
--cntlid-min N
Specifies minimum values of the controller identifier (CNTLID) for use with the mmvdisk nvmeof create command to create an NVMe target port.
--cntlid-max N
Specifies maximum values of the controller identifier (CNTLID) for use with the mmvdisk nvmeof create command to create an NVMe target port.
--allowed-hosts {any | Node[,Node ...]}
Specifies allowed node that can discover and connect for use with the mmvdisk nvmeof create command to create an NVMe target subsystem. Multiple nodes can be specified by separating them with commas. The node name must belong to current cluster. The any value indicates that an NVMe target subsystem might be discovered and connected by any hosts. By default, the NVMe target subsystem does not allow any host to discover and connect.
--device-path Dev[,Dev ...]
Specifies a device path for use with the mmvdisk nvmeof create, delete, change, enable and disable commands. For example, /dev/nvme0n1. It supports the diskExpr format also. More information, refer the --disk-list DiskExp option in the mmvdisk recoverygroup command.
--namespace N[,N ...]
Specifies a namespace for use with the mmvdisk nvmeof delete, change, enable and disable commands. N must be an existing namespace ID under an NVMe targt subsystem. The device path and namespace have a one-to-one correspondence. Only the --device-path or --namespace option can be used at a time.
--ana-groupid N
--ana-stat State
Specifies an ANA group ID for use with the mmvdisk nvmeof create and change commands. An ANA group is a property of the namespace in the NVMe target subsystem, and its state is managed by the NVMe target port. The value of N must be an integer and greater than 0, and the default value is 1. The valid values for the State option are optimized, nonoptimized, inaccessible, persistent-loss, and change.
--addr IpAddr
--ifname IfDev
--family {ipv4|ipv6}
Specifies an IP address for use with the mmvdisk nvmeof create commands when you create an NVMe target port. The IpAddr value can be specified in the IPv4 or IPv6 format. If you specify the IPv6 address format, set the --family option to ipv6. The IfDev parameter can refer to either an InfiniBand device name (such as mlx5_0/1) or a network device name (such as ib0). The mmvdisk command retrieves the IP address by specifying the IfDev parameter. The default value of the --family option is ipv4.
--type {rdma|tcp}
Specifies type for use with the mmvdisk nvmeof create commands when creating an NVMe target port. The valid values are rdma and tcp.
--svcid ServicePort
Specifies a service port for use with mmvdisk nvmeof create commands when create an NVMe target port. Its default value is 4420.
--vblock
Specifies exporting VDisk flag for use with the mmvdisk nvmeof config commands.
--add
Adds an ANA group to an NVMe target port.
--delete
Deletes an ANA group from an NVMe target port.
--confirm
Confirms the NVMe configuration deletion. This option bypasses an interactive prompt that confirms NVMe configuration deletion.
-Y
Displays a colon-delimited raw output.

Exit status

0
Successful completion.
nonzero
A failure occurred.

Security

You must have root authority to run the mmvdisk nvmeof command.

The node on which the command is issued must be able to run remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages.

Example

  • Create a port by using the nc3500 node class:
    # mmvdisk nvmeof create --port --ifname ib0 --type rdma --nc nc3500
    A sample output is as follows:
    c145f11san08a: mmvdisk: Create Port '1' addr '13.1.145.132' svcid '4420' type 'rdma' successfully.
c145f11san08b: mmvdisk: Create Port '1' addr '13.1.145.134' svcid '4420' type 'rdma' successfully.
  • Create a subsystem by using the nc3500 node class.
    mmvdisk nvmeof create --subsys ustore --device-path 78E4001-[7-12,19-24] --cntlid-min 1 --cntlid-max 1024 --allowed-hosts any -N c145f11san08a
    A sample output is as follows:
    c145f11san08a: mmvdisk: Create Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '1' device path '/dev/nvme16n1(/dev/disk/by-id/nvme-eui.35344b304ea005820025384500000004)' for Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '2' device path '/dev/nvme23n1(/dev/disk/by-id/nvme-eui.35344b304ea003330025384500000004)' for Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '3' device path '/dev/nvme17n1(/dev/disk/by-id/nvme-eui.35344b304ea004820025384500000004)' for Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '4' device path '/dev/nvme24n1(/dev/disk/by-id/nvme-eui.35344b304ea005790025384500000004)' for Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '5' device path '/dev/nvme18n1(/dev/disk/by-id/nvme-eui.35344b304ea004770025384500000004)' for Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '6' device path '/dev/nvme25n1(/dev/disk/by-id/nvme-eui.35344b304ea004830025384500000004)' for Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '7' device path '/dev/nvme8n1(/dev/disk/by-id/nvme-eui.35344b304ea005860025384500000004)' for Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '8' device path '/dev/nvme2n1(/dev/disk/by-id/nvme-eui.35344b304ea006610025384500000004)' for Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '9' device path '/dev/nvme7n1(/dev/disk/by-id/nvme-eui.35344b304ea006880025384500000004)' for Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '10' device path '/dev/nvme1n1(/dev/disk/by-id/nvme-eui.35344b304ea004140025384500000004)' for Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '11' device path '/dev/nvme6n1(/dev/disk/by-id/nvme-eui.35344b304ea005110025384500000004)' for Subsys 'ustore'.
c145f11san08a: mmvdisk: Create namespace '12' device path '/dev/nvme0n1(/dev/disk/by-id/nvme-eui.35344b304ea005030025384500000004)' for Subsys 'ustore'.
  • Bind all subsystems to all ports by using the nc3500 node class:
    /usr/lpp/mmfs/bin/mmvdisk nvmeof bind --port all --subsys all --node-class nc3500
    A sample output is as follows:
    c145f11san08a: mmvdisk: Bind Subsys 'ustore' to Port '1'.
c145f11san08b: mmvdisk: Bind Subsys 'ustore' to Port '1'.
  • List all NVMe target configuration.
    mmvdisk nvmeof list --nc nc3500
    A sample output is as follows:
    
 node
number  server         port   type    addr          svcid   subsystems
------  -------------  -----  ------  ------------  ------  ------------
     1  c145f11san08b  1      rdma    13.1.145.134  4420    ustore
     2  c145f11san08a  1      rdma    13.1.145.132  4420    ustore

 node
number  server         subsystem     NS      path           state
------  -------------  ------------  ------  -------------  --------
     1  c145f11san08b  ustore        1     /dev/nvme16n1  Enabled
                                     2     /dev/nvme23n1  Enabled
                                     3     /dev/nvme17n1  Enabled
                                     4     /dev/nvme24n1  Enabled
                                     5     /dev/nvme18n1  Enabled
                                     6     /dev/nvme25n1  Enabled
                                     7     /dev/nvme8n1   Enabled
                                     8     /dev/nvme2n1   Enabled
                                     9     /dev/nvme7n1   Enabled
                                     10    /dev/nvme1n1   Enabled
                                     11    /dev/nvme6n1   Enabled
                                     12    /dev/nvme0n2   Enabled
     2  c145f11san08a                1     /dev/nvme16n1  Enabled
                                     2     /dev/nvme23n1  Enabled
                                     3     /dev/nvme17n1  Enabled
                                     4     /dev/nvme24n1  Enabled
                                     5     /dev/nvme18n1  Enabled
                                     6     /dev/nvme25n1  Enabled
                                     7     /dev/nvme8n1   Enabled
                                     8     /dev/nvme2n1   Enabled
                                     9     /dev/nvme7n1   Enabled
                                     10    /dev/nvme1n1   Enabled
                                     11    /dev/nvme6n1   Enabled
                                     12    /dev/nvme0n1   Enabled
  • Clear all NVMe target configuration. 
    mmvdisk nvmeof clear --nc nc3500
    A sample output is as follows:
    mmvdisk: This command will clear all nvmet configuration.

mmvdisk: Do you wish to continue (yes or no)? yes

See also

Location

/usr/lpp/mmfs/bin