mmvdisk server command

Manages mmvdisk recovery group servers for IBM Storage Scale RAID.

Synopsis

mmvdisk server configure {--node-class NcName | -N Node[,Node...]}  
                         [--maxblocksize {2M | 4M | 8M | 16M}] 
                         [--pagepool {nM |nG | n%}]
                         [--disk-list DiskExpr]
                         [--ignore-disk-list DiskExpr]
                         [--custom-config Attribute[=value][,Attribute[=value]...]]
                         [--update] [--fcm-enable-throttling {yes|no}] [--fcm-oos-threshold n%]
                         [--recycle {none |one |all | Number}] [--confirm]
or
mmvdisk server configure -N Node[,Node...]
                         --target-node-class NodeClass
                         [--recycle {none |one |all | Number}] [--confirm]
or
mmvdisk server configure --node-class NcName --verify [--disk-list DiskExpr]
                         [--ignore-disk-list DiskExpr] [-Y]
or
mmvdisk server configure -N Node[,Node...] --rack {clear | rack_id,[,rack_id...]} [--confirm]
or
mmvdisk server unconfigure {--node-class NcName | -N Node[,Node...]}
                           [--recycle {none | one | all | Number}] [--confirm]
or
mmvdisk server change --node-class NcName
                      {[--pagepool {nM | nG | n%}] 
                      [--vdisk-space n%] [--vdisk-threads n%] [--map-memory n%]}   
                      [--recycle {none | one | all | Number}] 
                      [--fcm-enable-throttling {yes|no}] [--confirm]
or
mmvdisk server list [-Y]
or
mmvdisk server list {--node-class NcName |-N Node[,Node...] | --recovery-group RgName} 
                    [{[--version][--config][--disk-topology ][--disk-list DiskExpr]} 
                    [--fanout N]] [-Y]
or
mmvdisk server list -N Node --disk-topology {[--disk-topology][--disk-list DiskExpr]} -L  [-Y]

Availability

Available on all IBM Storage Scale editions.

Description

Use the mmvdisk server command to manage mmvdisk recovery group servers.

Before an IBM Storage Scale RAID recovery group can be created, the mmvdisk node class for the intended recovery group or recovery group pair must be configured to run IBM Storage Scale RAID and the IBM Storage Scale daemon must be restarted on each server in the node class. The mmvdisk server configure command examines the nodes in an mmvdisk node class, verifies that the node class members meet the same memory and disk topology requirements, and sets the appropriate configuration parameters. The mmvdisk server configure command can be used to optionally recycle the IBM Storage Scale daemon on the node class to ensure that the new configuration is in effect.

The configuration parameters set by mmvdisk server configure depend on the amount of server memory and the type of server disk topology. All node class members must have, within ten percent, the same amount of memory, and all node class members must have the same general type of server disk topology: either a paired recovery group (Elastic Storage Server) disk topology or a scale-out recovery group (IBM Spectrum® Scale Erasure Code Edition) disk topology or a shared recovery group (IBM Elastic Storage® System 3000) disk topology. The mmvdisk server configure command will not configure a node class where the server memory differs by more than ten percent or where a mixture of recovery group topologies is present.

The mmvdisk server configure command can be used to optionally recycle the IBM Storage Scale daemon on the recovery group server node class to ensure that the new configuration is in effect for a subsequent mmvdisk recoverygroup create command.

The mmvdisk server configure command can be used with the --update option on a node class where a recovery group is already present. This can be useful when an updated version of the mmvdisk command uses new configuration parameters, or when the server memory or disk topology has changed and the configuration parameters based on memory or the number of disks should be recalculated. The --recycle option can be used with --update to make the updated configuration parameters take effect by restarting the IBM Storage Scale daemon on the servers in the node class.

The mmvdisk server configure command can also be used to configure individual nodes for IBM Storage Scale RAID to prepare for maintenance use cases where servers are replaced or added to a recovery group and its node class.

The --verify option of the mmvdisk server configure command verifies whether or not the configuration settings for an mmvdisk server node class are as expected. The --verify option will check the real memory and server disk topology for each node in the node class, and check whether the IBM Storage Scale RAID configuration attributes for the node class are set to the expected values for the memory and topology. The mmvdisk server configure --verify command can be used to check whether a new release of IBM Storage Scale RAID has updated best practice configuration values; to check whether configuration values are affected by changes that have been made to server memory or server disk topology; or to check whether accidental changes have been made to the node class configuration. The mmvdisk command bypasses the check for the attributes that belong to customized configuration attributes of node class. The customized configuration attributes might be set by the --custom-config option or the mmvdisk nodeclass change command. If an unexpected configuration is reported, the mmvdisk server configure --update command can be used to reset the node class to the correct configuration values.

The --rack option of the mmvdisk server configure command specifies the rack ID for one or more nodes when the -N option is used. The rack ID identifies the physical rack where the node is installed, if applicable. The system uses the rack ID to distribute data across racks, optimize stripe placement, and improve rack-level fault tolerance. This option is useful when multiple nodes are installed in the same rack.

The following rules apply to rack IDs configuration:
Consistent configuration within a node class
All nodes in one node class must have rack IDs, or none must have rack IDs. Mixed or partial configuration is not supported.
Immutable after the node-class configuration
After a node class is configured, the rack ID for any node in the node class cannot be changed.
Supported characters
Rack IDs must not contain spaces or special characters.
Case sensitivity
Rack IDs are case-sensitive. For example, RackA and racka are different rack IDs.

The mmvdisk server unconfigure command will remove IBM Storage Scale RAID configuration from a node class where a recovery group was either deleted or never created. The mmvdisk server unconfigure command can also be used to unconfigure IBM Storage Scale RAID on individual nodes for maintenance use cases where servers are replaced or deleted from a recovery group and its node class.

Use the mmvdisk server change command to safely change the server memory configuration of an mmvdisk server node class. The memory configuration of a node class affects the number and size of vdisk sets that can be defined in the recovery groups of the node class. The mmvdisk server change command will not allow changes that reduce the available memory below what is required by the vdisk set definitions in the node class. The three options that affect node class server memory for vdisk sets are:
  1. The --pagepool option, which adjusts the node class IBM Storage Scale pagepool attribute.
  2. The --vdisk-space option, which adjusts the node class nsdRAIDBufferPoolSizePct attribute for the percentage of pagepool dedicated to vdisks and IBM Storage Scale RAID.
  3. The --map-memory option, which adjusts the node class nsdRAIDNonStealableBufPct attribute for the percentage of vdisk space dedicated to the track address maps for each vdisk.

The mmvdisk server change command should only very rarely be necessary. The default values for the server memory configuration options should be suitable for most circumstances. The pagepool default is dependent on the real server memory and the recovery group type in the node class. The vdisk space default is always 80% of pagepool, and the map memory default is always 50% of the vdisk space. In the event that, for example, the map memory needs to be increased to address very large vdisks with very small block sizes, the mmvdisk server change --node-class NC --map-memory 60% command can be used to safely adjust the server memory configuration in node class NC.

When GNR server acts as both a GNR I/O server and a GPFS traditional NSD server (in the case of UStore), you must set the nsdRAIDAllowTraditionalNSD and nsdRAIDDesiredThreadPct attributes. By default, for GNR I/O server the nsdRAIDDesiredThreadPct attribute is set to 100%. When you needed to allow the GPFS traditional NSD server, issue the mmvdisk server change command to specify the --vdisk-threads option to change the value of the nsdRAIDDesiredThreadPct attribute. When nsdRAIDDesiredThreadPct attribute value is 100%, which is the default, the mmvdisk also sets the nsdRAIDAllowTraditionalNSD attribute value to no. Whereas, the NsdRAIDAllowTraditionalNSD parameter value is set to yes.

The --recycle one option can be used with the mmvdisk server change command to make any changes take effect by restarting IBM Storage Scale one node at a time in the node class.

The mmvdisk server list command, without any options, will list all the configured IBM Storage Scale RAID servers in the cluster. It identifies all recovery group servers and all nodes configured to be recovery group servers, regardless of whether the servers are members of an mmvdisk node class. This can be useful for planning server maintenance and planning the conversion of recovery groups and servers to mmvdisk management.

The mmvdisk server list command supports several options useful for showing server characteristics. The --node-class option will show whether the servers in a node class are active and the recovery group they serve. The --version option can be used to show the IBM Storage Scale and operating system versions for servers. The --config option can be used to show important aspects of the IBM Storage Scale RAID server configuration.

The mmvdisk server list --disk-topology option is important for reporting the supported IBM Storage Scale RAID disk topology on a node class or server. It should be used before mmvdisk server configure on a node class, to verify that all the node class members have the same server disk topology. It can also be used to help identify disk configuration problems that prevent nodes from acting as recovery group servers.

The mmvdisk server list --disk-list diskExpr option is important for reporting IBM Storage Scale RAID disk list on a node class or server. diskExpr is a subset of disk list which might be useful if you do not want to use all disks on a node to create recovery group.

Parameters

mmvdisk server configure
Configure nodes or an mmvdisk node class to be IBM Storage Scale RAID servers. The nodes or node class must not already be serving a recovery group or already be configured, except when the --update option is used for a node class where a recovery group is present.
mmvdisk server unconfigure
Unconfigure nodes or an mmvdisk node class to no longer be IBM Storage Scale RAID servers. The nodes or node class must no longer be associated with a recovery group.
mmvdisk server change
Change server memory configuration attributes for an mmvdisk server node class. The pagepool, vdisk buffer space, and vdisk map memory can be adjusted for a node class. The mmvdisk server change command will not allow changes that reduce the available memory below what is required for the vdisk set definitions in the node class.
mmvdisk server list
List IBM Storage Scale RAID servers and their characteristics. A node is considered to be a server if it is in the server list of a recovery group, or if it is member of an mmvdisk node class, or if it has the IBM Storage Scale RAID nsdRAIDTracks attribute set.
--node-class NcName
Specifies a single mmvdisk node class name to be configured, unconfigured, or listed.
-N Node[,Node...]
Specifies individual nodes to be configured, unconfigured, or listed.
--rack {clear | rack_id,[,rack_id...]}
Sets the rack ID for the specified node or nodes.
When you specify multiple nodes by using a comma-separated list with the -N option, you can also specify multiple rack IDs as a comma-separated list. The system maps rack IDs to nodes by position.
--confirm
Applies to the mmvdisk server configure, mmvdisk server unconfigure, and mmvdisk server change commands.
When you use the --recycle all option, this option provides advance confirmation that you accept the potential risk of quorum loss. When you use the --rack option, this option confirms the rack assignment for the specified node.
--target-node-class NcName
Specifies a single name to an mmvdisk node class to add to the individual nodes. This option must be used with the -N option.
--recycle {none | one | all | Number}
With mmvdisk server configure or mmvdisk server unconfigure, specifies the number of nodes to be simultaneously restarted so that IBM Storage Scale configuration changes can take effect. The default is none, meaning that the administrator is responsible for using the mmshutdown and mmstartup commands to recycle the nodes or node class. The keyword one will recycle one node at a time, the keyword all will recycle all specified nodes simultaneously, or a Number of nodes can be chosen to be recycled at the same time. Care should be taken when recycling nodes if it is desired to maintain quorum availability in the IBM Storage Scale cluster.
--pagepool {nM | nG | n%}
With mmvdisk server configure and mmvdisk server change, choose a specific IBM Storage Scale pagepool value. Normally, mmvdisk chooses the best appropriate pagepool size for the real memory and disk topology in the server node class, so this option should only be used under instruction from IBM®.
--maxblocksize {2M | 4M | 8M | 16M}
With mmvdisk server configure, also set the IBM Storage Scale cluster maxblocksize value. Even though maxblocksize is not a server-specific configuration value, IBM Storage Scale RAID is often used with large file system block sizes, and this option provides a convenient way to set the maximum file system block size. This would typically be done only when configuring the first recovery group in a cluster, and it requires that the administrator ensure that the IBM Storage Scale daemon on all cluster nodes is recycled (not only on the recovery group server nodes).
--custom-config Attribute[=value][,Attribute[=value]...]
With mmvdisk server configure, also set the IBM Storage Scale cluster customized configuration value. When mmvdisk (See IBM Storage Scale RAID: Administration) executes, it calls mmchconfig to set the value of attributes and add these attributes to node class's customized configuration attributes.
--update
With mmvdisk server configure for a node class that already has a recovery group present, update the node class configuration parameters according to the current mmvdisk version and current server memory and disk topology.
--fcm-enable-throttling {yes|no}
With mmvdisk server configure and mmvdisk server change, controls whether an FCM‑based IBM Storage Scale System performs throughput throttling based on space usage. When this option is set to yes, the system throttles I/O operations when the system enters Warning or Critical space usage states. When this option is set to no, system does not throttle I/O operations in these states. Regardless of this setting, the system stops write operations when the system state is Out‑of‑Space (OOS) state.
You can modify this flag at any time:
  • During initial system configuration, set this option by using the following command:
    mmvdisk server configure ... --fcm-enable-throttling {yes|no}
  • After the system is configured, set this option by using the following command:
    mmvdisk server change --fcm-enable-throttling {yes|no}
--fcm-oos-threshold n%
With mmvdisk server configure, defines the Out‑of‑Space (OOS) threshold for an FCM‑based IBM Storage Scale System. The OOS threshold specifies the physical space usage level at which the system stops all I/O operations. The default OOS threshold is 75%.
This option has the following constraints:
  • You can set this option only during initial system configuration by using the following command:
    mmvdisk server configure ... --fcm-oos-threshold n%
  • You cannot change this option after the system is configured.
--verify
With the mmvdisk server configure command, verify whether or not the IBM Storage Scale RAID configuration attributes for an mmvdisk server node class are set to the expected values.
--vdisk-space n%
With the mmvdisk server change command, specify the percentage of pagepool that should be reserved as space for vdisks and IBM Storage Scale RAID. This corresponds to the nsdRAIDBufferPoolSizePct configuration attribute, and the default value is 80%. Only in rare circumstances should this value need to be changed, so this option should only be used under instruction from IBM.
--map-memory n%
With the mmvdisk server change command, specify the percentage of the vdisk space that should be reserved to map the track addresses of vdisks. This corresponds to the nsdRAIDNonStealableBufPct configuration attribute, and the default value is 50%. Only in rare circumstances should this value need to be changed, so this option should only be used under instruction from IBM.
--vdisk-threads
With the mmvdisk server change command, specify the percentage of threads that must be reserved as threads for vdisks and IBM Storage Scale RAID. This corresponds to the nsdRAIDDesiredThreadPct configuration attribute, and the default value is 100%. When the attribute value is set to 100%, mmvdisk sets the nsdRAIDAllowTraditionalNSD attribute value to no. However, when the attribute value is not 100%, mmvdisk sets the nsdRAIDAllowTraditionalNSD attribute to yes. Only in rare circumstances, this value needs to be changed. Therefore, this option must only be used under instructions from IBM.
--recovery-group RgName
With mmvdisk server list, lists the servers for the named recovery group.
--version
With mmvdisk server list, lists the IBM Storage Scale and operating system version for the servers.
--config
With mmvdisk server list, lists the IBM Storage Scale RAID configuration for the servers.
--disk-topology
With mmvdisk server list, lists the discovered IBM Storage Scale RAID disk topology on the servers.
--disk-list DiskExpr
--ignore-disk-list DiskExpr
With mmvdisk server list, --disk-list is used to list the discovered IBM Storage Scale RAID disks on the servers. If DiskExpr is specified, mmvdisk shows only the list of disks that belong to DiskExpr.
With mmvdisk server configure, --disk-list specifies the disk list to be used and the --ignore-disk-list ignores the disk list for servers. For more information, see mmvdisk recoverygroup command.
--fanout N
With mmvdisk server list --disk-topology, chooses how many servers to query simultaneously for their disk topologies. The default is 32. A fanout of 1 can be useful when disk fabric errors with twin-tailed server pairs are suspected.
-L
With mmvdisk server list --disk-topology and a single node, provide a detailed listing of the discovered IBM Storage Scale RAID disk topology. This is very useful when disk topology errors are suspected.
-Y
Specifies that the mmvdisk server list and mmvdisk server configure --verify commands produce colon-delimited raw output.

Exit status

0
Successful completion.
nonzero
A failure has occurred.

Security

You must have root authority to run the mmvdisk server command.

The node on which the command is issued must be able to execute remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages.

Examples

  • To display the IBM Storage Scale RAID disk topology of the servers in mmvdisk node class, ESS01:
    mmvdisk server list --node-class ESS01 --disk-topology
    The system displays output similar to the following:
     node                                       needs    matching
number  server                            attention   metric   disk topology
------  --------------------------------  ---------  --------  -------------
     1  server01.gpfs.net                 no          100/100  ESS GL6
     2  server02.gpfs.net                 no          100/100  ESS GL6
  • To display the IBM Storage Scale RAID disk topology and disk list of the servers in mmvdisk node class, nc3500: 
    mmvdisk server list --node-class nc3500 --disk-topology --disk-list
    The system displays output similar to the following:
    
 node                                       needs    matching
number  server                            attention   metric   disk topology
------  --------------------------------  ---------  --------  -------------
     2  c145f11san08a                     no          100/100  ESS 3500 FN2 24 NVMe
     1  c145f11san08b                     no          100/100  ESS 3500 FN2 24 NVMe

                                      disk           list of
type  size(Mb)  spin   server         count       disk/location
----  --------  -----  -------------  -----  ------------------------
NVMe  3662830   0      c145f11san08a  24     nvme[0-23]n1 / 78E4001-[1-24]
                       c145f11san08b  24     nvme[0-23]n1 / 78E4001-[1-24]
  • To configure mmvdisk node class ESS01 to be IBM Storage Scale RAID recovery group servers and restart the daemons one at a time to make the configuration take effect: 
    mmvdisk server configure --node-class ESS01 x one

    The system displays output similar to the following:

    
mmvdisk: Checking resources for specified nodes.
mmvdisk: Node 'server01.gpfs.net' has a paired recovery group disk topology.
mmvdisk: Node 'server02.gpfs.net' has a paired recovery group disk topology.
mmvdisk: Node class 'ESS01' has a paired recovery group disk topology.
mmvdisk: Setting configuration for node class 'ESS01'.
mmvdisk: Node class 'ESS01' is now configured to be recovery group servers.
mmvdisk: Restarting GPFS on the following nodes:
mmvdisk:     server01.gpfs.net
mmvdisk: Restarting GPFS on the following nodes:
mmvdisk:     server02.gpfs.net
  • To show the status of the newly-configured mmvdisk node class ESS01:
    mmvdisk server list --node-class ESS01

    The system displays output similar to the following:

     node
number  server                            active   remarks
------  --------------------------------  -------  -------
     1  server01.gpfs.net                 yes      configured, idle
     2  server02.gpfs.net                 yes      configured, idle

    This shows that the servers are configured and running, but IBM Storage Scale RAID is idle because recovery groups have not yet been created.

  • To display the disk topology of the scale-out recovery group servers of recovery group RG01:
    mmvdisk server list --recovery-group RG01 --disk-topology

    The system displays output similar to the following:

     node                                       needs    matching
number  server                            attention   metric   disk topology
------  --------------------------------  ---------  --------  -------------
     1  s01                               no          100/100  ECE 2 SSD/NVMe and 5 HDD
     2  s02                               no          100/100  ECE 2 SSD/NVMe and 5 HDD
     3  s03                               no          100/100  ECE 2 SSD/NVMe and 5 HDD
     4  s04                               no          100/100  ECE 2 SSD/NVMe and 5 HDD
     5  s05                               no          100/100  ECE 2 SSD/NVMe and 5 HDD
  • To display the current status of the servers of recovery group RG01:
    mmvdisk server list --recovery-group RG01

    The system displays output similar to the following:

     node
number  server                            active   remarks
------  --------------------------------  -------  -------
     1  s01                               yes      serving RG01: LG003, LG008
     2  s02                               yes      serving RG01: root, LG005, LG010
     3  s03                               yes      serving RG01: LG002, LG007
     4  s04                               yes      serving RG01: LG001, LG006
     5  s05                               yes      serving RG01: LG004, LG009
  • In an mmvdisk server node class where vdisk sets are defined, a vdisk space of 20% and a map memory of 30% will almost certainly be a memory configuration error, and the mmvdisk server change command will not allow it:
    mmvdisk server change --node-class ESS --vdisk-space 20% --map-memory 30%

    The system displays output similar to the following:

    
mmvdisk: Checking resources for specified nodes.
mmvdisk: The requested change to node class 'ESS':
mmvdisk:   --vdisk-space 20% (config attribute 'nsdRAIDBufferPoolSizePct')
mmvdisk:   --map-memory 30% (config attribute 'nsdRAIDNonStealableBufPct')
mmvdisk: results in an available vdisk set map memory of 4558 MiB.
mmvdisk: This is less than the required vdisk set map memory of 26 GiB
mmvdisk: based on the current vdisk set definitions in the node class.
mmvdisk: Command failed. Examine previous error messages to determine cause.
    This example is shown only as a reference. Ensure that the mmvdisk server change command should almost never be necessary, since the default values for the server memory configuration options are suitable for most circumstances.
  • To verify that the mmvdisk server node class ESS is properly configured:
    mmvdisk server configure --node-class ESS --verify

    The system displays output similar to the following:

    
mmvdisk: Checking resources for specified nodes.
mmvdisk: Node class 'ESS' has 123 GiB total real memory per server.
mmvdisk: Node class 'ESS' has a paired recovery group disk topology.
mmvdisk: Node class 'ESS' has server disk topology 'ESS GL6'.
mmvdisk: Node class 'ESS' uses 'default.paired' recovery group configuration.

daemon configuration attribute      expected value  configured value
----------------------------------  --------------  ----------------
pagepool                            79665522278     as expected
nsdRAIDTracks                       128K            as expected
nsdRAIDBufferPoolSizePct            80              as expected
nsdRAIDNonStealableBufPct           50              as expected
pagepoolMaxPhysMemPct               90              as expected
nspdQueues                          64              as expected
nsdRAIDBlockDeviceMaxSectorsKB      0               as expected
nsdRAIDBlockDeviceNrRequests        0               as expected
nsdRAIDBlockDeviceQueueDepth        0               as expected
nsdRAIDBlockDeviceScheduler         off             as expected
nsdRAIDFlusherFWLogHighWatermarkMB  1000            as expected
nsdRAIDMaxPdiskQueueDepth           128             as expected
nsdRAIDSmallThreadRatio             2               as expected
nsdRAIDThreadsPerQueue              16              as expected
ignorePrefetchLUNCount              yes             as expected
maxBufferDescs                      2m              as expected
maxFilesToCache                     128k            as expected
maxMBpS                             30000           as expected
maxStatCache                        128k            as expected
nsdMaxWorkerThreads                 3842            as expected
nsdMinWorkerThreads                 3842            as expected
nsdMultiQueue                       256             as expected
numaMemoryInterleave                yes             as expected
panicOnIOHang                       yes             as expected
pitWorkerThreadsPerNode             32              as expected
workerThreads                       1024            as expected

mmvdisk: All configuration attribute values are as expected or customized.
  • To set rack ID for multiple nodes, run the following command:
    mmvdisk server configure -N node1,node2,node3 --rack rackA,rackB,rackB
    where, the rack IDs are mapped as follows:
    • node1 is mapped to rackA
    • node2 is mapped to rackB
    • node3 is mapped to rackB
  • To remove rack IDs from the specified node or nodes, run the following command:
    mmvdisk server configure -N node1,node2 --rack clear

See also

Location

/usr/lpp/mmfs/bin