mmvdisk recoverygroup command

Manages mmvdisk recovery groups for IBM Spectrum Scale RAID.

Synopsis

or
mmvdisk recoverygroup create  --recovery-group RgName
                             --node-class NcName [--match  N]
                             [--log-home-da DaName] [-v {yes | no}]
                             [--fanout N] [--defer-log-format]
                             [--trim-da DaType[,DaType...]]
                             [--split-da [DaType:]N[,DaType:N...]]
             
or
mmvdisk recoverygroup create --complete-log-format
            
or
mmvdisk recoverygroup convert --recovery-group RgName1[,RgName2]
                              --node-class NcName
                              [--inherit-config {yes | no [--pagepool {nM | nG | n%}] 
                              [--recycle {none | one | all | Number}]}]
                              [--dry-run]
            
or
mmvdisk recoverygroup resize --recovery-group RgName1[,RgName2]
                             [-v {yes | no}] [--fanout N]
                             [--dry-run]
            
or
mmvdisk recoverygroup add  --recovery-group RgName
                             -N Node[,Node...] [--match  N]
                             [--fanout N] [-v {yes | no}]                             
            
or
mmvdisk recoverygroup add --recovery-group RgName --complete-node-add
or
mmvdisk recoverygroup replace  --recovery-group RgName
                             -N Node --new-node NewNode [--match N]
                             [--fanout N] [-v {yes | no}]                             
            
or
mmvdisk recoverygroup delete  --recovery-group RgName [-N Node[,Node...]] 
                             [--confirm] [-p] [--sanitize]
or
mmvdisk recoverygroup change --recovery-group RgName 
                             {--primary Node [--backup Node] [-v {yes | no}] |
                             --active {Node | DEFAULT} | 
                             --version {VersionString | LATEST} | 
                             --declustered-array DaName {[--spares NumberOfSpares]
                             [--vcd-spares NumberOfVcdSpares] [--scrub-duration NumberOfDays]
                             [--replace-threshold NumberOfPdisks]
                             [--bit-error-rate {enable | disable}]
                             [--trim-da {yes | no}]} |
                             --declustered-array {all | DaName[,DaName...]} 
                             {--spare-nodes {DEFAULT | N} | --spare-pdisks {DEFAULT | N}} |
                             --suspend -N Node [--window N]|
                             --resume -N Node |
                             --restart |
                             --refresh-pdisk-info}
            
or
mmvdisk recoverygroup list [-Y]
or
mmvdisk recoverygroup list --recovery-group RgName
                           [--version] [--server] [--declustered-array] [--pdisk]
                           [--log-group] [--vdisk-set] [--vdisk] [--fault-tolerance] [-Y]
            
or
mmvdisk recoverygroup list --recovery-group RgName  --all [-Y]
            
or
mmvdisk recoverygroup list  --declustered-array [-Y]
            
or
mmvdisk recoverygroup list  --not-ok [-Y]
            
or
mmvdisk recoverygroup list --recovery-group RgName  --events
                           [--days Days] [--decimal]
                           [--long-term Codes] [--short-term Codes]
            

Availability

Available on all IBM Spectrum Scale editions.

Description

Use the mmvdisk recoverygroup command to manage IBM Spectrum Scale RAID recovery groups.

The mmvdisk recoverygroup command will:
  • Create new mmvdisk paired recovery groups (Elastic Storage Server).
  • Create a new mmvdisk scale-out recovery group (IBM Spectrum® Scale Erasure Code Edition).
  • Create a new mmvdisk shared recovery group (IBM Elastic Storage® System 3000).
  • Convert existing non-mmvdisk paired recovery groups to run under mmvdisk management.
  • Resize existing mmvdisk recovery groups to a supported disk topology upgrade.
  • Add servers to an mmvdisk scale-out recovery group.
  • Replace a server in an mmvdisk scale-out recovery group.
  • Delete servers from an mmvdisk scale-out recovery group.
  • Change the attributes of mmvdisk recovery groups.
  • List recovery group information.
  • Delete mmvdisk recovery groups.

Use the mmvdisk recoverygroup create command to create an IBM Spectrum Scale RAID recovery group pair (Elastic Storage Server). This requires an mmvdisk node class containing the two servers. The node class must already have been configured using the mmvdisk server configure command, and the servers must have been restarted with the configuration in effect. The two servers must have identical IBM Spectrum Scale RAID disk topologies. The names of the two paired recovery groups must be supplied using the --recovery-group RgName1,RgName2 parameter. The primary and backup servers for the two recovery groups are determined by a case-insensitive sort of the two server names. The first supplied recovery group will get the server that sorts first as primary and the server that sorts second as backup; the second supplied recovery group will get the server that sorts second as primary and the server that sorts first as backup. If desired, these assignments can be changed later using the mmvdisk recoverygroup change command.

Use the mmvdisk recoverygroup create command to create an IBM Spectrum Scale RAID scale-out recovery group (IBM Spectrum Scale Erasure Code Edition). This requires an mmvdisk node class containing from 4 to 32 identically equipped servers. The node class must already have been configured using the mmvdisk server configure command, and the servers must have been restarted with the configuration in effect. The servers must have identical IBM Spectrum Scale RAID disk topologies. The name of the recovery group must be supplied using the --recovery-group Rgname parameter.

Use the mmvdisk recoverygroup create command to create an IBM Spectrum Scale RAID shared recovery group (IBM Elastic Storage System 3000). This requires an mmvdisk node class containing two servers that share the same set of disks. The node class must already have been configured using the mmvdisk server configure command, and the servers must have been restarted with the configuration in effect. The servers must have identical IBM Spectrum Scale RAID disk topologies. The name of the recovery group must be supplied using the --recovery-group RgName parameter.

The mmvdisk recoverygroup create command treats a recovery group and its log vdisks as a unit: Creating mmvdisk recovery groups also formats the appropriate log vdisks for the recovery groups as a serial part of the process. When creating large numbers of recovery groups in sequence, it might be desired to defer the format of the log vdisks until after all the recovery groups proper have been created. This can speed the process by taking advantage of parallelization to format the deferred log vdisks all at once. The --defer-log-format flag can be used with mmvdisk recoverygroup create to postpone log vdisk formatting. After all the base recovery groups have been created, the mmvdisk recoverygroup create --complete-log-format command will format any deferred log vdisks. User vdisk sets can not be defined in a recovery group with deferred log vdisks until the log format is completed.

Before new mmvdisk recovery groups can be created, all existing non-mmvdisk recovery groups in a cluster must be converted to mmvdisk management.

Existing non-mmvdisk paired recovery groups can be placed under mmvdisk management with the mmvdisk recoverygroup convert command. This procedure is applied once per recovery group pair. The name of one or both existing recovery groups in a pair must be supplied; if only one is supplied, the other is automatically determined since they share the same two servers. A node class name must also be supplied. The node class name must not be in use, and will be created as an mmvdisk node class containing the two servers for the recovery group pair. The mmvdisk recoverygroup convert command examines each vdisk in the two recovery groups, and matches them automatically into converted vdisk sets. Converted vdisk sets collect from all converted recovery groups those vdisks that have identical attributes and are in the same file system. Converted vdisk sets are given constructed names such as VS002_fs2, which can later be changed to something more meaningful using the mmvdisk vdiskset rename command. The completion of recovery group conversion requires that the IBM Spectrum Scale daemon be restarted on the converted servers. This can be performed using the --recycle flag with the mmvdisk recoverygroup convert command.

Existing mmvdisk recovery group can be resized to a supported disk topology upgrade using the mmvdisk recoverygroup resize command. Supported upgrade paths involve adding new disk capacity in to an existing recovery group. The procedure requires careful hardware and software coordination performed by IBM® Service. After the upgraded disk topology hardware has been validated, the mmvdisk recoverygroup resize command integrates the additional capacity into the recovery groups.

The mmvdisk recoverygroup add command can be used to add servers to an mmvdisk scale-out recovery group. Multiple server nodes can be added at a time, and all the servers must be configured using the mmvdisk server configure command. Two mmvdisk recoverygroup add commands are required to add a server node to a scale-out recovery group. The first mmvdisk recoverygroup add command adds the disk capacity of the -N Node[,Node...] servers to the recovery group and initiates a rebalance of RAID stripes across the new capacity. It creates a rebalance callback for the second command. When the rebalance has completed, the second command, mmvdisk recoverygroup add --complete-node-add, is invoked by the callback automatically to format the added log groups for the server and to extend any vdisk sets and file systems that use the recovery group into the new capacity.

The mmvdisk recoverygroup replace command can be used to replace a failed or undesired server in an mmvdisk scale-out recovery group. The undesired server is designated with the -N Node parameter, and the replacement server is designated with the --new-node NewNode parameter. The replacement server node must be equipped identically to the other servers in the recovery group (especially regarding memory and disk topology), and must first have been configured using the mmvdisk server configure command. Replacing a server migrates the RAID data from the old server's pdisks to the replacement server's pdisks; no changes are performed to any file systems or vdisk sets.

The mmvdisk recoverygroup delete command with the -N Node[,Node...] parameter can be used to delete server nodes from a scale-out recovery group. Because this removes two log groups of vdisk set members from the scale-out recovery group for each server node, all file systems that use vdisk sets from the recovery group must first be reduced by server using the mmvdisk filesystem delete -N Node command. It is not always possible to delete a server from a scale-out recovery group or file system. The reallocated disk capacity when the server's disks are removed must be able to accommodate the remaining log groups and vdisk set members, and the number of pdisks remaining in the declustered arrays must be sufficient to maintain vdisk RAID code width and VCD (vdisk configuration data) fault tolerance. The mmvdisk filesystem delete and mmvdisk recoverygroup delete commands will check first to see that these conditions will be satisfied before proceeding to delete servers from a file system or recovery group.

The mmvdisk recoverygroup change command can be used to change the server assignments of an mmvdisk paired recovery group, to change the recovery group feature version, or to change certain declustered array attributes. The normal use case for changing the server assignments is to swap the primary and backup servers of a paired recovery group, using the --primary Node --backup Node parameter. There is also a maintenance use case for temporarily deleting a backup server and then restoring a possibly different backup server. See IBM Spectrum Scale RAID: Administration for more information. Regardless of the use case, the recovery group will immediately switch over to being served by the new primary server. Only paired recovery groups have primary and backup server assignments.

The active server for a paired recovery group can be changed immediately, without permanently changing the primary and backup assignments, by using the mmvdisk recoverygroup change --active Node command. The new active server must be a member of the recovery group node class. For planned temporary recovery group reassignment, use --active for maintenance operations. If the keyword DEFAULT is used instead of a server name, it means to make the primary server the active server for the specified paired recovery group.

For a shared recovery group, the mmvdisk recoverygroup change --active Node command means to make the specified node the server for all four user log groups and the root log group. The specified node therefore temporarily becomes the sole active server for the entire shared recovery group, leaving the other server idle. This should only be done in unusual maintenance situations, since it is normally considered an error condition for one of the servers of a shared recovery group to be idle. If the keyword DEFAULT is used instead of a server name, it restores the normal default balance of log groups, making each of the two servers responsible for two user log groups.

The mmvdisk recoverygroup change --active command does not apply to scale-out recovery groups.

The recovery group feature version can be changed using the mmvdisk recoverygroup change --version {VersionString | LATEST} command, where the keyword LATEST means the latest feature version available with the currently installed IBM Spectrum Scale RAID software.

For paired and shared recovery groups, the effective pdisk spares and VCD spares (vdisk configuration data replicas) of each declustered array are set to recommended values when the recovery group is created. Under the direction of IBM Service, the effective pdisk spares can be changed using the mmvdisk recoverygroup change --declustered-array DaName command with --spares parameter. For paired recovery groups, the same applies to VCD spares with the --vcd-spares parameter. For shared recovery groups, the value of VCD spares is a function of the spare and non-spare pdisk totals and cannot be changed directly.

For scale-out recovery groups, the effective pdisk spares of each declustered array is set to a minimum allowable default value when the recovery group is created. The mmvdisk recoverygroup change command can be used to increase pdisk spares so that rebuild can accommodate a larger number of disk or node failures. Either the --spare-pdisks or --spare-nodes parameter can be used for the declustered arrays of scale-out recovery groups. The --spare-pdisks parameter gives the exact number of spare pdisks desired for the specified declustered arrays. The --spare-nodes parameter determines the effective spare pdisks value according to the number of pdisks each node contributes to a particular declustered array. For example, if each node in a scale-out recovery group contributes 5 pdisks to declustered array DA2, --spare-nodes 3 will attempt to set 15 spare pdisks in DA2. The spares for scale-out recovery groups are subject to minimum and maximum values that depend on the number of nodes in the recovery group and the number of pdisks per node. As with shared recovery groups, the VCD spares setting for a scale-out recovery group declustered array is derived automatically from the spare and non-spare pdisk totals and can not be changed.

For more information on declustered array spares, see IBM Spectrum Scale RAID: Administration.

The scrub duration, replace threshold, and bit error rate of each declustered array in a recovery group are set to recommended values when the recovery group is created. They can be changed using the mmvdisk recoverygroup change --declustered-array DaName command with the --scrub-duration , --replace-threshold and --bit-error-rate parameters. See the Parameters section for more information.

The mmvdisk recoverygroup change --suspend command can be used to temporarily suspend a single node in a scale-out recovery group so that maintenance can be performed on the node. IBM Spectrum Scale is stopped on the suspended node and its pdisks are also suspended. Scale-out server maintenance should be performed quickly to avoid unnecessary RAID rebuild overhead. By default, there is a window of 20 minutes before the data on a suspended server's pdisks is rebuilt onto other nodes. The --window N option can be used to adjust the number of minutes before the suspended server's pdisks are rebuilt. The value of N must be a number between 10 and 60 inclusive. When node maintenance is complete, the mmvdisk recoverygroup change --resume command must be used to start IBM Spectrum Scale on the node and make its pdisks available again.

The mmvdisk recoverygroup change --restart command can be used to try to restart a recovery group where the servers are still active in the cluster, but one or more log groups of the recovery group are not. Restarting a recovery group is much less disruptive to a cluster than using the mmshutdown and mmstartup commands on the recovery group server node class. The normal use case for the mmvdisk recoverygroup change --restart command is after a recovery group has deliberately resigned due to loss of connectivity to file system vdisk NSD client nodes performing replicated writes, but it can also be used when individual log groups have resigned. If a recovery group is already completely active, the mmvdisk recoverygroup change --restart command will refuse to restart it.

The mmvdisk recoverygroup change --refresh-pdisk-info command can be used to make a recovery group reread the hardware information for all its pdisks. This can be useful if maintenance procedures have updated, for example, the pdisk firmware.

Recovery group information is listed using the mmvdisk recoverygroup list command. With no arguments. mmvdisk recoverygroup list shows all recovery groups in the cluster, mmvdisk and non-mmvdisk alike, whether they are active or not and the current server. This can be useful to tell which recovery groups have and have not been converted to mmvdisk management. With a specific --recovery-group RgName argument, this information can be displayed for a single recovery group.

The mmvdisk recoverygroup list --declustered-array command gives an at-a-glance summary for each declustered array in each recovery group. The mmvdisk recoverygroup list --not-ok command shows only those recovery groups that are down, have inactive servers, or have pdisks in need of replacement.

For a specified recovery group that is under mmvdisk management, the mmvdisk recoverygroup list --recovery-group RgName can provide more detailed information; use the following flags to choose different output sections:
  • --version shows recovery group version information.
  • --server shows recovery group server information.
  • --declustered-array shows recovery group declustered array information.
  • --pdisk shows recovery group pdisk information.
  • --log-group shows recovery group log group information.
  • --vdisk-set shows recovery group vdisk set information.
  • --vdisk shows recovery group individual vdisk information.
  • --fault-tolerance shows recovery group fault tolerance information.
Any combination of the detailed information flags can be used, or the --all flag can be used to show all sections.

The recovery group event log is displayed independently of the detailed information by using the mmvdisk recoverygroup list --events command. By default, all available events in the short- and long-term event logs are displayed. The parameters --days Days, --short-term Codes, and --long-term Codes can be combined to select a subset of the event log to display. The event Codes argument is a string containing one or more of the letters:

  • C for configuration events.
  • E for error events.
  • W for warning events.
  • I for informational events.
  • D for detail events.

An mmvdisk recovery group is deleted using the mmvdisk recoverygroup delete command. Even though paired recovery groups are created in pairs, they are deleted individually. The order in which the two recovery groups of a pair are deleted does not matter. A recovery group can not be deleted until it is undefined from any vdisk sets. By necessity, deleting a recovery group also deletes all the log vdisks in the recovery group.

Parameters

mmvdisk recoverygroup create

Create mmvdisk recovery groups.

mmvdisk recoverygroup convert

Convert existing non-mmvdisk paired recovery groups to mmvdisk management.

mmvdisk recoverygroup resize

Resize existing mmvdisk recovery groups to a supported disk topology upgrade.

mmvdisk recoverygroup add

Add server nodes to a scale-out recovery group.

mmvdisk recoverygroup replace

Replace a server node in a scale-out recovery group.

mmvdisk recoverygroup delete

Delete an mmvdisk recovery group, or delete specified servers from a scale-out recovery group.

mmvdisk recoverygroup change

Change the characteristics of an mmvdisk recovery group.

mmvdisk recoverygroup list

List all recovery groups, or list detailed information for mmvdisk recovery groups.

--recovery-group Rgname1,RgName2

Specifies a recovery group pair for use with the mmvdisk recoverygroup create, convert, and resize commands.

--recovery-group Rgname

Specifies a single recovery group for use with the mmvdisk recoverygroup create, add, replace, delete, change, list, and resize commands.

--node-class NcName

With the mmvdisk recoverygroup create or mmvdisk recoverygroup convert commands, specifies the mmvdisk node class to be associated with the recovery groups.

--match N

Specifies the minimum IBM Spectrum Scale RAID disk topology matching metric to be accepted for recovery group creation. The default is 100 (out of 100), meaning that only perfectly matching disk topologies can be used to create recovery groups. It is strongly advised against creating recovery groups with imperfectly matching disk topologies.

--multiple-da-legacy

Specifies that multiple declustered arrays be created within recovery groups on GL4 and GL6 IBM Spectrum Scale RAID disk topologies. This option is provided only for compatibility with existing GL4 and GL6 installations that use multiple DAs. If compatibility with existing multiple DA installations is not required, this option should not be used.

--bit-error-rate {enable | disable}

Specifies whether bit error rate enforcement shall be enabled for this declustered array. If the bit error rate of a disk within a DA is predicted to be beyond an expected threshold, that pdisk will be marked as failing. The default setting for declustered arrays that support bit error rate enforcement is enable. If mmvdisk rg list shows a value of N/A under the BER column, it means that bit error rate enforcement is not supported on this declustered array and is permanently disabled.

--split-da [DaType:]N[,DaType:N...]

With the mmvdisk recoverygroup create command, specifies maximum pdisk number per node for each declustered array base on different disk types. The disk type can be all, nvme, ssd, or hdd. If specified a number without disk type, that means all declustered array use the same pdisk number per node per disk type.

--trim-da DaType[,DaType...]

With the mmvdisk recoverygroup create command, specify that the trim function should be enabled for declustered arrays of the specified disk hardware types. The disk hardware type can be all or a comma-separated list chosen from nvme, ssd, or hdd. The default is to not enable the trim function for any declustered array in a recovery group. The IBM Spectrum Scale RAID daemon will refuse to create a recovery group when trim is requested on a disk hardware type that is not supported. See IBM Spectrum Scale RAID: Administration for a description of the declustered array trim function.

--trim-da {yes | no}

With the mmvdisk recoverygroup change command, specify that the trim function should be enabled or disabled for a declustered array. The IBM Spectrum Scale RAID daemon will refuse to enable trim on a declustered array with a disk hardware type that is not supported. See IBM Spectrum Scale RAID: Administration for a description of the declustered array trim function.

--log-home-da DaName

With the mmvdisk recoverygroup create command for scale-out recovery groups with multiple declustered arrays, choose the declustered array where log home vdisks are to be placed. The default is to choose the SSD/NVMe DA with the greatest number of pdisks; or, if no SSD/NVMe DA is present, the HDD DA with the greatest number of pdisks.

-v {yes | no}

With the mmvdisk recoverygroup create, resize, add, and replace commands, specifies whether pdisks should be verified to be empty of IBM Spectrum Scale RAID data before they are initialized in a recovery group. The default is yes, verify that pdisks are empty of data. With the mmvdisk recoverygroup change --primary Node [--backup Node] command, specifies whether or not to verify that the recovery group pdisks are present on the new servers. The default is yes.

--fanout N

Specifies how many servers to query simultaneously for their disk topologies. The default is 32.

--defer-log-format

Specifies that the log vdisks should not be formatted at the time recovery groups are created.

--complete-log-format

Specifies that all deferred log vdisks in any recovery groups should be formatted.

-N Node

With the mmvdisk recoverygroup replace command, specifies the server node to be replaced.

-N Node[,Node...]

With the mmvdisk recoverygroup add and delete commands, specifies the server nodes to be added or deleted.

--new-node NewNode

With the mmvdisk recoverygroup replace command, specifies the replacement server node to take the place of the node being replaced.

--complete-node-add

With the mmvdisk recoverygroup add command, complete the addition of a scale-out recovery group server by creating new log groups and extending the vdisk sets and file systems that use the recovery group. This action will be invoked by relanace callback automatically once the rebalance has completed. It will verify that the new server's pdisks have rebalanced to make sufficient fault tolerance available for the new log groups and vdisks.

--inherit-config {yes | no}

With the mmvdisk recoverygroup convert command, specifies whether the configuration settings of non-mmvdisk managed recovery groups are preserved after conversion. The default value is yes, which means the configuration settings are preserved. If no is specified, mmvdisk command uses the recommended settings to overwrite the configuration of the non-mmvdisk recovery groups.

--pagepool {nM | nG | n%}

With the mmvdisk recoverygroup convert command and --inherit-config specified as no, choose a specific IBM Spectrum Scale pagepool value. Normally, mmvdisk recoverygroup convert chooses the best appropriate pagepool size for the real memory available in the server node class, so this option should only be used under instruction from IBM.

--recycle {none | one | all | Number}

With the mmvdisk recoverygroup convert command and --inherit-config specified as no, specifies the number of nodes to be simultaneously restarted so that the converted IBM Spectrum Scale configuration changes can take effect. The default is none, meaning that the administrator is responsible for using the mmshutdown and mmstartup commands to recycle the nodes or node class. The keyword one will recycle one node at a time, the keyword all will recycle all specified nodes simultaneously, or a Number of nodes can be chosen to be recycled at the same time. Care should be taken when recycling nodes if it is desired to maintain quorum availability in the IBM Spectrum Scale cluster.

--dry-run

With the mmvdisk recoverygroup convert command, specifies that the process of converting an existing non-mmvdisk recovery group pair merely be simulated, without making any actual changes to the cluster configuration. This will show the conversion and vdisk set discovery process and allow it to be examined prior to making the changes permanent. With the mmvdisk recoverygroup resize command, --dry-run can be used to validate that a recovery group pair is able to be resized to a supported disk topology upgrade, without actually performing the resize.

--primary Node [--backup Node]

Specifies the new primary and backup servers for a recovery group. Caution must be used when omitting a backup server, as this reduces recovery group availability. See IBM Spectrum Scale RAID: Administration for more information.

--active {Node | DEFAULT}
With the mmvdisk recoverygroup change command, manipulate the current active servers for paired and shared recovery groups. This parameter does not apply to scale-out recovery groups. For a paired recovery group, specifying the node name of one of the two servers makes the specified node the current active server for the recovery group. Specifying the keyword DEFAULT makes the normal primary server the current active server. For a shared recovery group, specifying the node name of one of the two servers makes the specified node the active server for all log groups of the recovery group, and leaves the other server idle. Specifying the keyword DEFAULT restores the normal balance where each of the two servers serves two of the four user log groups.
--version {VersionString | Latest}

With the mmvdisk recoverygroup change command, specifies that the recovery group feature version be updated. The keyword LATEST updates to the latest version available with the currently installed IBM Spectrum Scale RAID software.

--declustered-array DaName

With the mmvdisk recoverygroup change command, specifies a target declustered array for which attributes are to be changed.

--spares NumberOfSpares

For paired or shared recovery groups with the mmvdisk recoverygroup change command, specifies the number of effective spares to be reserved in the target declustered array. The default depends on the recovery group IBM Spectrum Scale RAID disk topology, and should only be changed under the guidance of IBM Service. For scale-out recovery groups, the --spare-pdisks or --spare-nodes option must be used.

--vcd-spares NumberOfVcdSpares

For paired recovery groups with the mmvdisk recoverygroup change command, specifies the number of disks that can become unavailable while still maintaining replication of vdisk configuration data (VCD). The default depends on the recovery group IBM Spectrum Scale RAID disk topology, and should only be changed under the guidance of IBM Service. This option is not valid for shared or scale-out recovery groups.

--scrub-duration NumberOfDays

With the mmvdisk recoverygroup change command, specifies the background rate at which vdisks should be checked for errors. The value is the number of days (from 1 to 365) that the process should take for the entire target declustered array. The default is 14.

--replace-threshold NumberOfPdisks

With the mmvdisk recoverygroup change command, specifies the number of out-of-service pdisks in the target declustered array at which point the recovery group will report that it needs service for disk replacement. The default is 2 for DAs with more than 12 pdisks, or 1 for DAs with 12 or fewer.

--declustered-array {all | DaName[,DaName...]}

For a scale-out recovery group with the mmvdisk recoverygroup change command, specifies the target declustered arrays for the --spare-pdisks or --spare-nodes parameter.

--spare-pdisks {DEFAULT | N}

For a scale-out recovery group with the mmvdisk recoverygroup change command, specifies the number of effective spares to be reserved in the target declustered arrays. The default depends on the number of nodes in the recovery group and the number of pdisks in the declustered array. The keyword DEFAULT can be used to reset the effective spares to the default for the target declustered arrays.

--spare-nodes {DEFAULT | N}

For a scale-out recovery group with the mmvdisk recoverygroup change command, specifies the number of effective spares in terms of the number of pdisks each node contributes to the target declustered arrays. For example, if each node contributes 3 pdisks to DA1 and 7 pdisks to DA2, mmvdisk recoverygroup change --declustered-array DA1,DA2 --spare-nodes 2 will set 6 effective spares in DA1 and 14 effective spares in DA2, permitting those declustered arrays to completely rebuild from 2 failed nodes. The keyword DEFAULT can be used to reset the effective spares to the default for the target declustered arrays.

--suspend -N Node

For a scale-out recovery group with the mmvdisk recoverygroup change command, specifies a node to be suspended for maintenance. IBM Spectrum Scale is stopped on the node and its pdisks are suspended. Only one node can be suspended at a time.

--window N

With the --suspend option, specify the number of minutes before the data on a suspended server's pdisks is rebuilt onto other nodes. The value of N must be a number between 10 and 60 inclusive. The default is 20 minutes if not specified.

--resume -N Node

For a scale-out recovery group with the mmvdisk recoverygroup change command, specifies a node to be resumed. IBM Spectrum Scale is started on the node and its pdisks are resumed.

--restart
With the mmvdisk recoverygroup change command, restart a recovery group or any inactive log groups within a recovery group. This is for the special case where the recovery group servers are still active but the recovery group itself or log groups within the recovery group have been resigned. If the recovery group and all of its log groups are already active, this command has no effect. If the servers for the recovery group are not active, this command will have no effect; to start a recovery group where the servers are down, use the mmstartup -N nodeClass command, where nodeClass is the mmvdisk server node class for the recovery group.
--refresh-pdisk-info

With the mmvdisk recoverygroup change command, instructs the recovery group to query each pdisk and reread its hardware information.

--version

Specifies that recovery group version information should be listed.

--server

Specifies that recovery group server information should be listed.

--declustered-array

Specifies that recovery group declustered array information should be listed.

--pdisk

Specifies that recovery group pdisk information should be listed.

--log-group

Specifies that recovery group log group information should be listed.

--vdisk-set

Specifies that recovery group vdisk set information should be listed.

--vdisk

Specifies that recovery group vdisk information should be listed.

--fault-tolerance

Specifies that recovery group fault tolerance information should be listed.

--all

Specifies that all sections of recovery group information should be listed.

--not-ok

With the mmvdisk recoverygroup list command, shows those recovery groups that are down, have inactive servers, or have pdisks in need of replacement.

--events

With the mmvdisk recoverygroup list command, specifies that the recovery group event log should be listed.

--days Days

Specifies the number of past days for which recovery group events are to be displayed. The default is to display events as far back as recovery group creation.

--decimal

Specifies that recovery group event timestamps be displayed in decimal format.

--long-term Codes

Specifies that long-term recovery group events that correspond to the Codes string are displayed.

--short-term Codes

Specifies that short-term recovery group events that correspond to the Codes string are displayed.

--confirm

Confirms the deletion of a recovery group. This flag bypasses the interactive prompt that would otherwise be printed to confirm recovery group deletion.

-p

With the mmvdisk recoverygroup delete command, the -p flag indicates that the recovery group is "permanently damaged" and must be deleted without accessing its pdisks. This option is not permitted if the recovery group is active. This is an unusual maintenance case that should only be performed under the direction of IBM Service. Before a recovery group can be deleted using -p, any user vdisks and vdisk sets must be deleted from the recovery group using the mmvdisk vdiskset delete -p command for each vdisk set, followed by undefining the recovery group from each vdisk set using the mmvdisk vdiskset undefine command (note that -p is not necessary to undefine a damaged recovery group from a vdisk set, since the actual member vdisks in the recovery group have been deleted).

--sanitize
Erases data securely from the drives of the deleted recovery group.
-Y

Specifies that the mmvdisk recoverygroup list command produce colon-delimited raw output.

Exit status

0
Successful completion.
nonzero
A failure has occurred.

Security

You must have root authority to run the mmvdisk recoverygroup command.

The node on which the command is issued must be able to execute remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages.

Example

To create paired recovery groups ESS01L and ESS01R using the mmvdisk node class ESS01:

mmvdisk recoverygroup create --node-class ESS01 --recovery-group ESS01L,ESS01R

The system displays output similar to the following:


mmvdisk: Checking node class configuration.
mmvdisk: Checking daemon status on node 'ess01io1.gpfs.net'.
mmvdisk: Checking daemon status on node 'ess01io2.gpfs.net'.
mmvdisk: Analyzing disk topology for node 'ess01io1.gpfs.net'.
mmvdisk: Analyzing disk topology for node 'ess01io2.gpfs.net'.
mmvdisk: Creating recovery group 'ESS01L'.
mmvdisk: Creating recovery group 'ESS01R'.
mmvdisk: Formatting log vdisks for recovery groups.
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LOGTIP
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LOGTIPBACKUP
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG002LOGTIP
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG002LOGTIPBACKUP
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG002LOGHOME
mmvdisk: Created recovery groups 'ESS01L' and 'ESS01R'.

Example

To list all recovery groups present in the cluster:

mmvdisk recoverygroup list

The system displays output similar to the following:

                                                   
                                                            needs    user
recovery group  active   current or master server          service  vdisks  remarks
--------------  -------  --------------------------------  -------  ------  -------
ESS01L          yes      ess01io1.gpfs.net                 no            2
ESS01R          yes      ess01io2.gpfs.net                 no            2

Example

To list the declustered array information for recovery group BB01L:

mmvdisk recoverygroup list --recovery-group BB01L --declustered-array

The system displays output similar to the following:


                    declustered                     current       allowable
 recovery group       arrays     vdisks  pdisks  format version format version
 -----------------  -----------  ------  ------  -------------- --------------
 BB01L                       1       7      49   5.1.2.0        5.1.2.0


declustered    needs                            replace                             scrub       background activity
    array     service  vdisks  pdisks  spares  threshold  BER    trim  free space  duration  task   progress  priority
 -----------  -------  ------  ------  ------  ---------  ------ ----  ----------  --------  -------------------------
 DA1          yes           7      49    2,27          2  enable no       338 TiB    4 days  scrub       74%  low
 

Example

To convert existing recovery groups ESS01L and ESS01R with a new mmvdisk node class ESS01:

mmvdisk recoverygroup convert --recovery-group ESS01L,ESS01R --node-class ESS01 --inherit-config no

The system displays output similar to the following:

mmvdisk: This command will permanently change the GNR configuration
mmvdisk: attributes and disable the legacy GNR command set for the
mmvdisk: servers and recovery groups involved, and their subsequent
mmvdisk: administration must be performed with the mmvdisk command.

mmvdisk: Do you wish to continue (yes or no)? yes

mmvdisk: Converting recovery groups 'ESS01L' and 'ESS01R'.
mmvdisk: Creating node class 'ESS01'.
mmvdisk: Adding 'ess01io1' to node class 'ESS01'.
mmvdisk: Adding 'ess01io2' to node class 'ESS01'.
mmvdisk: Associating recovery group 'ESS01L' with node class 'ESS01'.
mmvdisk: Associating recovery group 'ESS01R' with node class 'ESS01'.
mmvdisk: Recording pre-conversion cluster configuration in /var/mmfs/tmp/mmvdisk.convert.ESS01L.ESS01R.before.m24
mmvdisk: Updating server configuration attributes.
mmvdisk: Checking resources for specified nodes.
mmvdisk: Setting configuration for node class 'ESS01'.
mmvdisk: Defining vdisk set 'VS001_fs3' with recovery group 'ESS01L' (vdisk 'ESS01Lv3fs3').
mmvdisk: Defining vdisk set 'VS002_fs4' with recovery group 'ESS01L' (vdisk 'ESS01Lv4fs4').
mmvdisk: Defining vdisk set 'VS003_fs1' with recovery group 'ESS01L' (vdisk 'ESS01Lv1fs1data').
mmvdisk: Defining vdisk set 'VS004_fs2' with recovery group 'ESS01L' (vdisk 'ESS01Lv2fs2').
mmvdisk: Defining vdisk set 'VS005_fs1' with recovery group 'ESS01L' (vdisk 'ESS01Lv1fs1meta').
mmvdisk: Adding recovery group 'ESS01R' to vdisk set 'VS003_fs1' (vdisk 'ESS01Rv1fs1data').
mmvdisk: Adding recovery group 'ESS01R' to vdisk set 'VS004_fs2' (vdisk 'ESS01Rv2fs2').
mmvdisk: Adding recovery group 'ESS01R' to vdisk set 'VS001_fs3' (vdisk 'ESS01Rv3fs3').
mmvdisk: Adding recovery group 'ESS01R' to vdisk set 'VS005_fs1' (vdisk 'ESS01Rv1fs1meta').
mmvdisk: Adding recovery group 'ESS01R' to vdisk set 'VS002_fs4' (vdisk 'ESS01Rv4fs4').
mmvdisk: Committing cluster configuration changes.
mmvdisk: Recording post-conversion cluster configuration in /var/mmfs/tmp/mmvdisk.convert.ESS01L.ESS01R.after.m24

mmvdisk: For configuration changes to take effect, GPFS should be restarted
mmvdisk: on node class 'ESS01'.

Example

To create a scale-out recovery group RG01 using the mmvdisk node class NC01 that contains servers s01, s02, s03, s04, and s05:

mmvdisk recoverygroup create --node-class NC01 --recovery-group RG01

The system displays output similar to the following:

mmvdisk: Checking node class configuration.
mmvdisk: Checking daemon status on node 's04'.
mmvdisk: Checking daemon status on node 's02'.
mmvdisk: Checking daemon status on node 's01'.
mmvdisk: Checking daemon status on node 's05'.
mmvdisk: Checking daemon status on node 's03'.
mmvdisk: Analyzing disk topology for node 's04'.
mmvdisk: Analyzing disk topology for node 's02'.
mmvdisk: Analyzing disk topology for node 's01'.
mmvdisk: Analyzing disk topology for node 's05'.
mmvdisk: Analyzing disk topology for node 's03'.
mmvdisk: Creating recovery group 'RG01'.
mmvdisk: Formatting log vdisks for recovery group.
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001ROOTLOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG001LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG002LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG003LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG004LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG005LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG006LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG007LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG008LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG009LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG010LOGHOME
mmvdisk: Created recovery group 'RG01'.

Example

To perform the first step in adding server node s06 to scale-out recovery group RG01:

mmvdisk recoverygroup add --recovery-group RG01 -N s06

The system displays output similar to the following:

mmvdisk: Checking daemon status on node 's06'.
mmvdisk: Checking resources for specified nodes.
mmvdisk: Adding 's06' to node class 'NC01'.
mmvdisk: Analyzing disk topology for node 's04'.
mmvdisk: Analyzing disk topology for node 's02'.
mmvdisk: Analyzing disk topology for node 's01'.
mmvdisk: Analyzing disk topology for node 's05'.
mmvdisk: Analyzing disk topology for node 's03'.
mmvdisk: Analyzing disk topology for node 's06'.
mmvdisk: Updating server list for recovery group 'RG01'.
mmvdisk: Updating pdisk list for recovery group 'RG01'.
mmvdisk: The following pdisks will be formatted on node s04:
mmvdisk:     //s06/dev/sdg
mmvdisk:     //s06/dev/sdh
mmvdisk:     //s06/dev/sdf
mmvdisk:     //s06/dev/sde
mmvdisk:     //s06/dev/sdc
mmvdisk:     //s06/dev/sdd
mmvdisk:     //s06/dev/sdb
mmvdisk: Updating parameters for declustered array 'DA1'.
mmvdisk: Updating parameters for declustered array 'DA2'.
mmvdisk: Node 's06' added to recovery group 'RG01'.
mmvdisk: Log group and vdisk set operations for recovery group 'RG01'
mmvdisk: will do until rebalance completes in all declustered arrays.
mmvdisk: A callback 'RG001CompletNodeAdd' has been created to monitor the rebalance state.
mmvdisk: Once rebalance completes, log group and vdisk set will be created automatically.

Example

To complete the addition of node s06 to scale-out recovery group RG01:

mmvdisk recoverygroup add --recovery-group RG01 --complete-node-add

The system displays output similar to the following:

mmvdisk: Verifying that the DAs in recovery group 'RG01' are idle.
mmvdisk: Updating log vdisks for recovery group 'RG01'.
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG011LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG012LOGHOME
mmvdisk: Updating vdisk sets for recovery group 'RG01.'
mmvdisk: 2 vdisks and 2 NSDs will be created in vdisk set 'vs01'.
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG011VS001
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG012VS001
mmvdisk: Created all vdisks in vdisk set 'vs01'.
mmvdisk: (mmcrnsd) Processing disk RG001LG011VS001
mmvdisk: (mmcrnsd) Processing disk RG001LG012VS001
mmvdisk: Created all NSDs in vdisk set 'vs01'.
mmvdisk: Extending file system 'fs01'.
mmvdisk: The following disks of fs01 will be formatted on node s01:
mmvdisk:     RG001LG011VS001: size 1365 GB
mmvdisk:     RG001LG012VS001: size 1365 GB
mmvdisk: Extending Allocation Map
mmvdisk: Checking Allocation Map for storage pool system
mmvdisk: Completed adding disks to file system fs01.

Example

To replace server s06 of recovery group RG01 with new server s07:

mmvdisk recoverygroup replace --recovery-group RG01 -N s06 --new-node s07

The system displays output similar to the following:

mmvdisk: Checking daemon status on node 's07'.
mmvdisk: Analyzing disk topology for node 's04'.
mmvdisk: Analyzing disk topology for node 's02'.
mmvdisk: Analyzing disk topology for node 's01'.
mmvdisk: Analyzing disk topology for node 's05'.
mmvdisk: Analyzing disk topology for node 's03'.
mmvdisk: Analyzing disk topology for node 's07'.
mmvdisk: Adding 's07' to node class 'NC01'.
mmvdisk: Checking resources for specified nodes.
mmvdisk: Updating server list for recovery group 'RG01'.
mmvdisk: Updating pdisk list for recovery group 'RG01'.
mmvdisk: This could take a long time.
mmvdisk: Adding node 's07' pdisks to recovery group 'RG01'.
mmvdisk: The following pdisks will be formatted on node s04:
mmvdisk:     //s07/dev/sdc
mmvdisk:     //s07/dev/sde
mmvdisk:     //s07/dev/sdg
mmvdisk:     //s07/dev/sdb
mmvdisk:     //s07/dev/sdd
mmvdisk:     //s07/dev/sdf
mmvdisk:     //s07/dev/sdh
mmvdisk: Deleting node 's06' pdisks from recovery group 'RG01'.
mmvdisk: Removing node 's06' from node class 'NC01'.
mmvdisk: Updating server list for recovery group 'RG01'.

Example

To list all declustered arrays in all recovery groups in a concise format:

mmvdisk recoverygroup list --declustered-array

The system displays output similar to the following:

                declustered   needs                                             capacity             pdisks  
recovery group     array     service  type   BER      trim   total raw free raw free%  total spare  background task
--------------  -----------  -------  ----   -----    ----   --------- -------- -----  ----- -----  ---------------
BB01L           DA1          no       HDD    enable   no     421 TiB   336 TiB   79%     49     2   scrub (84%)
BB01R           DA1          no       HDD    enable   no     448 TiB   358 TiB   80%     52     2   scrub (49%)

Example

To move all log groups in the shared recovery group ESS3000 to the server canister01, leaving the server canister02 idle:

mmvdisk recoverygroup change --recovery-group ESS3000 --active canister01

The system displays output similar to the following:

mmvdisk: Waiting up to 5 minutes for log groups to change servers.

 node
number  server                            active   remarks
------  --------------------------------  -------  -------
     1  canister01                        yes      serving ESS3000: root, LG001, LG002, LG003, LG004
     2  canister02                        yes      configured 

Example

To see how a shared recovery group running on only one of the two servers is reported as an error condition:

mmvdisk recoverygroup list --not-ok

The system displays output similar to the following:

recovery group  remarks
--------------  -------
ESS3000         server canister01 has 4 user log groups
                server canister02 has 0 user log groups

Example

To restart recovery group ECERG1:

mmvdisk recoverygroup change --recovery-group ECERG1 --restart

The system displays output similar to the following:

mmvdisk: Waiting up to 5 minutes for recovery group 'ECERG1' to restart.

 node
number  server                            active   remarks
------  --------------------------------  -------  -------
     1  server1                           yes      serving ECERG1: LG003, LG006
     2  server2                           yes      serving ECERG1: LG002, LG008
     3  server3                           yes      serving ECERG1: LG005, LG011
     4  server4                           yes      serving ECERG1: LG001, LG007
     5  server5                           yes      serving ECERG1: LG004, LG010
     6  server6                           yes      serving ECERG1: root, LG009, LG012

mmvdisk: Recovery group 'ECERG1' has been restarted.

See also

Location

/usr/lpp/mmfs/bin