mmrestripefs command
Rebalances or restores the replication factor of all the files in a file system. Alternatively, this command performs any incomplete or deferred file compression or decompression of all the files in a file system.
Synopsis
mmrestripefs Device {-m | -r | -b [--strict] | -R | -p | -z | --check-conflicting-replicas}
[-N {Node[,Node...] | NodeFile | NodeClass}] [-o InodeResultFile]
[-P PoolName] [--inode-criteria CriteriaFile]
[--pit-continues-on-error] [--qos QosClass]
or
mmrestripefs Device {-r | -b [--strict] | -R | --check-conflicting-replicas}
[-N {Node[,Node...] | NodeFile | NodeClass}] [-o InodeResultFile]
[--inode-criteria CriteriaFile] --metadata-only
[--pit-continues-on-error] [--qos QosClass]
Availability
Available on all IBM Spectrum Scale editions.
Description
Alternatively, you can issue the mmrestripefs command to perform any deferred or incomplete file compression or decompression in all the files of a file system.
You must specify one of the options (-m, -r, -b, -R, -p, -z, or --check-conflicting-replicas) to indicate how much file system data to move or whether to perform file compression or decompression. You can issue this command against a mounted or unmounted file system.
If the file system uses replication, then restriping the file system also replicates it. Also, if the file system uses replication the -r option and the -m options treat suspended disks differently. The -r option removes all data from a suspended disk. But the -m option leaves data on a suspended disk if at least one replica of the data remains on a disk that is not suspended.
The -b option performs all the operations of the -m and -r options.
Use the -z option to perform any deferred or incomplete file compression or decompression.
Consider the necessity of restriping and the current demands on the system. New data that is added to the file system is correctly striped. Restriping a large file system requires many insert and delete operations and might affect system performance. Plan to perform this task when system demand is low.
Parameters
- Device
- The device name of the file system to be restriped. File system names need not be fully
qualified. Device must be the first parameter. It can take the following parameters:
- -m
- Migrates all critical data off any suspended disk in this file system. Critical data is all data that would be lost if currently suspended disks were removed.
- -r
- Migrates all data off suspended disks. It also restores all replicated files in the file system
to their designated degree of replication when a previous disk failure or removal of a disk makes
some replica data inaccessible. Use this parameter either immediately after a disk failure to
protect replicated data against a subsequent failure, or before you take a disk offline for
maintenance to protect replicated data against failure of another disk during the maintenance
process.Note: If the file system uses replication, before running mmrestripefs Device -r, you should run mmlsdisk Device -L to check the number of failure groups available. If the number of failure groups available is less than your default replication, you should not run mmrestripefs Device -r because it will remove the data replica for files that have replicas that are located on suspended or to be emptied disks.
- -b [--strict]
- Rebalances the file system to improve performance. Rebalancing attempts to distribute file
blocks evenly across the disks of the file system.
In
IBM Spectrum
Scale 5.0.0 and later, rebalancing is implemented
by a lenient round-robin method that typically runs faster than the previous method of strict round
robin. To rebalance the file system with the strict round-robin method, include the
--strict option that is described in the following text.
- --strict
- Rebalances the file system with a strict round-robin method. In IBM Spectrum Scale 4.2.3 and earlier versions, rebalancing always uses this method.
Note: Rebalancing of files is an I/O intensive and time-consuming operation and is important only for file systems with large files that are mostly invariant. In many cases, normal file update and creation rebalance a file system over time without the cost of a complete rebalancing.Note: Rebalancing distributes file blocks across all the disks in the cluster that are not suspended, including stopped disks. For stopped disks, rebalancing does not allow read operations and allocates data blocks without writing them to the disk. When the disk is restarted and replicated data is copied onto it, the file system completes the write operations. - -R
- Changes the replication settings of each file, directory, and system metadata object so that
they match the default file system settings (see the mmchfs command
-m and -r options) as long as the maximum
(-M and -R) settings for the object allow it.
Next, it replicates or unreplicates the object as needed to match the new settings. This option can
be used to replicate all of the existing files that were not previously replicated or to unreplicate
the files if replication is no longer needed or wanted. All data is also migrated off
disks that have either a
suspended
orto be emptied
status. - --check-conflicting-replicas
- Scans the file system and compares replicas of metadata and data for conflicts. Each such
conflict is reported.Attention: If this option reports inconsistent replicas, contact IBM® Service for guidance.
- -c [--read-only]
- Deprecated in favor of the --check-conflicting-replicas option. Both
the -c option and the -c --read-only option now
have exactly same function as the --check-conflicting-replicas option. The
function of -c in earlier releases, which was to attempt to fix replica
conflicts, is no longer available. For more information, see the description of the option
--check-conflicting-replicas.
- --metadata-only
- Limits the specified operation to metadata blocks. Data blocks are not affected. This option is valid only with the -r, -b, -R, or --check-conflicting-replicas option.
- -p
- Directs mmrestripefs to repair the file placement within the storage
pool.
Files that are assigned to one storage pool, but with data in a different pool, have their data migrated to the correct pool. Such files are referred to as ill-placed. Utilities, such as the mmchattr command, might change a file's storage pool assignment, but not move the data. The mmrestripefs command might then be invoked to migrate all of the data at once, rather than migrating each file individually. The placement option (-p) rebalances only the files that it moves. In contrast, the rebalance operation (-b) performs data placement on all files.
- -z
- Performs any deferred or incomplete file compression or decompression of files in the file system. For more information, see the topic File compression.
- -P PoolName
- Directs mmrestripefs to repair only the files that are assigned to the
specified storage pool. This option is convenient for migrating ill-placed data blocks between
pools, for example after you change a file's storage pool assignment with
mmchattr or mmapplypolicy with the
-I defer flag.
Do not use for other tasks, in particular, for any tasks that require metadata processing, such as re-replication. By design, all GPFS metadata is kept in the system pool, even for files that have blocks in other storage pools. Therefore, a command that must process all metadata must not be restricted to a specific storage pool.
- -N {Node[,Node...] | NodeFile | NodeClass}
- Specify the nodes that participate in the restripe of the file system. This command supports all
defined node classes. The default is all or the current value of the
defaultHelperNodes parameter of the mmchconfig
command.
For general information on how to specify node names, see Specifying nodes as input to GPFS commands.
- -o InodeResultFile
- Contains a list of the inodes that met the interesting inode flags that were specified on the
--inode-criteria parameter. The output file contains the following:
- INODE_NUMBER
- This is the inode number.
- DISKADDR
- Specifies a dummy address for later tsfindinode use.
- SNAPSHOT_ID
- This is the snapshot ID.
- ISGLOBAL_SNAPSHOT
- Indicates whether or not the inode is in a global snapshot. Files in the live file system are considered to be in a global snapshot.
- INDEPENDENT_FSETID
- Indicates the independent fileset to which the inode belongs.
- MEMO (INODE_FLAGS FILE_TYPE [ERROR])
- Indicates the inode flag and file type that will be printed:Inode flags:
BROKEN exposed dataUpdateMiss illCompressed illPlaced illReplicated metaUpdateMiss unbalanced
File types:BLK_DEV CHAR_DEV DIRECTORY FIFO LINK LOGFILE REGULAR_FILE RESERVED SOCK *UNLINKED* *DELETED*
Notes:- An error message will be printed in the output file if an error is encountered when repairing the inode.
- DISKADDR, ISGLOBAL_SNAPSHOT, and FSET_ID work with the tsfindinode tool (/usr/lpp/mmfs/bin/tsfindinode) to find the file name for each inode. tsfindinode uses the output file to retrieve the file name for each interesting inode.
- --inode-criteria CriteriaFile
- Specifies the
interesting inode criteria flag, where CriteriaFile
contains a list of the following flags with one per line:
- BROKEN
- Indicates that a file has a data block with all of its replicas on disks that have been
removed.Note: BROKEN is always included in the list of flags even if it is not specified.
- dataUpdateMiss
- Indicates that at least one data block was not updated successfully on all replicas.
- exposed
- Indicates an inode with an exposed risk; that is, the file has data where all replicas are on suspended disks. This could cause data to be lost if the suspended disks have failed or been removed.
- illCompressed
- Indicates an inode in which file compression or decompression is deferred, or in which a compressed file is partly decompressed to allow the file to be written into or memory-mapped.
- illPlaced
- Indicates an inode with some data blocks that might be stored in an incorrect storage pool.
- illReplicated
- Indicates that the file has a data block that does not meet the setting for the replica.
- metaUpdateMiss
- Indicates that there is at least one metadata block that has not been successfully updated to all replicas.
- unbalanced
- Indicates that the file has a data block that is not well balanced across all the disks in all failure groups.
Note: If a file matches any of the specified interesting flags, all of its interesting flags (even those not specified) will be displayed. - --pit-continues-on-error
- Allows the mmrestripefs command to continue repairing the remaining files, if
errors are encountered in the parallel inode traverse (PIT) phase that performs scanning of user
file metadata. An output file is generated if an error occurs in the PIT phase. The location of the
file that logs the errors is displayed in the command output. Note:
- The mmrestripefs command continues to run only if the error is reported in the PIT phase of the command execution. If the error is reported in other phases of command execution, the command stops running.
- The --pit-continues-on-error option works only if the minimum release level of the IBM Spectrum Scale cluster is 5.1.1 or later.
- --qos QosClass
- Specifies the Quality of Service for I/O operations (QoS) class to which the instance of the command
is assigned. If you do not specify this parameter, the instance of the command is assigned by
default to the maintenance
QoS class. This parameter has no effect unless the QoS service is enabled. For more information, see the topic mmchqos command. Specify one of the following QoS classes:
- maintenance
- This QoS class is typically configured to have a smaller share of file system IOPS. Use this class for I/O-intensive, potentially long-running GPFS commands, so that they contribute less to reducing overall file system performance.
- other
- This QoS class is typically configured to have a larger share of file system IOPS. Use this class for administration commands that are not I/O-intensive.
Exit status
- 0
- Successful completion.
- nonzero
- A failure has occurred.
Security
You must have root authority to issue the mmrestripefs command.
The node on which you issue the command must be able to execute remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages. For more information, see Requirements for administering a GPFS file system.
Examples
- To move all critical data from any suspended disk in file system fs1,
issue the following command:
The system displays information similar to the following output:mmrestripefs fs1 -m
GPFS: 6027-589 Scanning file system metadata, phase 1 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 2 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 3 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 4 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-565 Scanning user file metadata ... 8.00 % complete on Tue Feb 24 16:56:55 2009 ( 708608 inodes 346 MB) 100.00 % complete on Tue Feb 24 16:56:56 2009 GPFS: 6027-552 Scan completed successfully.
- To rebalance all files in file system fs1 across all defined,
accessible disks that are not stopped or suspended, issue the following command:
The system displays information similar to the following output:mmrestripefs fs1 -b
GPFS: 6027-589 Scanning file system metadata, phase 1 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 2 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 3 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 4 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-565 Scanning user file metadata ... 3.00 % complete on Tue Feb 24 16:56:39 2009 ( 180224 inodes 161 MB) 100.00 % complete on Tue Feb 24 16:56:44 2009 GPFS: 6027-552 Scan completed successfully.
- The following command scans file system
gpfs1 for replica conflicts of metadata and
data:
#mmrestripefs gpfs1 --check-conflicting-replicas Scanning file system metadata, phase 1 ... Inode 0 in fileset 0 and snapshot 0 has mismatch in replicated disk address 2:104859136 Scan completed successfully. Scanning file system metadata, phase 2 ... Scan completed successfully. Scanning file system metadata, phase 3 ... Scan completed successfully. Scanning file system metadata, phase 4 ... Scan completed successfully. Scanning user file metadata ... 100.00 % complete on Tue Jul 30 03:32:44 2013 Scan completed successfully.
- To fix the pool placement of files in file system fs1 and also
determine which files are illReplicated (for example, as a result of a failed disk), issue the
following
command:
The system displays information similar to the following output:mmrestripefs fs1 -p --inode-criteria /tmp/crit -o /tmp/inodeResultFile
GPFS: 6027-589 Scanning file system metadata, phase 1 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 2 ... Scanning file system metadata for data storage pool GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 3 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 4 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-565 Scanning user file metadata ... 100.00 % complete on Wed Apr 15 10:15:15 2015 (65792 inodes with total 400 MB data processed) GPFS: 6027-552 Scan completed successfully. GPFS: 6027-3902 Check file '/tmp/inodeResultFile' on vmip1 for inodes that were \ found matching the criteria. #10:15:15# vmip1:/fs1 # cat /tmp/crit illReplicated #10:15:19# vmip1:/fs1 # cat /tmp/inodeResultFile This inode list was generated in the Parallel Inode Traverse on Wed Apr 15 10:15:14 2015 INODE_NUMBER DISKADDR SNAPSHOT_ID ISGLOBAL_SNAPSHOT FSET_ID MEMO(INODE_FLAGS FILE_TYPE [ERROR]) 24320 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE 24322 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE 24321 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE 24324 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE 24325 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE 24323 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE 24326 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE 24327 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE 24328 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE 24329 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE
- The following command moves all critical data from any suspended disk
in the file system gpfs0, and also determines which files with data are failed
to migrate (for example, as a result of a failed disk
I/O):
# mmrestripefs gpfs0 -m
The system displays information similar to the following output:GPFS: 6027-589 Scanning file system metadata, phase 1 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 2 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 3 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-589 Scanning file system metadata, phase 4 ... GPFS: 6027-552 Scan completed successfully. GPFS: 6027-565 Scanning user file metadata ... 83.11 % complete on Mon Feb 22 02:45:04 2021 ( 200640 inodes with total 7423 MB data processed) 100.00 % complete on Mon Feb 22 02:45:24 2021 ( 501760 inodes with total 10858 MB data processed) GPFS: 6027-566 Error processing user file metadata. No space left on device Check the file /var/mmfs/tmp/gpfs0.pit.interestingInodes.8589934594 on scale-002 for inodes with broken disk addresses or failures. mmrestripefs: tsrestripefs failed.
See also
Location
/usr/lpp/mmfs/bin