mmfsckx command
Checks and repairs corruptions on a mounted IBM Storage Scale file system.
Synopsis
mmfsckx Device
[--repair]
[--check-reserved-files-only ][--max-threads MaxThreads]
[--max-pagepool-percent MaxPagepoolPercent] [--qos QosClass]
[-N {all | mount | Node[,Node...] | NodeFile | NodeClass}]
Availability
Available on all IBM Storage Scale editions.

Description
- MMFS_FSSTRUCT error entries in system log. For more information, see Operating system error logs.
- Corrupted output when querying file system directories.
- Disk startup failure.
- File system mount failure.
- Storage media failure.
Type of Error | Description | Critical |
Reserved file corruption | Reserved files are files internal to GPFS and not visible to users. They contain all file system metadata. | Yes |
Improperly freed blocks. | Blocks that are marked as available in the block allocation map system file but are referenced in some files. | Yes |
Duplicate disk addresses | When two or more disk addresses in the same or different files point to overlapping sectors on the disk. | Yes |
Inconsistent inode allocation status | In-use files are marked as available in the inode allocation map or vice-versa. | No |
Metadata corruption | Includes the file inode, indirect blocks, and extended attribute blocks. | No |
Lost blocks | Blocks that are marked as in-use in the block allocation map system file but do not belong to any file. | No |
- bad directory blocks
- bad directory entries
- fileset identifier validation
- file link counts
- orphaned inodes
- mmchfs
- mmchmgr
- mmadddisk
- mmdeldisk
- mmchdisk
- mmcrsnapshot
- mmdelsnapshot
- mmcrfileset
- mmdelfileset
- mmchfileset
mmlinkfileset
mmunlinkfileset
- mmrestripefs
- mmdefedquota
- mmdefquotaon
- mmrepquota
- mmfsck
- mmcheckquota
Auto inode space expansion
Creation of new ACLs
To run the mmfsckx command, all nodes that mount the file system, including remote mounts must be running IBM Storage Scale 5.2.0.0 or later.
When the mmfsckx command is executing, attempts to mount the file system on nodes running an IBM Storage Scale version older than 5.2.0.0 will be blocked until the mmfsckx command exits.
When run in report mode, the default, not all metadataOnly or dataAndMetadata NSDs need to be available. The metadata on NSDs that are not available do not get validated, and any disk addresses pointing to these unavailable NSDs is considered valid. However, when the mmfsck command runs with the --repair option, metadataOnly and dataAndMetadata NSDs must be available. If unavailable disks are physically damaged, then they must be deleted before running the mmfsckx command.
Before running the mmfsckx command, consider the issues and ongoing demands on the system. For example, running the command on a large file system might affect the system performance. In such cases, plan to run the command when the system demand is low.

Results
If the metadata in the file system is corrupted, the mmfsckx
command displays information about the corruption in the command output along with other information
like scan progress and statistics. All corruption-related information is prefixed with a
!
to
distinguish it from other information.
On completion of the file system scan, the mmfsckx command prints a summary table of all the corrupted inodes that were identified in the scan. The mmfsckx command reports the inode numbers of corrupted files only. Run the mmfindinode command from samples/ilm/ directory in the IBM Storage Scale installation path to convert inode numbers to file paths.

awk ‘/^-/,/^$/{if (/^-|^$/)next; if ($2==0) print $3}’ /tmp/mmfsckx.out > /tmp/inodefile

- A user who uses ctrl+c on the mmfsckx command line.
- An environmental error like disk, network, or insufficient memory.
The node, where the file system manager is located for the file system, that is being checked either fails or panics the file system.
- Any node that is participating in the file system scan fails or panics the file system.


If the mmfsckx command is aborted, the cause for the failure is
included in the mmfsckx output, and logged on the node currently acting as the
file system manager for the file system being checked.
Retrying the mmfsckx command operation restarts the file system scan from the
beginning. It does not resume from the last interrupted scan.
Parameters
- Device
- The device name of the file system to be checked and repaired. File system names do not have to
be fully qualified. fs0 is as acceptable as
/dev/fs0.
This parameter must be the first parameter.
--repair
Specifies that the mmfsckx command repairs corruptions that it finds in the file system.
- --check-reserved-files-only
- Specifies that the mmfsckx command checks only the reserved files in the file
system.Note: Reserved files are special files used internally by IBM Storage Scale for maintaining file system metadata.
- --maxthreads MaxThreads
- The maximum number of parallel threads that scans the file system on a single node. The default
is 32.
- --maxpagepool-percent MaxPagepoolPercent
- The maximum percentage of available page-pool memory on any node that is used by the
mmfsckx command during its scan. The default is 50%.
- --qos QosClass
- Specifies the Quality of Service for I/O operations (QoS) class to which the instance of the command
is assigned. If you do not specify this parameter, the instance of the command is assigned by
default to the maintenance
QoS class. This parameter has no effect unless the QoS service is enabled. For more information, see the topic mmchqos command. Specify one of the following QoS classes:
- maintenance
- This QoS class is typically configured to have a smaller share of file system IOPS. Use this class for I/O-intensive, potentially long-running IBM Storage Scale commands, so that they contribute less to reducing overall file system performance.
- other
- This QoS class is typically configured to have a larger share of file system IOPS. Use this class for administration commands that are not I/O-intensive.
- [-N {all | mount | Node[,Node...] | NodeFile | NodeClass}]
- Specifies the nodes that are to participate in the checking of the file system. Only nodes that
are at IBM Storage Scale 5.1.4 or higher can participate in the
check. The file system manager node always participates in the check even if it is not specified in
this parameter. This command supports all defined node classes. The default is
all or the current value of the
defaultHelperNodes parameter of the mmchconfig
command.
To run the mmfsckx command on a single node, use the mmchmgr command to make the target node the file system manager and then run mmfsckx with
-N targetNode
.For information on how to specify node names, see Specifying nodes as input to GPFS commands.
Exit status
- 0
- Successful completion.
- nonzero
- A failure has occurred.
Security
You must have root authority to run the mmfsckx command.
The node on which the command is issued must be able to run remote shell commands on any other node in the cluster. It must run the commands without the use of a password and without producing any extraneous messages. For more information, see Requirements for administering a GPFS file system.
Examples
- An example output of mmfsckx command run on a file system without any
corruptions.
mmfsckx fs1 2022-01-01_00:13:18+0530: starting mmfsckx mmfsckx /dev/fs1 master node is h1 GPFS version is 5.1.4.0 system Linux x86_64, release 4.18.0-305.19.1.el8_4.x86_64 maintenanceMode is off maxThreads is 32 (default 32) maxPagePoolPercent is 50% (default 50%) nInodesToLock is 1008 (default 1008) lineNumAdjust is 0 File system info features are FastEA; StripedLogs; IExpandV2 allocation map layout is vertical type is cluster replicas metadata is 1 of 2 data is 1 of 2 log is 0 of 2 subblocksPerFullBlock is 32 Sizes sector is 512 B metadata subblock is 2 KB (4 sectors) metadata fullblock is 64 KB (128 sectors) data subblock is 2 KB (4 sectors) data fullblock is 64 KB (128 sectors) inode is 4 KB (8 sectors) indirect block is 8 KB (16 sectors) ea overflow block is 64 KB (128 sectors) directory block is 64 KB (128 sectors) Recovering file system Waiting for async recovery to complete Initializing mounted nodes Creating fsckx snapshot fsckx_2022-01-01_00.13.18 Created fsckx snapshot with snapId 13 Initializing caches on master node Initializing reserved files Checking inode of reserved files Reserved files info snapshots global are 2 fileset are 0 inode0 file inodeNum is 0 subblocksPerFullBlock is 32 recordsPerBlock is 16 inodeSpaceMask is 0x0 inodeBlockMask is 0x1FFF block alloc map file of system pool inodeNum is 1 subblocksPerFullBlock is 32 recordsPerBlock is 16 disks are 2 regions are 773 segments are 1 holdsData yes inode alloc map file inodeNum is 2 subblocksPerFullBlock is 32 recordsPerBlock is 32 type is cluster regions are 257 segments are 1 iallocSpaceMask is 0x0 iallocSegmtMask is 0x0 access control list file inodeNum is 4 subblocksPerFullBlock is 32 recordsPerBlock is 1 extended attribute file inodeNum is 5 subblocksPerFullBlock is 8 recordsPerBlock is 1 fileset metadata file inodeNum is 38 subblocksPerFullBlock is 32 recordsPerBlock is 16 independent are 1 afm are 0 dependent are 0 Selecting worker nodes Selected 1 worker nodes Combined pagepool usage across all scanner nodes Total pagepool: 4.00 GB Stealable pagepool: 3.39 GB Fsckx Usable pagepool: 1.69 GB (50%) Needed pagepool: 26.43 MB (1%) Initializing worker nodes Checking reserved files metadata pass 1 of 1 Initializing caches on worker nodes Checking user files metadata pass 1 of 1 2022-01-01_00:13:22: checked 0 of 65792 inodes 2022-01-01_00:13:25: checked 65792 of 65792 inodes Checking inode allocation map for inodes 0 to 131071 Checking block allocation map of poolId 0 for segment 0 regions 387 to 772 on node h1 Checking block allocation map of poolId 0 for segment 0 regions 0 to 386 on node h2 Deleting fsckx snapshots Deleting fsckx snapshot fsckx_2022-01-01_00.13.18 Inodes: 65792 total (in active file system) 62 used 1 directories 3 user-files 65920 scanned (including snapshots) 0 fatal 0 data-loss 0 inconsistent 0 bad-alloc 0 lost-alloc Disk-sectors: 8388608 total (all disks) 8388608 used 512 directories 274120 user-files 1601040 read from disk (all passes) 0 corrupt 0 duplicate (critical) 0 bad-alloc (critical) 0 lost-alloc Disk-addresses: 16708 valid (including snapshots) 0 corrupt 0 duplicate 0 deleted-disk 0 reserved-file-hole 0 metadata-replica-mismatches File system has no metadata corruptions 2022-01-01_00:13:28+0530: mmfsckx completed after 10 seconds.
- An example output of mmfsckx command run on a file system with
corruptions.
mmfsckx fs1 2022-01-01_00:34:19+0530: starting mmfsckx mmfsckx /dev/fs1 master node is h1 GPFS version is 5.1.4.0 system Linux x86_64, release 4.18.0-305.19.1.el8_4.x86_64 maintenanceMode is off maxThreads is 32 (default 32) maxPagePoolPercent is 50% (default 50%) nInodesToLock is 1008 (default 1008) lineNumAdjust is 0 File system info features are FastEA; StripedLogs; IExpandV2 allocation map layout is vertical type is cluster replicas metadata is 1 of 2 data is 1 of 2 log is 0 of 2 subblocksPerFullBlock is 32 Sizes sector is 512 B metadata subblock is 2 KB (4 sectors) metadata fullblock is 64 KB (128 sectors) data subblock is 2 KB (4 sectors) data fullblock is 64 KB (128 sectors) inode is 4 KB (8 sectors) indirect block is 8 KB (16 sectors) ea overflow block is 64 KB (128 sectors) directory block is 64 KB (128 sectors) Recovering file system Waiting for async recovery to complete Initializing mounted nodes Creating fsckx snapshot fsckx_2022-01-01_00.34.19 Created fsckx snapshot with snapId 19 Initializing caches on master node Initializing reserved files Reserved files info snapshots global are 2 fileset are 0 inode0 file inodeNum is 0 subblocksPerFullBlock is 32 recordsPerBlock is 16 inodeSpaceMask is 0x0 inodeBlockMask is 0x1FFF block alloc map file of system pool inodeNum is 1 subblocksPerFullBlock is 32 recordsPerBlock is 16 disks are 2 regions are 773 segments are 1 holdsData yes inode alloc map file inodeNum is 2 subblocksPerFullBlock is 32 recordsPerBlock is 32 type is cluster regions are 257 segments are 1 iallocSpaceMask is 0x0 iallocSegmtMask is 0x0 access control list file inodeNum is 4 subblocksPerFullBlock is 32 recordsPerBlock is 1 extended attribute file inodeNum is 5 subblocksPerFullBlock is 8 recordsPerBlock is 1 fileset metadata file inodeNum is 38 subblocksPerFullBlock is 32 recordsPerBlock is 16 independent are 1 afm are 0 dependent are 0 Selecting worker nodes Selected 1 worker nodes Combined pagepool usage across all scanner nodes Total pagepool: 4.00 GB Stealable pagepool: 3.39 GB Fsckx Usable pagepool: 1.69 GB (50%) Needed pagepool: 26.43 MB (1%) Initializing worker nodes Checking reserved files metadata pass 1 of 1 Initializing caches on worker nodes Checking user files metadata pass 1 of 1 2022-01-01_00:34:21: checked 0 of 65792 inodes !Inode 8288 snap 0 fset 0 "user file" inode @1:2195712: disk address 1:100 in slot 0 replica 0 pointing to data block 0 code 1928 is corrupt !Inode 8288 snap 0 fset 0 "user file" inode @1:2195712: disk address 1:100 in slot 1 replica 1 pointing to data block 0 code 1928 is corrupt !Inode 8288 snap 0 fset 0 "user file" inode @1:2195712: lastBlockSubblocks 12 should be 0 !Inode 10560 snap 0 inode @1:2204800: deleted inode fileSize 23744 should be 0 !Inode 11136 snap 0 fset 0 "user file" inode @1:2207104: nFullBlocks 0 should be 1 2022-01-01_00:34:25: checked 65792 of 65792 inodes Checking inode allocation map for inodes 0 to 131071 !Inode 10560 allocation state used should be free Checking block allocation map of poolId 0 for segment 0 regions 387 to 772 on node h1 Checking block allocation map of poolId 0 for segment 0 regions 0 to 386 on node h2 !Block 1:131584 has map status: 0x00000000 expected: 0xFFFFFFFF !Block 1:131712 has map status: 0x00000000 expected: 0xFFFFFFFF !Block 1:132096 has map status: 0x000FFFFF expected: 0x00000000 !Block 1:924672 has map status: 0x000FFFFF expected: 0xFFFFFFFF !Block 1:1761408 has map status: 0x00000000 expected: 0x0FFFFF00 !Block 1:1761536 has map status: 0x00000000 expected: 0xFFFFFFFF !Block 1:1761664 has map status: 0x00000000 expected: 0xFFFFFFFF !Block 1:1761792 has map status: 0x00000000 expected: 0xFFFFFFFF !Block 1:1761920 has map status: 0x00000000 expected: 0xFFFFFFFF Deleting fsckx snapshots Deleting fsckx snapshot fsckx_2022-01-01_00.34.19 No. SnapId InodeNum FileType Corruption Severity --- ---------- -------- ------------------------ ------------ ------------ 1 0 8288 user file inconsistent non-critical 2 0 10560 deleted inode fatal non-critical 3 0 11136 user file inconsistent non-critical Inodes: 65792 total (in active file system) 61 used 1 directories 4 user-files 65952 scanned (including snapshots) 1 fatal 0 data-loss 2 inconsistent 0 bad-alloc 1 lost-alloc Disk-sectors: 4194304 total (all disks) 4193328 used 256 directories 101668 user-files 1328736 read from disk (all passes) 8 corrupt 0 duplicate (critical) 80 bad-alloc (critical) 896 lost-alloc Disk-addresses: 8072 valid (including snapshots) 2 corrupt 0 duplicate 0 deleted-disk 0 reserved-file-hole 0 metadata-replica-mismatches File system has critical metadata corruptions File system is NOT safe to use without repair 2022-01-01_00:34:30+0530: mmfsckx completed after 11 seconds
An example output of mmfsckx command to repair a file system with corruptions.
mmfsckx fs1 --repair 2023-09-20_13:02:22-0400: starting mmfsckx mmfsckx /dev/fs1 --repair master node is node3 GPFS version is 5.1.9.0 system Linux x86_64, release 3.10.0-693.el7.x86_64 maintenanceMode is off maxThreads is 32 (default 32) maxPagePoolPercent is 50% (default 50%) nInodesToLock is 1024 (default 1024) lineNumAdjust is 0 File system info format version is 33.00 (5.1.9.0) features are FastEA; StripedLogs; 4KAligned; IExpandV2 allocation map layout is vertical type is scatter replicas metadata is 1 of 2 data is 1 of 2 log is 0 of 2 subblocksPerFullBlock is 512 Sizes sector is 512 B metadata subblock is 8 KB (16 sectors) metadata fullblock is 4 MB (8192 sectors) data subblock is 8 KB (16 sectors) data fullblock is 4 MB (8192 sectors) inode is 4 KB (8 sectors) indirect block is 32 KB (64 sectors) ea overflow block is 64 KB (128 sectors) directory block is 256 KB (512 sectors) Checking mounted nodes Creating fsckx snapshot fsckx_2023-09-20_13.02.22 Created fsckx snapshot with snapId 3 Initializing mounted nodes Initializing reserved files Initializing caches on master node Checking inode of reserved files Reserved files info snapshots global are 1 fileset are 0 inode0 file inodeNum is 0 subblocksPerFullBlock is 512 recordsPerBlock is 1024 inodeSpaceMask is 0x300 inodeBlockMask is 0xFF block alloc map file of system pool inodeNum is 1 subblocksPerFullBlock is 512 recordsPerBlock is 512 disks are 14 regions are 787 segments are 2 holdsData yes inode alloc map file inodeNum is 2 subblocksPerFullBlock is 16 recordsPerBlock is 16 type is cluster regions are 257 segments are 4 iallocSpaceMask is 0x3 iallocSegmtMask is 0x0 access control list file inodeNum is 4 subblocksPerFullBlock is 512 recordsPerBlock is 1 extended attribute file inodeNum is 5 subblocksPerFullBlock is 2 recordsPerBlock is 1 fileset metadata file inodeNum is 38 subblocksPerFullBlock is 512 recordsPerBlock is 1024 independent are 4 afm are 0 dependent are 0 Selecting worker nodes Selected 1 worker nodes node4 Combined pagepool usage across all scanner nodes Total pagepool: 2.00 GB Stealable pagepool: 1.43 GB Fsckx Usable pagepool: 734.68 MB (50%) Needed pagepool: 595.63 MB (41%) Initializing worker nodes Checking reserved files metadata pass 1 of 1 Initializing caches on worker nodes Checking user files metadata pass 1 of 1 2023-09-20_13:02:23: checking 0 of 444416 inodes Checking inode allocation map for inodes 0 to 131071 Checking inode allocation map for inodes 131072 to 262143 !Inode 280540 snap 0 fset 1 "user file" inode @9:7962336: checksum 0x0 should be 0x4AC09003; inode will be deleted Checking inode allocation map for inodes 262144 to 393215 !Inode 280540 allocation state used should be free Checking inode allocation map for inodes 393216 to 524287 Checking inode allocation map for inodes 524288 to 655359 Checking inode allocation map for inodes 655360 to 786431 2023-09-20_13:02:28: checked 444416 of 444416 inodes Checking inode allocation map for inodes 786432 to 886783 Checking block allocation map of poolId 0 for segmentNum 0 regionNums 0 to 393 on node node3 !Block 3:188416 has map status: 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 expected: 0x0001FFFFF8000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 !Block 5:188416 has map status: 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 expected: 0x0000000000000000 0x3FFFFFFFC0000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 Checking block allocation map of poolId 0 for segmentNum 0 regionNums 394 to 786 on node node4 Checking for duplicate and bad alloc references in pass 1 2023-09-20_13:02:35: checking 0 of 444416 inodes Checking inode allocation map for inodes 0 to 131071 !Inode 280544 snap 0 fset 1 "user file" inode @9:7962368: disk address 7:332464 in slot 0 replica 0 pointing to data block 0 code 2136 is a duplicate !Inode 280544 snap 0 fset 1 "user file" inode @9:7962368: lastBlockSubblocks 22 should be 512 Checking inode allocation map for inodes 262144 to 393215 Checking inode allocation map for inodes 524288 to 655359 !Inode 792543 snap 0 fset 3 "user file" inode @1:16678648: disk address 7:332464 in slot 0 replica 0 pointing to data block 0 code 2136 is a duplicate !Inode 792543 snap 0 fset 3 "user file" inode @1:16678648: lastBlockSubblocks 36 should be 512 2023-09-20_13:02:39: checked 444416 of 444416 inodes Checking inode allocation map for inodes 786432 to 886783 Checking block allocation map of poolId 0 for segmentNum 0 regionNums 0 to 393 on node node3 !Block 7:327680 has map status: 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 expected: 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x00000000001FFFFF 0xFFFE000000000000 0x0000000000000000 0x0000000000000000 Checking block allocation map of poolId 0 for segmentNum 0 regionNums 394 to 786 on node node4 Deleting fsckx snapshots Deleting fsckx snapshot fsckx_2023-09-20_13.02.22 No. SnapId InodeNum FileType Corruption Severity --- ------ -------- ------------------------ ------------ ------------ 1 0 280540 user file fatal non-critical 2 0 280544 user file data-loss critical 3 0 792543 user file data-loss critical Inodes: 444416 total (in active file system) 26098 used 40 directories 26019 user-files 890880 scanned (including snapshots) 1 fatal 2 data-loss 0 inconsistent 0 bad-alloc 1 lost-alloc Disk-sectors: 285293992 total (all disks) 15442264 used 3776 directories 7864880 user-files 11486008 read from disk (all passes) 0 bad 352 duplicate 0 bad-alloc 1440 lost-alloc Disk-addresses: 26206 valid (including snapshots) 0 corrupt 2 duplicate (critical) 0 bad-alloc (critical) 0 deleted-disk 0 reserved-file-hole 0 metadata-replica-mismatches File system had critical metadata corruptions that were repaired. 2023-09-20_13:02:45-0400: mmfsckx completed after 23 seconds
See also
Location
/usr/lpp/mmfs/bin