File system fails to mount

There are indications leading you to the conclusion that your file system does not mount and courses of action you can take to correct the problem.

Some of those indications include:
  • On performing a manual mount of the file system, you get errors from either the operating system or GPFS.
  • If the file system was created with the option of an automatic mount, you have failure return codes in the GPFS log.
  • Your application cannot access the data it needs. Check the GPFS log for messages.
  • Return codes or error messages from the mmmount command.
  • The mmlsmount command indicates that the file system is not mounted on certain nodes.
If your file system does not mount, then follow these steps:
  1. On a quorum node in the cluster that owns the file system, verify that quorum has been achieved. Check the GPFS log to see if an mmfsd ready message has been logged, and that no errors were reported on this or other nodes.
  2. Verify that a conflicting command is not running. This applies only to the cluster that owns the file system. However, other clusters would be prevented from mounting the file system if a conflicting command is running in the cluster that owns the file system.

    For example, a mount command may not be issued while the mmfsck command is running. The mount command may not be issued until the conflicting command completes. Note that interrupting the mmfsck command is not a solution because the file system is not mountable until the command completes. Try again after the conflicting command has completed.

  3. Verify that sufficient disks are available to access the file system by issuing the mmlsdisk command. GPFS requires a minimum number of disks to find a current copy of the core metadata. If sufficient disks cannot be accessed, the mount fails. The corrective action is to fix the path to the disk. See NSD and underlying disk subsystem failures.

    Missing disks can also cause GPFS to be unable to find critical metadata structures. The output of the mmlsdisk command shows any unavailable disks. If you have not specified metadata replication, the failure of one disk may result in your file system being unable to mount. If you have specified metadata replication, it requires two disks in different failure groups to disable the entire file system. If there are down disks, issue the mmchdisk start command to restart them and retry the mount.

    For a remote file system, mmlsdisk provides information about the disks of the file system. However mmchdisk must be run from the cluster that owns the file system.

    If there are no disks down, you can also look locally for error log reports, and follow the problem determination and repair actions specified in your storage system vendor problem determination guide. If the disk has failed, follow the procedures in NSD and underlying disk subsystem failures.

  4. Verify that communication paths to the other nodes are available. The lack of communication paths between all nodes in the cluster may impede contact with the file system manager.
  5. Verify that the file system is not already mounted. Issue the mount command.
  6. Verify that the GPFS daemon on the file system manager is available. Run the mmlsmgr command to determine which node is currently assigned as the file system manager. Run a trivial data access command such as an ls on the mount point directory. If the command fails, see GPFS daemon went down.
  7. Check to see if the mount point directory exists and that there is an entry for the file system in the /etc/fstab file (for Linux®) or /etc/filesystems file (for AIX®). The device name for a file system mount point is listed in column one of the /etc/fstab entry or as a dev= attribute in the /etc/filesystems stanza entry. A corresponding device name must also appear in the /dev file system.

    If any of these elements are missing, an update to the configuration information may not have been propagated to this node. Issue the mmrefresh command to rebuild the configuration information on the node and reissue the mmmount command.

    Do not add GPFS file system information to /etc/filesystems (for AIX) or /etc/fstab (for Linux) directly. If after running mmrefresh -f the file system information is still missing from /etc/filesystems (for AIX) or /etc/fstab (for Linux), follow the procedures in Information to be collected before contacting the IBM Support Center, and then contact the IBM Support Center.

  8. Check the number of file systems that are already mounted. There is a maximum number of 256 mounted file systems for a GPFS cluster. Remote file systems are included in this number.
  9. If you issue mmchfs -V compat, it enables backwardly-compatible format changes only. Nodes in remote clusters that were able to mount the file system before it is still able to do so.

    If you issue mmchfs -V full, it enables all new functions that require different on-disk data structures. Nodes in remote clusters running an older GPFS version is no longer able to mount the file system. If there are any nodes running an older GPFS version that have the file system mounted at the time this command is issued, the mmchfs command fails. For more information, see Completing the upgrade to a new level of IBM Storage Scale.

    All nodes that access the file system must be upgraded to the same level of GPFS. Check for the possibility that one or more of the nodes was accidently left out of an effort to upgrade a multi-node system to a new GPFS release. If you need to return to the earlier level of GPFS, you must re-create the file system from the backup medium and restore the content in order to access it.

  10. If DMAPI is enabled for the file system, ensure that a data management application is started and has set a disposition for the mount event. Refer to the IBM Storage Scale: Command and Programming Reference Guide and the user's guide from your data management vendor. The data management application must be started in the cluster that owns the file system. If the application is not started, then other clusters are not able to mount the file system. Remote mounts of DMAPI managed file systems may take much longer to complete than those not managed by DMAPI.
  11. Issue the mmlsfs -A command to check whether the automatic mount option has been specified. If automatic mount option is expected, check the GPFS log in the cluster that owns and serves the file system, for progress reports indicating: 
    starting ...
mounting ...
mounted ....
  12. If quotas are enabled, check if there was an error while reading quota files. See MMFS_QUOTA.
  13. Verify the maxblocksize configuration parameter on all clusters involved. If maxblocksize is less than the block size of the local or remote file system you are attempting to mount, you cannot mount it.
  14. If the file system has encryption rules, see Mount failure for a file system with encryption rules.
  15. To mount a file system on a remote cluster, ensure that the cluster that owns and serves the file system and the remote cluster have proper authorization in place. The authorization between clusters is set up with the mmauth command.
    Authorization errors on AIX are similar to the following:
    c13c1apv6.gpfs.net:  Failed to open remotefs.
c13c1apv6.gpfs.net:  Permission denied
c13c1apv6.gpfs.net:  Cannot mount /dev/remotefs on /gpfs/remotefs: Permission denied
    Authorization errors on Linux are similar to the following:
    mount: /dev/remotefs is write-protected, mounting read-only
mount: cannot mount /dev/remotefs read-only
mmmount: 6027-1639 Command failed. Examine previous error messages to determine cause.

    For more information about mounting a file system that is owned and served by another GPFS cluster, see Mounting a remote GPFS file system.