Steps to follow if the GPFS daemon does not come up

This topic describes the steps that you need to follow if the GPFS daemon does not come up after installation of IBM Storage Scale.

  1. See GPFS modules cannot be loaded on Linux if your node is running Linux®, to verify that you have built the portability layer.
  2. Verify that the GPFS daemon is active by issuing:
    ps -e | grep mmfsd
    The output of this command should list mmfsd as operational. For example:
    12230 pts/8 00:00:00 mmfsd
    If the output does not show this, the GPFS daemon needs to be started with the mmstartup command.
  3. If you did not specify the autoload option on the mmcrcluster or the mmchconfig command, you need to manually start the daemon by issuing the mmstartup command.

    If you specified the autoload option, someone may have issued the mmshutdown command. In this case, issue the mmstartup command. When using autoload for the first time, mmstartup must be run manually. The autoload takes effect on the next reboot.

  4. Verify that the network upon which your GPFS cluster depends is up by issuing:
    ping nodename
    to each node in the cluster. A properly working network and node correctly replies to the ping with no lost packets.
    Query the network interface that GPFS is using with:
    netstat -i
    A properly working network reports no transmission errors.
  5. Verify that the GPFS cluster configuration data is available by looking in the GPFS log. If you see the message:
    6027-1592
    Unable to retrieve GPFS cluster files from node nodeName.
    Determine the problem with accessing node nodeName and correct it.
  6. Verify that the GPFS environment is properly initialized by issuing these commands and ensuring that the output is as expected.
    • Issue the mmlscluster command to list the cluster configuration. This command also updates the GPFS configuration data on the node. Correct any reported errors before continuing.
    • List all file systems that were created in this cluster. For an AIX® node, issue:
      lsfs -v mmfs
      For a Linux node, issue:
      cat /etc/fstab | grep gpfs

    If any of these commands produce unexpected results, this may be an indication of corrupted GPFS cluster configuration data file information. Follow the procedures in Information to be collected before contacting the IBM Support Center, and then contact the IBM Support Center.

  7. GPFS requires a quorum of nodes to be active before any file system operations can be honored. This requirement guarantees that a valid single token management domain exists for each GPFS file system. Prior to the existence of a quorum, most requests are rejected with a message indicating that quorum does not exist.
    To identify which nodes in the cluster have daemons up or down, issue:
    mmgetstate -L -a

    If insufficient nodes are active to achieve quorum, go to any nodes not listed as active and perform problem determination steps on these nodes. A quorum node indicates that it is part of a quorum by writing an mmfsd ready message to the GPFS log. Remember that your system may have quorum nodes and non-quorum nodes, and only quorum nodes are counted to achieve the quorum.

  8. This step applies only to AIX nodes. Verify that GPFS kernel extension is not having problems with its shared segment by invoking:
    cat /var/adm/ras/mmfs.log.latest
    
    Messages such as:
    6027-319
    Could not create shared segment.
    must be corrected by the following procedure:
    1. Issue the mmshutdown command.
    2. Remove the shared segment in an AIX environment:
      1. Issue the mmshutdown command.
      2. Issue the mmfsadm cleanup command.
    3. If you are still unable to resolve the problem, reboot the node.
  9. If the previous GPFS daemon was brought down and you are trying to start a new daemon but are unable to, this is an indication that the original daemon did not completely go away. Go to that node and check the state of GPFS. Stopping and restarting GPFS or rebooting this node often returns GPFS to normal operation. If this fails, follow the procedures in Additional information to collect for GPFS daemon crashes, and then contact the IBM Support Center.