GPFS has declared NSDs as down

GPFS reactions to NSD failures and the recovery procedure.

There are several situations in which disks can appear to fail to GPFS. Almost all of these situations involve a failure of the underlying disk subsystem. The following information describes how GPFS reacts to these failures and how to find the cause.

GPFS will stop using a disk that is determined to have failed. This event is marked as MMFS_DISKFAIL in an error log entry (see Operating system error logs). The state of a disk can be checked by issuing the mmlsdisk command.

The consequences of stopping disk usage depend on what is stored on the disk:
  • Certain data blocks may be unavailable because the data residing on a stopped disk is not replicated.
  • Certain data blocks may be unavailable because the controlling metadata resides on a stopped disk.
  • In conjunction with other disks that have failed, all copies of critical data structures may be unavailable resulting in the unavailability of the entire file system.
The disk will remain unavailable until its status is explicitly changed through the mmchdisk command. After that command is issued, any replicas that exist on the failed disk are updated before the disk is used.
GPFS can declare disks down for a number of reasons:
  • If the first NSD server goes down and additional NSD servers were not assigned, or all of the additional NSD servers are also down and no local device access is available on the node, the disks are marked as stopped.
  • A failure of an underlying disk subsystem may result in a similar marking of disks as stopped.
    1. Issue the mmlsdisk command to verify the status of the disks in the file system.
    2. Issue the mmchdisk command with the -a option to start all stopped disks.
  • Disk failures should be accompanied by error log entries (see The operating system error log facility) for the failing disk. GPFS error log entries labelled MMFS_DISKFAIL will occur on the node detecting the error. This error log entry will contain the identifier of the failed disk. Follow the problem determination and repair actions specified in your disk vendor problem determination guide. After performing problem determination and repair issue the mmchdisk command to bring the disk back up.