Network Shared Disk server and disk failure

The three most common reasons why data becomes unavailable are disk failure, disk server failure with no redundancy, and failure of a path to the disk.

In the event of a disk failure in which GPFS can no longer read or write to the disk, GPFS will discontinue use of the disk until it returns to an available state. You can guard against loss of data availability from disk failure by:
  • Utilizing hardware data protection as provided by a Redundant Array of Independent Disks (RAID) device (see Figure 1)
    Figure 1. An example of a highly available SAN configuration for a GPFS file system
    This graphic depicts a two-node cluster where the nodes are SAN-attached to two Fibre Channel switches, which are twin-tailed to a RAID/ESS controller.
  • Utilizing the GPFS data and metadata replication features (see Increased data availability) along with the designation of failure groups (see Network Shared Disk (NSD) creation considerations and Figure 2)
    Figure 2. Configuration using GPFS replication for improved availability
    This graphic depicts a two-node GPFS cluster where the nodes are SAN-attached to two Fibre Channel switches, which are attached to two storage controllers. GPFS replication keeps a copy of the data on each storage controller.
It is suggested that you consider RAID as the first level of redundancy for your data and add GPFS replication if you desire additional protection.

In the event of an NSD server failure in which a GPFS client can no longer contact the node that provides remote access to a disk, GPFS discontinues the use of the disk. You can guard against loss of an NSD server availability by using common disk connectivity on multiple NSD server nodes and specifying multiple Network Shared Disk servers for each common disk.

Note: In the event that a path to a disk fails, GPFS reports a disk failure and marks the disk down. To bring the disk back online, first follow the directions supplied by your storage vendor to determine and repair the failure.

Guarding against loss of data availability due to path failure

You can guard against loss of data availability from failure of a path to a disk by doing the following:
  • Creating multiple NSD servers for each disk

    As GPFS determines the available connections to disks in the file system, it is recommended that you always define more than one NSD server for each disk. GPFS allows you to define up to eight NSD servers for each NSD. In a SAN configuration where NSD servers have also been defined, if the physical connection is broken, GPFS dynamically switches to the next available NSD server (as defined on the server list) and continues to provide data. When GPFS discovers that the path has been repaired, it moves back to local disk access. This is the default behavior, which can be changed by designating file system mount options. For example, if you never want a node to use the NSD server path to a disk, even if the local path fails, you can set the -o useNSDserver mount option to never. You can set the mount option using the mmchfs, mmmount, mmremotefs, and mount commands.

    Important: In Linux® on Z, it is mandatory to have multiple paths to one SCSI disk (LUN) to avoid single path of failure. The coalescing of the paths to one disk is done by the kernel (via the device-mapper component). As soon as the paths are coalesced, a new logical, multipathed device is created, which is used for any further (administration) task. (The single paths can no longer be used.)
    The multipath device interface name depends on the distribution and is configurable:
    SUSE
    /dev/mapper/Unique_WW_Identifier

    For example: /dev/mapper/36005076303ffc56200000000000010cc

    Red Hat®
    /dev/mapper/mpath*
    To obtain information about a multipathed device, use the multipath tool as shown in the following example:
    # multipath -ll

    The system displays output similar to this:

    36005076303ffc56200000000000010cc dm-0 IBM,2107900
[size=5.0G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=2][active]
 \_ 1:0:0:0 sdb   8:16  [active][ready]
 \_ 0:0:0:0 sda   8:0   [active][ready]

    See the question, What considerations are there when setting up DM-MP multipath service in the IBM Storage Scale FAQ in IBM® Documentation.

  • Using an I/O driver that provides multiple paths to the disks for failover purposes

    Failover is a path-management algorithm that improves the reliability and availability of a device because the system automatically detects when one I/O path fails and reroutes I/O through an alternate path.