This page has not been liked. Updated 5/11/15, 5:35 PM by ScottGPFSTags: None

This section is dedicated to providing information on running NFS servers accessing GPFS file systems.

How do I:

Best Practices when running cNFS

  1. When all cNFS servers have SAN access to a common set of LUNS set useNSDservernever=never on the cNFS servers
    • Setting the useNSDserver=never mount option on the NFS servers keeps them from using NSD Protocol network access to the disk in the event of a SAN failure. It is a good idea to set this mount option on NFS servers because you do not want the NFS servers to use network access to the storage in the event of a SAN failure. cNFS provides availability by failing client access over to another NFS server until the SAN issue can be resolved. Another approach is to not define NSD servers for the NSDs. This works as well but most clusters have a mix of NFS and NSD client access to the data so this is not a practical option. And for another reason see best practice #2. See Managing File Systems to learn how to set the useNSDserver mount option on each node separately.
    • Of course if you have a 2 tier GPFS architecture where the cNFS servers are NSD clients then you cannot set this as the cNFS access the data through the NSD servers. This is okay because in this case the nodes have sufficient network bandwidth to the NSD servers for data access.
  2. Define NSD servers for all the NSDs
    • It is a good idea to define NSD servers for all of the NSD's whenever creating a file system. This may be obvious if you have NSD clients in your cluster but even if you do not have NSD clients it is a good idea. This is because with GPFS 3.4 or earlier NSD servers cannot be configured dynamically and require an unmount of the file system. By configuring NSD servers up front you can easily add NSD or mulit-cluster access to the data without having to unmount the file system.
  3. Ensure the NFS services are not started by default and only managed by mmnfsmonitor
    • For RHEL:
# chkconfig nfs off
# chkconfig nfslock off
# chkconfig rpcbind on

Using multiple interfaces in a cNFS node


You can set up a cNFS server to serve two different subnets at the same time. cNFS understands multiple networks and ensures that the IP address is placed on the correct interface on start-up and during a fail-over event. For example if you have a Linux node with two interfaces for NFS connections:

1Gbit 	eth1
10Gbit 	eth2

You define the networks using mmchnode command

mmchnode \--cnfs-interface="," --N node1

The GPFS cNFS code checks to see what interface (eth2 for example) an IP address should be assigned to. There are two ways to assign addresses for cNFS, you can use the device IP address or an alias.

Using the Device IP

If you are using the device IP address then eth1 and eth2 are already configured in /etc/sysconfig. In this case cNFS uses that information to bind the IPs to the correct interface.

Using an Alias

If you are using aliases, then eth1 and eth2 must already be configured (with different IPs) in /etc/sysconfig and cNFS uses the netmask to figure out which interfaces to use (for example, if eth1 is and eth2 is, then the IPs will map to eth1 and eth2 respectively).


Configure the IP addresses used for cNFS file serving?



It is recommended that you use virtual IP addresses (an alias) instead of the interface address when configuring you IP address space for a CNFS cluster.

The advanatges of using Virual IP addresses include:

  • The node does not need to wait for the switch to accept the link when an IP address is filaed back. Since the address is an alias the interface on which it resides is already up.
  • IP address fail over is much faster
  • It simplifies administration by providing a clear distinction between the "system ip" and the "nfs ip". For example in a two node cluster if one of the nodes has a problem that induces fail over and someone logs into the suspected node to reboot it. If the system address has been migrated to the surviving node, the surviving node may get rebooted by accident.

How to use an alias

To use an alias with cNFS you need to provide a static IP address that is not already defined as an alias (in the /etc/sysconfig/network-scripts directory). CNFS sets up the alias when it starts the interface So to set up an alias:

  1. Define a staic IP address for the device
  2. Ensure that there are not aliases defined in the network-scripts directory for this interface
    # ls -l /etc/sysconfig/network-scripts/ifcfg-eth1:*
    ls: /etc/sysconfig/network-scripts/ifcfg-eth1:*: No such file or directory
  3. Assign the alias IP as the cNFS interface
    mmchnode --cnfs-interface="" -N node1


Debug cNFS problems that are causing reboots?


So why are my cnfs nodes rebooting? There are many reasons this could be happening:

  • Nodes are out of CPU cycles
  • Network problems

By default when cNFS detects a failure (NFS, GPFS, OS etc) it reboots the server to attempt to recover from the failure. Sometimes to debug a node it may be useful to temporarily disable this feature. You can disable the automatic reboot by setting the cnfsreboot parameter.

mmchconfig cnfsreboot=no -N problem_node

It is recommended that reboot only be disabled when you are debugging a problem and you want to see the node in the failed state. Disabling reboot can cause IP failover to fail under some conditions.


Display the quota values of a fileset for NFS clients?


GPFS supports the ability to set hard and soft quotas on a fileset. It is common to create a fileset in a GPFS file system that is then used as an export for NFS clients. When exporting a GPFS fileset you can configure GPFS such that when a NFS client issues the df command against the NFS mounted fileset they see the fileset hard quota level as the "Available" size instead of the space available in the entire file system. This feature is called filesetdf and is a GPFS file system configuration parameter. This allows you to utilize GPFS storage pools to share space amongst many different projects.

There are two steps required to enable the filesetdf feature for a file system.

1. Enable quotas for the file system and set a hard fileset quota

2. Enable filesetdf for the file system.

To enable filesetdf for a filesystem use the mmchfs command. For example:

mmchfs gpfs0 --filesetdf

To disable filesetdf

mmchfs gpfs0 --nofilesetdf

The filesetdf setting can be changed while the file system is online.

This feature is available starting in GPFS 3.2.1-16 and 3.3.0-2.