Overview

The Networked File System (NFS) is one of a category of filesystems known as distributed filesystems. It allows users to access files resident on remote systems without even knowing that a network is involved and thus allows filesystems to be shared among computers. These remote systems could be located in the same room or could be miles away.

In order to access such files, two things must happen. First, the remote system must make the files available to other systems on the network. Second, these files must be mounted on the local system to be able to access them. The mounting process makes the remote files appear as if they are resident on the local system. The system that makes its files available to others on the network is called a server, and the system that uses a remote file is called a client.

NFS is implemented as a set of Remote Procedure Call (RPC) by the client.

Back to top

Remote Procedure Call (RPC) -- backbone of NFS

RPC is a library of procedures. The procedures allow the client process to direct the server process to execute procedure calls as if the client process had executed the calls in its own address space. Since the client and the server are two separate processes, they need not exist on the same physical system. Because the server and client processes can reside on two different systems which may have completely different architectures, RPC must address the possibility that the two systems may not represent data in the same format. So, RPC uses data types defined by the eXternal Data Representation (XDR) protocol.

Prior to AIX 4.2.1, UDP (User Datagram Protocol) was the exclusive transport protocol for NFS packets. TCP (Transmission Control Protocol) was added as an alternative protocol in AIX 4.2.1. UDP works well over clean networks and responsive servers. For wide area networks or for busy networks or networks with slower servers, TCP may provide better performance because its inherent flow control can minimize retransmits on the network.

Also, prior to AIX 4.3, UDP was the default transport protocol for both NFS Version 2 and 3. With AIX 4.3 and later, TCP is the default protocol for NFS Version 3, but using a new mount option (proto) the client can select TCP or UDP. For example:

# mount -o proto=tcp

For reasons of speed and simplicity UDP is the preferred transport protocol for NFS. However, although it is fast, it is unreliable. The network does not guarantee that the packets transmitted via UDP will ever be delivered, or that they will be delivered in order. NFS works around this problem by requiring the NFS server to acknowledge every RPC command with a result code that indicates whether the command was successfully completed or not. If the NFS client does not get an acknowledgement within a certain amount of time, it retransmits the original command.

What are the implications of this? If the client does not receive an acknowledgement, then UDP lost either the original RPC command or the RPC acknowledgement. If the original RPC command was lost the server sees it for the first time when it is retransmitted. But if the acknowledgement was lost, then the server will actually get the same RPC command twice.

For most NFS commands, this duplication of requests poses no problem. With read , for example, it does not matter if the same block of data is read once or a dozen times. The other commands, however, cannot be executed twice in a row. For example, mkdir will fail the second time. For commands of this kind, some NFS servers maintain a cache of the last 400 commands that were executed. When the server receives a mkdir request, it first checks the cache to see if it already has received the mkdir request. If so, the server merely retransmits the acknowledgement. If NFS client still receives no acknowledgement, it will retransmit again and again, each time doubling the retry time. If the network file system was mounted with the soft option, the request will eventually time out. With hard option, the client continues sending the request until it is rebooted or gets an acknowledgement. (More about these options later).

Back to top

NFS servers -- connectionless and stateless

By design, NFS servers are connectionless and stateless. Connectionless means that the server program does not keep track of every client that has remotely mounted the filesystem. Stateless means that all of the information that the client needs to mount a remote filesystem is kept on the client. Once a file handle is issued for a file, that file handle will remain good even if the server is shutdown and rebooted, as long as the file continues to exist and no major changes are made to the configuration of the server.

The advantage of a connectionless, stateless system is that clients can continue using a network file server even if that server crashes and restarts, because there is no connection that must be re-established, and all of the state information associated with the remote mount is kept on the clients, i.e. no information is lost when an NFS server crashes. Clients can simply wait until the server returns and then continue as if nothing had happened. If the client should crash or if the network should become disconnected, valuable resources are not tied up on the server maintaining a connection and state for that client.

A second advantage of this approach is that it scales up. It works equally well whether ten servers are using a filesystem or ten thousand servers are using it.

Although system performance suffers under extremely heavy use, every file request made by a client will eventually be satisfied, and there is absolutely no performance penalty if a client mounts a filesystem but never uses it.

Back to top

Have your own NFS environment

NFS consists of a number of components including a mounting protocol, a file locking protocol, an export file and daemons (mountd, nfsd, biod, rpc.lockd, rpc.stad) that coordinate basic file services.

The export file - /etc/exports

Systems using NFS make the files available to other systems on the network by "exporting" their directories to the network. An NFS server exports its directories by putting the names of these directories in the /etc/exports file and executing the exportfs command. In its simplest form, /etc/exports consists of lines of the form:

pathname -option, option ...

where pathname is the name of the file or directory to which network access is to be allowed; if pathname is a directory, then all of the files and directories below it within the same filesystem are also exported, but not any filesystems mounted within it. The next fields in the entry consist of various options that specify the type of access to be given and to whom. For example, a typical /etc/exports file may look like this:

/cyclop/users	  -access=homer:bart, root=homer
/usr/share/man	  -access=marge:maggie:lisa
/usr/mail

This export file permits the filesystem /cyclops/users to be mounted by homer and bart, and allows root access to it from homer (as well as from cyclop, the real owner of the filesystem). In addition, it lets /usr/share/man be mounted by marge, maggie and lisa. The filesystem /usr/mail can be mounted by any system on the network. Filesystems listed in the export file without a specific set of hosts are mountable by all machines. This can be a sizable security hole.

When used with the -a option, the exportfs command reads the /etc/exports file and exports all the directories listed to the network. This is usually done at system startup time.

# /usr/sbin/exportfs -a

If the contents of /etc/exports change, you must tell mountd to reread it. This can be done by re-executing the exportfs command after the export file is changed.

The exact attributes that can be specified in the /etc/exports file vary from system to system. The most common attributes are:

For example:

/cyclop/users	-rw=moe,anon=-1
/usr/inorganic	-ro

This allows moe to mount /cyclop/users for reading and writing, and maps anonymous users - usernames from other hosts that do not exist on the local system and the root user from any remote system - to the UID -1. This corresponds to the nobody account, and it tells NFS not to allow such users access to anything. On some systems, the UID -2 may be used to allow anonymous users access only to world-readable files. The read-only access is referred to as read-mostly.

Server programs and daemons

The NFS server programs mountd, nfsd, biod, rpc.lockd, rpc.statd and portmap perform the actual network management of the directories. These must be running at all times for NFS to work. Although you can start them manually whenever you want to have your system run NFS, in AIX they are included in the /etc/rc.nfs which is launched from the startup file called /etc/inittab.

$ cat /etc/rc.nfs
.......... [Deleted].........
dspmsg cmdnfs.cat -s 8 1 "Starting NFS services:\n"
trap 'dspmsg cmdnfs.cat -s 8 3 "Completed NFS services.\n"' 0
if [ -x /usr/sbin/biod ]; then
	start biod /usr/sbin/biod 8
fi

#
# If nfs daemon is executable and /etc/exports, become nfs server.
#
if [ -x /usr/sbin/nfsd -a -f /etc/exports ]; then
	> /etc/xtab
	/usr/sbin/exportfs -a
	start nfsd /usr/sbin/nfsd 8
	start rpc.mountd /usr/sbin/rpc.mountd
fi

#
# start up status monitor and locking daemon if present
#
if [ -x /usr/sbin/rpc.statd ]; then
	start rpc.statd /usr/sbin/rpc.statd
fi

if [ -x /usr/sbin/rpc.lockd ]; then
	start rpc.lockd /usr/sbin/rpc.lockd
fi

..............[Deleted]........

## Begin AutoFS startup   ## Do NOT remove
##
if [ -s /etc/auto_master ]; then 
	/usr/sbin/automount
fi
##
## End AutoFS startup   ## Do NOT remove
$

mountd daemon

After the files, directories and/or filesystems have been exported, an NFS client must explicitly mount them before it can use them. It is handled by the mountd daemon (sometimes called rpc.mountd). The server examines the mount request to be sure the client has proper authorization, then gives the client a secret "magic cookie" (actually a random number) that the client can use later to obtain access. The magic cookie scheme keeps the server stateless. Cookies persist across a reboot so that a server that crashes can return to its previous state. Usually, unmounting and remounting a filesystem on the server changes the cookie.

Once a client has a magic cookie, it uses RPC to make requests for filesystem operations such as creating a file or reading a data block. Because NFS is stateless, the server does not care what requests the client has or hasn't made before. In particular, the client is responsible for making sure that the server acknowledges write requests before it deletes its own copy of the data to be written.

The following syntax is used for the mount command. Note that the server's name is followed by a colon and the directory to be mounted:

#  mount  -t  nfs  server1:/usr/src  /src
#

Here, the directory structure /usr/src resident on the remote system server1 is mounted on the /src directory on the local system. The -t nfs specifies that the file system type is a remote NFS directory.

When the remote filesystem is no longer needed, it is unmounted with the umount

#  umount  server1:/usr/src
#

mount and df both provide information about the local and remote file systems:

The showmount command lists exported filesystems (using its -e option) or other hosts that have remotely mounted local systems (-a). The example below shows that the hosts server1 and server2 have mounted the sourcefiles and datafiles filesystems respectively:

#showmount  -a
server1:/sourcefiles
Server2:/datafiles

The mount command can be used to establish temporary network mounts, but mounts that are part of a system's permanent configuration should be either listed in /etc/filesystems (for AIX) or handled by an automatic mounting service such as automount or amd.

mount Command -- the options

The following options are available for use with the mount command:

The soft and hard options are worth special mention. They define the action taken when a remote filesystem becomes unavailable. If a remote filesystem is mounted as hard, NFS will try to complete any pending I/O request forever, even after the maximum number of retransmissions is reached; if it is mounted soft, then an error will occur and NFS will cancel the request.

If a remote filesystem is mounted hard and intr is not specified, then the process will block (be hung) until the remote filesystem reappears. For a terminal process especially, this can be quite annoying. If intr is specified, then sending an interrupt signal to the process will kill it. For a terminal, this can be done by typing CTRL-C (although it won't die instantly; you'll still have to wait for the timeout period). For a background process, sending an INT (2), or QUIT (3) signal will usually work (again not necessarily instantaneously).

Sending a KILL signal (-9) will not kill a hung process.

It would seem that mounting filesystems soft would get around the hanging problem. This is fine for filesystems mounted read-only. However, for a read-write filesystem, a pending request could be a write request, and so simply giving up could result in corrupted files on the remote filesystem. Therefore, read-write remote filesystems should always be mounted hard, and the intr option should be specifed to allow users to make their own decisions about hung processes.

Filesystems mounted hard can cause processes to hang when their servers go down. This is particularly bothersome when the processes in question are standard daemons. In general, the use of the soft and intr options will reduce the number of NFS-related headaches. However, these options can have their own undesirable side-effects, such as aborting a 20-hour simulation after it has run for 18 hours just because of a transient network glitch. The automount daemon amd also provides some remedies for mounting ailments.

NFS uses 32-byte packets, called file handles, to control access to files. When a client mounts a server's filesystem, the server passes back to the client a file handle for the root directory of the filesystem. The file handle for the directory can be used for any of the following functions:

A file handle for a file can be used for any of these functions:

nfsd: server-side daemon

Once a client's mount request has been validated by mountd, it is allowed to request various filesystem operations. These requests are handled on the server side by nfsd, the NFS operations daemon. (In reality, nfsd is essentially a one-line program that makes a non-returning system call to NFS server code embedded in the kernel. It is disguised as a user process only because this makes the scheduling issues easier to implement.) nfsd need not be run on an NFS client machine unless the client also exports filesystems of its own.

Beginning with AIX 4.2.1, a single nfsd and a single biod daemon manage the NFS services. Each of these daemons is multithreaded (multiple kernel threads within the process). Also, the number of threads is self-tuning in that it creates or deletes threads as needed. However, prior to AIX 4.2.1, nfsd and biod both required tuning. Each took one argument which specified the number of copies of itself to fork. Selecting the appropriate number of nfsds is important and, unfortunately, is something of a black art. If the number is too low or too high, NFS performance can suffer.

In AIX by default this number is 8. In general, 4 is adequate for a server that is used infrequently and is few enough so as not to impact on performance. In theory, you could start tens or hundreds of nfsd. However, too many nfsds will degrade performance because they will fight with each other for the CPU.

The maximum number of nfsds is closely tied to the number of hardware contexts supported by the computer's CPU chip. In today's RISC market the number of contexts on a chip may change with each version of the processor family, but it is usually in the range 8 to 32. If you can find out the exact number of contexts on your CPU (not always easy without the hardware manual for the chip), a good rule of thumb is to start no more than N-2 nfsds, where N is the number of contexts.

If you cannot determine the exact number of on-board contexts, you can make a good guess by trial and error. Increase the number of nfsds until the load average of the server (as reported by uptime) begins to increase significantly. When this happens, it's an indication that your system has more nfsds than it knows what to do with. Back off a few from that number, and you should be safe. Remember, this is the maximum number of nfsds, not the best number.

On a loaded NFS server, UDP sockets can overflow if requests arrive while all nfsds "piers" are already in use. The number of overflows can be monitored with netstat -s.

The first thing you should do when figuring out how many nfsds to run is to add more nfsds until UDP socket overflows drop to zero, or you reach the maximum number of nfsds computed above.

biod: client-side Daemon

In order to improve overall NFS performance, most systems include the biod daemon which does basic read-ahead and write-behind filesystem block caching. For example, when an NFS client requests three bytes from a file, a much larger chunk (usually 4K) is actually read. When the client reads the next three bytes, no network transaction needs to occur. It is strongly recommended to run this daemon on all NFS clients, but it is not strictly required.

Like nfsd, biod takes as its argument the number of copies of itself to start. The rules for nfsd apply to biod for choosing the right number. If both nfsd and biod run on the same machine, it may be wise to split the optimal number of process slots between them. You'll have to experiment.

Want to change the number of nfsd and biod?

To change the number of biod or nfsd daemons running on the system:

chnfs -n 10 -b 4

This sets the number of nfsd daemons to 10 and the number of biod daemons to 4. This command temporarily stops the daemons currently running on the system, modifies the SRC database code to reflect the new number, and restarts the daemons.

Manage NFS locks: rpc.lockd and rpc.statd

While lockd and statd are two distinct daemons, but they always run as a team. lockd is responsible for maintaining advisory locks on NFS files. It manages file locking on both server and client systems. statd allows processes to monitor the status of other machines that are running NFS. It handles lock crash and recovery services for both client and server. It is used by lockd to decide when to attempt to communicate with a remote machine.

When an application wants to obtain a lock on a local file, it sends its request to the kernel via the lockf, fcntl, or flockl subroutine. The kernel then processes the lock request. However, if an application on an NFS client makes a lock request for a remote file, the kernel forwards the request to the client's rpc.lockd daemon via an RPC routine. When the client's rpc.lockd daemon receives a remote lock request, it registers its interest in that server with the client's rpc.statd daemon. When the client's rpc.statd daemon notifies the client's rpc.lockd daemon that the server is operating, the client's rpc.lockd daemon sends its request (via an RPC routine) to the server's rpc.lockd daemon. The server's rpc.lockd daemon then requests the lock from its kernel and sends the reply back to the client.

portmap daemon

Each RPC application has associated with it a program number and a version number. These numbers are used to communicate with a server application on a system. The client, when making a request from a server, needs to know what port number that server is accepting requests on. This port number is associated with the UDP or TCP protocol that is being used by the service. The client knows the program number, the version number, and the system name or host name where the service resides. The client needs a way to map the program number and version number pair to the port number of the server application. This is done with the help of the portmap daemon.

How to start/stop NFS daemons and how to get their current status

In the following discussion, reference to daemon implies any one of the SRC-controlled daemons (such as nfsd or biod).

The NFS daemons can be automatically started at system (re)start by including the /etc/rc.nfs script in the /etc/inittab file.

They can also be started manually by executing the following command:

startsrc -s Daemon or startsrc -g nfs

where -s option will start the individual daemons and -g will start all of them.

These daemons can be stopped one at a time or all at once by executing the following command:

stopsrc -s Daemon or stopsrc -g nfs

You can get the current status of these daemons by executing the following commands:

lssrc -s Daemon or lssrc -a

If the /etc/exports file does not exist, the nfsd and the rpc.mountd daemons will not start. You can get around this by creating an empty/etc/exports file. This will allow the nfsd and the rpc.mountd daemons to start, although no filesystems will be exported.

Administrative conventions for NFS

Any user who wants to access a remote file on the network should have a login on the machine where the file actually resides. The login doesn't have to have a real shell associated with it. Furthermore, both UIDs and GIDs must be unique across all machines that share filesystems. To forestall administrative headaches, it's best if UIDs and GIDs are unique across the network cluster.

Back to top

Interactions between NFS and networks

Traditional NFS uses UDP as its underlying transport protocol. Although NFS does its own packet sequence reassembly and error checking, UDP and NFS both lack the congestion control algorithms that are essential for good performance on a large IP network. For this reason, one should avoid mounting traditional NFS partitions through a router, over long-haul (WAN) lines, or over the Internet. Although these are common practices at many sites, they are really bad ideas.

The solution is to use TCP as the underlying transport layer on mounts that cross a router or long-haul line.

Back to top

Automatic mounting

Mounting filesystems one at a time by listing them in /etc/filesystems introduces a number of problems in large networks. First, maintaining these files on a few hundred machines can be tedious. Each one may be slightly different and thus require individual attention.

Second, if filesystems are mounted from a dozen or more hosts, chaos ensues when one of those servers crashes, since every command that stats the mount points will hang.

Third, when an important server crashes, it may cripple users by making important partitions like /usr/share/man unavailable. In this situation, it's best if a copy of the partition can be mounted temporarily from a backup server.

An automount daemon mounts filesystems when they are referenced and unmounts them when they are no longer needed. This minimizes the number of active mount points and is mostly transparent to users. With most automounters, it is also possible to supply a list of replicated (identical) filesystems so that the network can continue to function when a primary server becomes unavailable.

To implement this behind-the-scenes mounting and unmounting, the automounter poses as an NFS server. But instead of mirroring an actual filesystem onto the network, the automounter makes up a filesystem hierarchy according to the specifications listed in the configuration file.

When a user references a directory within the automounter's fake fantasy filesystem, the automounter intercepts the reference, mounts the actual filesystem the user is trying to reach, and forwards the operation along. There is usually a single directory where filesystems actually get mounted, and symbolic links are used to create the illusion of mounts occurring throughout the system.

Unfortunately, automount is plagued with bugs and design flaws and is an unworthy opponent to the freely-available alternative, amd, written by Jan-Simon Pendry of Imperial College, London. It expands upon the idea behind automount and corrects many of automount's serious flaws and can easily be installed on a variety of Unix systems.

Back to top

Conclusion

In any system environment that consists of a number of filesystems, you can use NFS to distribute the filesystems and lessen the load on any one system. Doing this can improve the overall performance of the systems. As mentioned before, you have to be careful in creating the export file so that access to files is not accidentally extended to unintended audience. Also, instead of having filesystems mounted for the life of a process, you should also take a look at amd which keeps the filesystems mounted only as long as they are needed. This has the potential of significantly improving system performance.

Resources

• Managing NFS and NIS by Hal Stern, O'Reilly & Associates, Inc., ISBN 0-937175-75-7
  
  
• Unix System Administration Handbook by Evi Nemeth et al, Prentice Hall, ISBN 0-13-151051-7
  
  
• AMD: The 4.4BSD Automounter Reference Manual, 4.4BSD System Manager's Manual, by Jan-Simon Pendry and Nick Williams, Usenix and O'Reilly, 1994
  
  
• AIX 5L Performance Management Guide
  
  
• AIX 5L System Management Guide
  
  

About the author

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Overview
• Remote Procedure Call (RPC) -- backbone of NFS
• NFS servers -- connectionless and stateless
• Have your own NFS environment
• Interactions between NFS and networks
• Automatic mounting
• Conclusion
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into AIX and UNIX on developerWorks

Calling all developers

Download no charge Lotus Domino Designer

Special offers