NFS Presentation

Resolving The Problem

This document contains basic information on usage and problem determination for NFS. This document assumes that the user has basic knowledge of TCP/IP configuration and IBM OS/400 commands. The following will be covered in this presentation:

	Software and Hardware Requirements
	NFS Concepts
	Usage
	Basic problem determination
	Document collection for PDPSI and additional References

Hardware and Software Requirements

To use NFS, the OS/400 must be at R370 or higher and TCP/IP already configured and working. It replaces FSS/400 which was available on previous releases. NFS and FSS/400 are not compatible with each other. The OS/400 version of NFS is a ported version of Sun Microsystems Version. 2 product. RFC1094 describes NFS Version 2.

NFS Concepts

NSF is used to make remote objects stored in file systems appear to be local. All users in a network can share a single set of files. NFS accomplishes this by:

Exporting: Allows a central site administrator to make file objects available to remote NFS clients. Only objects which have been exported can be accessed by remote users.

Mounting: This maps a remote directory of file to a local directory called a mount point. The mounted file system will act and perform as if it existed on the local client system.

The OS/400 can function as both an NFS Server and as an NFS Client.

NFS Usage

Export

The OS/400 commands used to make directories and files available for use by NFS clients are CHGNFSEXP and EXPORTFS. These commands allow exporting of the following OS/400 file systems:

	Root (Integrated File System)
	QSYS.LIB
	QDLS (Document Services -- folders)
	QOpenSys (LAN Server)
	QOPT (Optical)
	UDFS

NFS EXPORT OPTIONS parameter includes:

	-A flag which indicates use for the /etc/exports file for the directories and options. All directory trees in this file will get exported as a result. -A is the same as *DFT.
	-I flag which will allow the user override export options in the /etc/exports file and will also allow exporting a specific directory in the directory parameter which may or may not be in the /etc/exports file. The -I flag is used in conjunction with the -O flag. This is a useful parameter for debugging.
	-O flag specifies additional export option keywords which describe export characteristics for the directory being exported. There is a default set of export options when -O is issued by itself.
	-U flag is used for unexporting a directory.

The NFS EXPORT DIRECTORY specifies the directory to export. The syntax of this directory is dependent on the file system being exported.

Other parameters in the CHGNFSEXP and EXPORTFS generally will not be used unless a specific NFS client has special requirements such as a code page change. You must have *IOSYSCFG special authority and *r and *w data authority to the /etc directory.

Below is an example of the EXPORTFS command:

Fig. 1
MOUNT

MOUNT and ADDMFS are commands which make file objects accessible to an NFS client local namespace. For an OS/400 client, these files are accessible to an Integrated File System namespace. For OS/2 or Windows clients, they would be accessible as a local drive name or a local directory. This namespace is considered the mount point.

Most NFS clients use a MOUNT command which has two important parameters:

1	The directory to be mounted: RCHASFAS:/QSYS.LIB/JPM.LIB/TRACE.FILE /QIBM/Prodata /qdls
2	The directory to mount over: /home /project z:\

There are MOUNT options which also control whether a MOUNT will be successful or not. Most clients have a default set of MOUNT options. Below are some screen examples of the above parameters:



Fig. 2



Fig. 3



Other Commands
RMVMFS/UNMOUNT - These are commands which release previously mounted file systems.
DSPMFSINF/STATFS - These commands show information about a mounted file system.

Non-OS/400 systems also have a SHOWMT or SHOWMOUNT command which shows which file systems are available to mount for a given NFS host server. However, PTFs are needed for this support: SF45634 (R370) and SF45631 (R410).

NFS Problem Determination
MOUNT

Most problems will be a file or directory that cannot be mounted. Some of the reasons for not being able to mount a file include:

	The file object specified cannot be found MSGCPFA0A9.
	The server has denied access to the file because of authority issues.
	The client's request has not gotten to the server to be processed or the server has received the request and cannot process it for some reason.

Where Does One Start?

1	Are the six NFS server jobs started? If not, did they start and then end abnormally? Is there a job log associated with a job which ended?
2	Does the object being mounted actually exist? What is this object's authority? What mount options are being used? Is the user using a user ID which maps to a user profile which has authority to the object being mounted?
3	Can the object be mounted from a different system such as the host which is doing the export?
4	Is there a SUNRPC port (111) in *LISTEN status? There should be one port 111 for UDP and one for TCP. There should also be UDP port 2049 in *LISTEN status.

Authority/Security

NFS Server client requests to mount must be mapped to a user profile for file object security. Users can be anonymous, in which
case they will usually be mapped to the profile QNSFANON. An administrator can control what QNFSANON has authority to and
can also specify in the CHGNFSEXP command that users should be mapped to another profile or not be allowed to access objects altogether.

Client users can also be mapped to a GID (Group ID) for file object authority purposes. Client users can also be mapped to unique
UID (individual id) for file object purposes. Client users can specify a GID or UID in their MOUNT options. Server administrators
can specify a GID or UID in their EXPORT options.

Each OS/400 user profile contains a GID and a UID parameter. Therefore, when trying to understand a mount authority problem, it
is imperative to find out what GID, UID or ANON parameters are being used in the EXPORT and MOUNT commands, what profile
is actually being used and what authorities to files object this user profile has.






The following text segments have been taken from the Exploring NFS on AS/400 IBM® Redbooks™ (SG24-2158).Exploring NFS.sg24-5128.pdf

NFS Server Side Daemons

__________________________________________
| NFS Server|NFS Server |Mount Daemon |
| Daemon | Daemon | |
|___________|___________|_________________|
| NFS Server|NFS Server |Network Lock |
| Daemon | Daemon | Manager |
|___________|___________|________________ |
|RPC Binder Daemon | Network Status |
| | Monitor |
|______________________ |________________ |

The NFS Server

NFS is similar to other RPC-based services in its use of server-side daemons to process incoming requests. NFS may also use multiple copies of some daemons to improve overall performance and efficiency.

We discuss the various daemons in the following list:

	RPC Binder Daemon (RPCD): This daemon is analogous to the port mapper daemon that many implementations of NFS use in UNIX. Clients determine the port of a specified RPC service by using the RPC Binder Daemon. Local services register themselves with the local RPC binder daemon (port mapper) when initializing.
	NFS Server Daemons (NFSD): The most pressing need for NFS server daemons centers around the need for multi-threading NFS RPC requests. Running daemons in user-level processes allows the server to have multiple independent threads of processes. In this way, the server can handle several NFS requests at once. As a daemon completes the processing of a request, the daemon returns to the end of a line of daemons that wait for new requests. Multiple instances of this daemon can perform tasks simultaneously.
	Mount Daemon (MNTD): Each NFS server system runs a mount daemon that listens to requests from client systems. This daemon acts on mount and unmount requests from clients. If the mount daemon receives a client mount request, the daemon checks the export table. The mount daemon compares it with the mount request to see if the client is allowed to perform the mount. If the mount is allowed, the mount daemon sends an opaque data structure (the file handle to the requesting client). This structure uniquely describes the mounting point that is requested by the client. This enables the client to represent the root of the mounted file system when making future requests.
	Network Status Monitor Daemon (NSMD): The Network Status Monitor (NSM) is a stateful NFS service that provides applications with information about the status of network hosts. The Network Lock Manager (NLM) daemon heavily uses the NSM to track hosts that have established advisory byte-range locks as well as hosts that maintain such locks. There is a single NSM server per host. It keeps track of the state of clients and notifies any interested party when this state changes (usually after recovery from a crash). The NSM daemon keeps a notify list that contains information on hosts to be informed after a state change. After a local change of state, the NSM notifies each host in the notify list of the new state of the local NSM. When the NSM receives a state change notification from another host, it notifies the local network lock manager daemon of the state change.
	Network Lock Manager Daemon (NLMD): The Network Lock Manager (NLM) daemon is another stateful service that provides advisory byte-range locking for NFS files. The NLM maintains state across requests, and makes use of the Network Status Monitor daemon (NSM) that maintains state across crashes. The NLM supports two types of byte-range locks: Monitored locks: These are reliable and helpful in the event of system failure. When an NLM server crashes and recovers, all the locks it had maintained are reinstated without client intervention. Likewise, NLM servers release all old locks when a client crashes and recovers. A Network Status Manager (NSM) must be functioning on both the client and the server to create monitored locks. Unmonitored locks: These locks require explicit action to be released after a crash and re-established after start up. This is an alternative to monitoring locks, which requires the NSM on both the client and the server systems.

OS/400 System as an NFS Client

Several entities work together to communicate with the server and local jobs on the NFS client. These processes are:

	RPC Binder Daemon: This daemon communicates with the local and remote daemons using the RPC protocol. Client requests for remote file systems are transmitted to the server in this fashion.
	Network Status Monitor and Network Lock Manager: These two daemons are not mandatory on the client. Many client applications, however, establish byte-range locks on parts of remote files on behalf of the client without notifying the user. For this reason, we recommend that the NSM and NLM daemons exist on both the NFS client and server.
	Block I/O daemon: This daemon manages the data caches and is, therefore, stateful in operation. It performs caching and assists in routing client-side NFS requests to the remote NFS server. Multiple instances of this daemon can perform tasks simultaneously.
	Data and attribute caches: These two caches enhance NFS performance by storing information on the client-side to prevent a client/server interaction. The attribute cache stores file and directory attribute information locally on the client while the data cache stores frequently used data on the client.
	Mount and unmount commands: Users can mount and unmount a file system in the client name space with these commands. These are general tools, used not only in NFS, but also to dynamically mount and unmount other local file systems. Refer to Chapter 3, "Operating and Using NFS" in topic 3.0 for more details.

NFS Client Side Daemons
________________________________________________
| Network | Block I/O | Block I/O |
| Lock | Daemon | Daemon |
| Manager | | |
|__________ |________________ | |
| | Block I/O | Block I/O|
| Network | Daemon | Daemon |
| Status | | |
| Manager |_____________________________|
| | |
| | RPC Binder Daemon |
| | |
|_________________________________________|


The NFS Client

Besides the RPC Daemon, the NFS client has only one daemon to process requests and to transfer data from and to the remote server, the block I/O daemon. NFS differs from typical client/server models in that processes on NFS clients make some RPC calls themselves, independently of the client block I/O daemon. An NFS client can optionally use both a Network Lock Manager (NLM) and a Network Status Monitor (NSM) locally, but these daemons are not required for standard operation. We recommend that you use both the NLM and NSM on your client because user applications often establish byte-range locks without the knowledge of the user.

The block I/O daemon handles requests from the client for remote files or operations on the server. Running only on NFS clients or servers that are also clients, this daemon manages the data and attribute caches for the user. The block I/O daemon is stateful and routes client application requests either to the caches or on to the NFS server. All data cached by the block I/O daemon is updated at regular intervals that a user can specify. Users can start multiple daemons to perform different operations simultaneously.

NFS Client Side Caches

Caching file data or attributes gives administrators a way of tuning NFS performance. The caching of information allows you to delay writes or to read ahead. Client-side caching in NFS reduces the number of RPC requests sent to the server. The NFS client can cache data, which can be read out of local memory instead of from a remote disk. The caching scheme available for use depends on the file system being accessed. Some caching schemes are prohibited because they cannot guarantee the integrity and consistency of data that multiple clients simultaneously change and update. The standard NFS cache policies ensure that performance is acceptable while also preventing the introduction of state into the client/server communication relationship. There are two types of client caches: the directory and file attribute cache and the data cache.

Directory and File Attribute Cache: Not all file system operations use the data in files and directories. Many operations get or set the attributes of the file or directory such as its length, owner, and modification time. Because these attribute-only operations are frequent and do not affect the data in a file or directory, they are prime candidates for using cached information.

The client-side file and directory cache store file attributes so that every operation that gets or sets attributes does not have to go through the connection to the NFS server. When a file's attributes are read, they remain valid on the client for some minimum period of time. This time period can be set using the acregmin option on the mount command. If the client modifies the file (therefore updating its attributes), that change is made to the local copy of the attributes and the cache validity period is extended for another minimum time period. The attributes of a file remain static for a maximum period and then they are flushed from the cache and written back to the server if they have been modified. This time period can be set with the acregmax option on the mount command. To force a refresh of remote attributes when opening a file, do not use the nocto option on the mount command. Specifying the noac option suppresses all local caching of attributes, negating the acregmin, acregmax, acdirmin, and the acdirmax options on the mount command. Refer to Chapter 3, "Operating and Using NFS" in topic 3.0 for more details.

The same mechanism is used for directory attributes. The minimum and maximum time period for directory attribute flushing from the cache is set by the acdirmin and acdirmax options on the mount command.

Attribute caching allows a client to make multiple changes to a file or directory without having to constantly get and set attributes on the server. Intermediate attributes are cached, and the sum total of all updates is later written to the server when the maximum attribute cache period expires. Frequently accessed files and directories have their attributes cached locally on the client so that some NFS requests can be performed without having to make an RPC call. By preventing this type of client/server interaction, caching attributes improves the performance of NFS.

Data Cache: The data cache is similar to the directory and file attribute cache in that it stores frequently used information locally on the client. The data cache, however, stores data that is frequently or likely to be used instead of file or directory attributes. The data cache provides data in cases where the client must access the server to retrieve information that has already been read. This operation improves the performance of NFS.

Whenever a user makes a request on a remote object, a request is sent to the server. If the request is to read a small amount of data (for example, one byte (B)), the server returns four kilobytes (KB) of data. This "extra" data is stored in the client caches because, presumably, it will soon be read by the client.

When users access the same data frequently over a given period of time, the client can cache this information to prevent a client/server interaction. This caching also applies to users who use data in one "area" of a file frequently. This is called locality and involves not only the primary data that is retrieved from the server, but also a larger block of data around it. When a user requests data frequently from one area, the entire block of data is retrieved and then cached. There is a high probability that the user will soon want to access this surrounding data. Because this information is already cached locally on the client, the performance of NFS is improved.

Client Time-out: If the client does not have a cache loaded, all requests go to the server. This takes extra time to process each client operation. With the mount command, users can specify a time-out value for re-sending the command. The client cannot distinguish between a slow server and a server that is not operational, so it will retry the command. The default time-out value is two seconds. If the server does not respond in this time, the client continues to retry the command, which overloads the server with client requests. The solution to this difficulty is to increase the time-out value on the mount command.

Document Collection for PDPSI and Additional References

The following documentation should be collected when a problem with NFS is encountered:

	Communications Trace of the line formatted for ASCII and IP addresses.
	Any Client Messages involved.
	A full description of the problem.
	Authority descriptions of objects having access problems.
	If further documentation is needed, the Support Center Rep will work with development on collecting a DEBUG trace.

Further NFS information is available below:

	OS/400 Network File System Support (SC41-4714)
	Exploring NFS on the AS/400 (Redbooks) (SG24-2158)

Currently there is no OS/400 NFS information on the Web, other than mentioning it is part of the TCP/IP product.