Disk subsystem configuration for NFS performance

One of the most common sources of bottlenecks in read/write-intensive workloads is poorly configured disk subsystems.

While you might consider tuning only the disk subsystem on the NFS server, note that a poorly configured disk setup on the NFS client might be the actual problem in certain scenarios. An example of this is a workload in which a file is copied by an application on the NFS client from an NFS-mounted filesystem to a local filesystem on the client. In this case, it is important that the disk subsystem on the client is properly tuned such that write performance to the local filesystem does not become the bottleneck. See the tuning techniques described in Logical volume and disk I/O performance. In particular, consider the following:

For a simple read or write workload over NFS, evaluate the performance capabilities of the disks which contain the file systems being used. You can do this by writing to or reading from a file locally on the file system. You should use the iostat command to measure the throughput capability of the disks since many test applications might complete without actually writing all the data to disk. For example, some data might still be in memory. You can then typically consider this throughput measured on local reads/writes as the upper bound on the performance you will be able to achieve over NFS, since you are not incurring the additional processing and latency overhead associated with NFS.
It is often necessary to achieve high parallelism on data access. Concurrent access to a single file system on a server by multiple clients or multiple client processes can result in throughput bottlenecks on the disk I/O for a specific device. You can use the iostat command to evaluate disk loading. In particular, the %tm_act parameter indicates the percentage of time that a particular disk was active, but a high value can also indicate that the associated disk adapter is overloaded.
While not directly relevant to tuning of the disk subsystem, it is worth mentioning that concurrent writes to a single file can result in contention on the inode lock of the file. Most file systems use an inode lock to serialize access to the file and thus ensure the consistency of the data being written to it. Unfortunately, this can severely hamper write performance in the case where multiple threads are attempting to write to the same file concurrently since only the thread holding the inode lock is allowed to write to the file at any single point in time.
For large NFS servers, the general strategy should be to evenly divide the disk I/O demand across as many disk and disk adapter devices as possible. On a system where disk I/O has been well-distributed, it is possible to reach a point where CPU load on the server becomes the limiting factor on the workload performance