Effects of NFS data caching on read throughput

NFS sequential read throughput, as measured at the client, is enhanced via the VMM read ahead and caching mechanisms.

Read ahead allows file data to be transferred to the client from the NFS server in anticipation of that data being requested by an NFS client application. By the time the request for data is issued by the application, it is possible that the data resides already in the client's memory, and thus the request can be satisfied immediately. VMM caching allows rereads of file data to occur instantaneously, assuming that the data was not paged out of client memory which would necessitate retrieving the data again from the NFS server.

While many applications might benefit from VMM caching of NFS data on the client, there are some applications, like databases, that might perform their own file data cache management. Applications that perform their own file data cache management might benefit from using direct I/O, or DIO, over NFS. You can enable DIO over NFS with the dio option of the mount command or by specifying the O_DIRECT flag with the open() system call.

The following list details the benefits of DIO:

You avoid double-caching of file data by the VMM and the application.
You can gain CPU efficiency on file reads and writes since the DIO function bypasses the VMM code.

Applications that perform their own file data cache management and file access serialization, like databases, might benefit from using concurrent I/O, or CIO. In addition to the benefits of DIO, CIO does not serialize read and write file accesses, which allows multiple threads to read or write to the same file concurrently.

Note: Using CIO or DIO might degrade performance for applications that rely heavily on VMM file caching and the read-ahead and write-behind VMM optimizations for increased system performance.

You can use CacheFS to further enhance read throughput in environments with memory-limited clients, very large files, and/or slow network segments by adding the potential to satisfy read requests from file data residing in a local disk cache on the client. See Cache file system for more information.

Data caching for sequential reads of large files might result in heavy page replacement activity as memory is filled with the NFS data cache. You can improve performance by avoiding the page replacement activity by using the release-behind on read, rbr, mount option or the nfs4cl setfsoptions argument for NFS version 4. For sequential reads of large files, the real memory for the previous reads is then freed as the sequential reads continue.

If the rbr mount option starts to release memory that you are going to need again soon, you can use the nfs_auto_rbr_trigger tunable of the nfso command instead. The nfs_auto_rbr_trigger tunable, which is measured in megabytes, serves as a read-offset threshold for when the release-behind on read option takes effect. For example, if the nfs_auto_rbr_trigger tunable is set to 100 MB, the first 100 MB of a sequentially-read file is cached and the rest of the file is released from memory.