Scale your file system with Parallel NFS
Read and write hundreds of gigabytes per second
Through NFS, which consists of server and client software and protocols running among them, a computer can share its physical file system with many other computers connected to the same network. NFS masks the implementation and type of the server's file system. To applications running on an NFS client, the shared file system appears as if it's local, native storage.
Figure 1 illustrates a common deployment of NFS within a network of heterogeneous operating systems, including Linux®, Mac OS X, and Windows®, all of which support the NFS standard. (NFS is the sole file system standard supported by the Internet Engineering Task Force.)
Figure 1. A simple NFS configuration
In Figure 1, the Linux machine is the NFS server; it shares or exports (in NFS parlance) one or more of its physical, attached file systems. The Mac OS X and Windows machines are NFS clients. Each consumes, or mounts, the shared file system. Indeed, mounting an NFS file system yields the same result as mounting a local drive partition—when mounted, applications simply read and write files, subject to access control, oblivious to the machinations required to persist data.
In the case of a file system shared through NFS, Read and Write operations traverse—represented by the blue shadow—through the client (in this case, the Windows machine) to the server, which ultimately fulfills requests to retrieve or persist data or alter file metadata, such as permissions and last modified time.
NFS is quite capable, as evidenced by its widespread use as Network Attached Storage (NAS). It runs over both Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) and is (relatively) easy to administer. Furthermore, NFS version 4, the most recent, ratified version of the standard, improves security, furthers interoperability between Windows and UNIX®-like systems, and provides better exclusivity through lock leases. (NFSv4 was ratified in 2003.) NFS infrastructure is also inexpensive, because it typically runs well on common Ethernet hardware. NFS suits most problem domains.
However, one domain not traditionally well served by NFS is high-performance computing (HPC), where data files are very large, sometimes huge, and the number of NFS clients can reach into the thousands. (Think of a compute cluster or grid composed of thousands of commodity computing nodes.) Here, NFS is a liability, because the limits of the NFS server—be it bandwidth, storage capacity, or processor speed—throttle the overall performance of the computation. NFS is a bottleneck.
Or, at least it was.
The next revision of NFS, version 4.1, includes an extension called Parallel NFS (pNFS) that combines the advantages of stock NFS with the massive transfer rates proffered by parallelized input and output (I/O). Using pNFS, file systems are shared from server to clients as before, but data does not pass through the NFS server. Instead, client systems and the data storage system connect directly, providing numerous parallelized, high-speed data paths for massive data transfers. After a bit of initialization and handshaking, the pNFS server is left "out of the loop," and it no longer hinders transfer rates.
Figure 2 shows a pNFS configuration. At the top are the nodes of a compute cluster, such as a large pool of inexpensive, Linux-powered blades. At the left is the NFSv4.1 server. (For this discussion, let's just call it a pNFS server.) At the bottom is a large parallel file system.
Figure 2. The conceptual organization of pNFS
Like NFS, the pNFS server exports file systems and retains and maintains the canonical metadata describing each and every file in the data store. As with NFS, a pNFS client—here a node in a cluster—mounts the server's exported file systems. Like NFS, each node treats the file system as if it were local and physically attached. Changes to metadata propagate through the network back to the pNFS server. Unlike NFS, however, a Read or Write of data managed with pNFS is a direct operation between a node and the storage system itself, pictured at the bottom in Figure 2. The pNFS server is removed from data transactions, giving pNFS a definite performance advantage.
Thus, pNFS retains all the niceties and conveniences of NFS and improves performance and scalability. The number of clients can be expanded to provide more computing power, while the size of the storage system can expand with little impact on client configuration. All you need to do is keep the pNFS catalog and storage system in sync.
The nuts and bolts of pNFS
So, how does it work? As shown in Figure 3, pNFS is implemented as a collection of three protocols.
Figure 3. The triad of pNFS protocols
The pNFS protocol transfers file metadata (formally known as a layout) between the pNFS server and a client node. You can think of a layout as a map, describing how a file is distributed across the data store, such as how it is striped across multiple spindles. Additionally, a layout contains permissions and other file attributes. With metadata captured in a layout and persisted in the pNFS server, the storage system simply performs I/O.
The storage access protocol specifies how a client accesses data from the data store. As you might guess, each storage access protocol defines its own form of layout, because the access protocol and the organization of the data must be concordant.
The control protocol synchronizes state between the metadata server and the data servers. Synchronization, such as reorganizing files on media, is hidden from clients. Further, the control protocol is not specified in NFSv4.1; it can take many forms, allowing vendors the flexibility to compete on performance, cost, and features.
Given those protocols, you can follow the client-access process:
- The client requests a layout for the file at hand.
- The client obtains access rights by opening the file on the metadata server.
- When authorized and given the layout, the client is free to access information from the data servers directly. Access proceeds according to the storage access protocol required for the type of store. (More on this below.)
- If the client alters the file, the client's instance of the layout is duly modified, and all modifications are committed back to the metadata server.
- When the client no longer needs the file, it commits any remaining changes, returns its copy of the layout to the metadata server, and closes the file.
More specifically, a Read operation is a series of protocol operations:
- The client sends a
LOOKUP+OPENrequest to the pNFS server. The server returns a file handle and state information.
- The client requests a layout from the server through the
LAYOUTGETcommand. The server returns the file layout.
- The client issues a
READrequest to the storage devices, which initiates multiple Read operations in parallel.
- When the client is finished reading, it expresses the end of the
- If the layout shared with clients is ever obsolete because of separate
activity, the server issues a
CB_LAYOUTRECALLto indicate that the layout is no longer valid and must be purged and/or refetched.
A Write operation is similar, except that the client must issue a
LAYOUTRETURN to "publish" the changes to the
file to the pNFS server.
Layouts can be cached in each client, further enhancing speed, and a client can voluntarily relinquish a layout from the server if it's no longer of use. A server can also restrict the byte range of a Write layout to avoid quota limits or to reduce allocation overhead, among other reasons.
To prevent stale caches, the metadata server recalls layouts that have become inaccurate. Following a recall, every affected client must cease I/O and either fetch the layout anew or access the file through plain NFS. Recalls are mandatory before the server attempts any file administration, such as migration or re-striping.
It's location, location, location
As mentioned above, each storage access protocol defines a type of layout, and new access protocols and layouts can be added freely. To bootstrap the use of pNFS, the vendors and researchers shaping pNFS have already defined three storage techniques: file, block, and object stores:
- File storage is commonly implemented with traditional NFS servers, such as those produced by Network Appliance. The storage farm is realized as a collection of NFS servers, and each file is striped across (a subset or all of) the servers so that clients can retrieve portions of the file simultaneously. Here, the layout enumerates the servers that hold the pieces of the file, the size of the stripe on each server, and the NFS file handle of each segment.
- Block storage is most often implemented with a storage-area network (SAN) composed of many disks or RAID arrays. Many vendors, including IBM and EMC, offer SAN solutions. With block storage, a file is divided into blocks, and the blocks are dispersed among the drives. The block storage layout maps the file blocks to physical storage blocks. The storage access protocol is the SCSI block command set.
- Object storage is similar to file storage, except that file handles are replaced by object IDs and striping tends to be more varied, complex, and capable.
No matter the type of layout, pNFS uses a common scheme to refer to servers. Instead of hostname or volume name, servers are referred to by a unique ID. This ID is mapped to the access protocol-specific server reference.
Which of these storage techniques is better? The answer is, "It depends." Budget, speed, scale, simplicity, and other factors are all part of the equation.
The state of pNFS
Before you break out your checkbook, let's look at the state of pNFS.
As of this writing in November 2008, the draft Request for Comments (RFC) for NFSv4.1 is entering "last call," a two-month period set aside to collect and consider comments before the RFC is published and opened to industry-wide scrutiny. When published, the formal RFC review period often lasts a year.
In addition to providing broad exposure, the draft proposed standard captured in the RFC lays a firm foundation for actual product development. As only minor changes to the standard are expected during the forthcoming review period, vendors can design and build workable, marketable solutions now. Products from multiple vendors will be available some time next year.
In the immediate term, open source prototype implementations of pNFS on Linux are available from a git repository located at the University of Michigan (see Related topics for a link). IBM, Panasas, Netapp, and the University of Michigan Center for Information Technology Integration (CITI) are leading the development of NFSv4.1 and pNFS for Linux.
The potential for pNFS as an open-source parallel file system client is enormous. The fastest supercomputer in the world (as ranked by the Top500 survey) and the first computer to reach a petaflop uses the parallel file system built by Panasas (a supporter of the pNFS object-based implementation). (A petaflop is one thousand trillion operations per second.) Dubbed Roadrunner, located at the Los Alamos National Laboratory and pictured in Figure 4, the gargantuan system has 12,960 processors, runs Linux, and is the first supercomputer to be constructed using heterogeneous processor types. Both AMD Opteron X64 processors and IBM's Cell Broadband Engine™ drive computation. In 2006, Roadrunner demonstrated a peak 1.6 gigabytes-per-second transfer rate using an early version of Panasas's parallel file system. In 2008, the Roadrunner parallel storage system can sustain hundreds of gigabytes per second. In comparison, traditional NFS typically peaks at hundreds of megabytes per second.
Figure 4. Roadrunner, the world's first petaflop supercomputer
The entire NFSv4.1 standard and pNFS are substantive improvements to the NFS standard and represent the most radical changes made to a twenty-something-year-old technology that originated with Sun Microsystems' Bill Joy in the 1980s. Five years in development, NFSv4.1 and pNFS now (or imminently) stands ready to provide super-storage speeds to super-computing machines.
We have seen the future, and it is parallel storage.
- You can get an open source prototype implementation of pNFS from the University of Michigan.
- Read more of Martin's articles on developerWorks.
- Learn more about the NFS v4.1 standard.
- Visit the Panasas pNFS Web site.
- Read more about the University of Michigan's Center for Information Technology Integration's efforts to integrate pNFS into Linux.
- Read the history of the Network File System standard.
- For more NFS background, read "NFSv4 delivers seamless network access" (developerWorks, September 2006).
- Learn more about the Roadrunner at Los Alamos National Laboratory.
- In the developerWorks Linux zone, find more resources for Linux developers (including developers who are new to Linux), and scan our most popular articles and tutorials.
- See all Linux tips and Linux tutorials on developerWorks.