What is the PVFS2 roll?

Resolving The Problem

What is the PVFS2 roll?

PVFS2 Roll

The PVFS2 (Parallel Virtual Filesystem 2) roll is a bundle of all the components you need to run a high-performance distributed file system.

The following groups have collaborated on or supported the development of PVFS:

• Parallel Architecture Research Laboratory at Clemson University
• Mathematics and Computer Science Division at Argonne National Laboratory
• Ohio Supercomputer Center
• NASA Goddard Space Flight Center Code 931
• National Computational Science Alliance (NCSA)

There are three components in PVFS2 1.3.2:

1. Meta server
2. Data server
3. Client

The roll creates a sample distributed file system that should then be fine tuned for your own configuration and hardware. PVFS2 allows the disk space in each node to be accessible to all nodes as a single file system, creating a high-speed file system ideal for datasets and job information. There are some limitations and an administrator should understand them before configuring it. The latest documentation is available from the PVFS2 Web site at: http://www.pvfs.org/pvfs2/documentation.html.

One host must be dedicated as the meta server and named
pvfs2-meta-server-0-0. Once this host is installed, all the others will be able to access a sample PVFS2 file system under /mnt/pvfs2.

During installation, an autofs configuration file is installed along with the binaries and source code on all clients. On the first startup after installation, the kernel module will be built. The source code is included so that a more optimal kernel module can be built. The default kernel module includes support for Ethernet. You can add InfiniBand and Myrinet?support by rebuilding the kernel module.

If the installation process was interrupted (for example, from a power outage), pvfs2-meta-server will not reinstall.

All nodes in the cluster are clients by default.

The data servers provide the disk space that is combined together to form the distributed file system. For example ten data servers with 10 GB of free space could create a distributed file system size of approximately 100 GB. Generally more data servers will provide more space and more speed. Loss of a data server will cause the loss of the portion of the file system it contained.

Compute nodes may be used as data servers with some additional configuration. If a machine is to be a dedicated data server, it should be installed as a PVFS2 meta server appliance. The batch queuing systems SGE, Lava, and LSF HPC will be disabled in this appliance type.

The PVFS2 meta server appliance provides both a meta server and a data server. The configuration is intended only as a demonstration.

A real production installation will require the use of one meta server and one or more data servers. Adding additional data servers after the file system is in use is difficult, so they should be allocated during cluster configuration.

The meta server is responsible for maintaining the distributed file system index. This is a critical component of the distributed file system. Currently PVFS2 only allows one meta server per file system. If this host goes down, or if you re-install, all the data is lost.

Name all meta server hosts with the following naming convention: pvfs2-meta-server-0-0.

Do not use the add-hosts tool to add PVFS2 meta or data servers.

MPI, Myrinet, and Infiniband

Support for MPI (Message Passing Interface), Myrinet, and Infiniband is not built in to this roll. For information, please see http://www.pvfs.org/pvfs2/pvfs2-quickstart.html.

If you have Myrinet and Cisco Topspin drivers and want to use them in PVFS2, you need to rebuild the package. Run the configure script and provide one or more of the following options:

--with-gm=

--with-ib=

Production cluster configuration

Consult the PVFS2 Web site for more detailed instructions. The following steps outline what is necessary to configure a production cluster.

1. Select one system to be meta server, and install the PVSF2 meta server roll. If possible create a mirrored file system to hold the PVFS2 meta data.
2. Select a number of machines to be data servers.
3. Install using the appropriate appliance:
   • If you do not intend the data servers to be compute nodes, install the PVSF2 meta server appliance on them.
   • If you intend the data servers to be compute nodes, install using the compute appliance. Disable automatic re-installation by running the following commands:
     1. • # service rocks-grub stop
        • # chkconfig --del rocks-grub
4. Run pvfs2-genconfig to generate the configuration files on the meta server.
5. Copy the configuration files to the data servers.
6. Format the file systems on the data servers by running the following command:

   # pvfs2-server pvfs2-fs.config pvfs2-server.conf-$HOSTNAME -f 

7. Start the meta and data servers: /etc/init.d/pvfs2-server start.
8. Mount the PVFS2 file system.