Performance Tuning Checklist

This page has not been liked. Updated 5/29/15, 12:33 PM by ScottGPFSTags: None

To begin analyzing GPFS performance or to design a system a "quick start" checklist can be a useful tool. This checklist is a good place to start and is a helpful reminder for those already familiar with GPFS. It contains a list of "areas to consider" when determining the performance of a GPFS solution. Tuning these parameters, in many cases, can greatly improve application performance.

This page is intended to remain a list for easy reference. Further details on these or other parameters or tuning tips should be provided on separate pages and linked to from here.

  • How to get started

Checklist topics

Sample Configurations

  • General use HPC Cluster
  • General use small Direct Attached Cluster

How to get started



The biggest challenge when tuning GPFS is developing a clear understanding of your goals. Before embarking on a performance tuning exercise it is a good idea to clearly define a performance target.

Developing a performance target

Developing a clear performance target allows you to keep your tuning effort focused. Since tuning is often a game of give and take a focused approach can help you make those value decisions as they arise. A performance target can take many forms:

  • A throughput target - " I need to achieve 200 MB/sec using myapp"
  • An IOP target - " I need to be able to create 1,000 files/second using filecreate.bash"
  • An application transaction - " I need to complete 100 cube requests a minute"
  • A backup or recovery target - " I need to be able to perform a differential backup on 10 million files in 2 hours with a 10% daily change rate and an average file size of 1MB"

Whatever your target there are certain characteristics the goal should have.

  • Measurable - the target should be a metric like "files created per second" that can be easily determined.
  • Tools - The tools used to execute and measure the test should be clearly defined. You should research your application requirement and find or develop a relevant tool to generate a workload. For the most efficient analysis the results of a test run should be readily apparent and not require extensive post processing before they can be interpreted.
  • Relevant - The test workload should be as relevant to the production application as possible. This is the #1 issue with most performance tuning exercises. Many performance tuning engagements fail because the workload is chosen because it is handy, and not necessarily relevant. For example, IOR is a great test of throughput but does not help if you are tuning a file system to provide SMTP email service.

GPFS Configuration Parameters



File System Configuration


-B Blocksize

-S atime updates

-E mtime updates

Storage Configuration


IO Balance

RAID Level

Cache Configuration

Interconnect Tuning


TCP/IP Tuning

Fibre Channel Settings

InfiniBand Configuration

Application Design


fsync() - A file sync is a heavy IO operation that adds latency to a file create or write operation. This means that a single application thread writing one file at a time can create fewer files per second then when not using fsync. To improve performance you can use fsync less or make sure your application has multiple processes running in parallel.



Sample Configurations



This section contains sample configurations with some discussion around why the specific parameters were chosen. Of course it is always recommended that you test any configuration parameters before implementing them on a production cluster


General use HPC Cluster



A general use HPC cluster is typical at many universities, for example. These clusters need to support a variety of I/O workloads so you typically cannot tune to a specific type of I/O pattern. Even with unpredictable IO workloads these clusters have many properties in common. For example, there are usually three types of server usage:

  1. GPFS NSD Servers
  2. User login nodes
  3. Compute Nodes

These recommendations make many assumptions about server memory and communication infrastructure. Remember these are just examples and are a good place to start though you need to adjust as necessary to fit your environment.

There are slightly different starting points for the three server types

NSD Servers

The assumption here is that these NSD servers are serving the rest of the cluster and providing occasional NFS service to external clients. In this configuration you need to account for local IO (NFS Service) and NSD server IO.

pagepool = 8GB or more - NSD servers do not require large amounts of memory.

maxMBpS = 16000

maxFilesToCache = 100,000

See NSD Server Tuning

User Login Nodes

The assumption here is that these servers are user interactive nodes. The IO throughput requirements may or may not be high but there are many users doing "ls" operations.

pagepool = 8GB

maxMBpS = 5000

maxFilesToCache = 100,000

Compute Nodes

The assumption here is that these servers are used by applications and that memory is best used by the application and not GPFS cache.

pagepool = 1GiB

maxMBpS = 5000

maxFilesToCache = default


General use small Direct Attached Cluster



In this configuration you have small cluster 2-10 nodes that all have direct access to a common set of LUNS over a SAN, for example. All of these nodes are running a variety of applications.

All Nodes

pagepool = 8GB

maxMBpS = 5000

maxFilesToCache = 100,000