Parameters for performance tuning and optimization

Tuning guide for frequently-changed parameters

Autoload

When Autoload is set to yes, GPFS™ starts automatically on the nodes that are rebooted. The rebooted nodes rejoin the cluster, the file system automount option is set to yes, and the file system is mounted. The default value of this parameter is no.

Important: Set Autoload to no before fixing hardware issues and performing system maintenance.

deadlockDetectionThreshold

When deadlockDetectionThreshold is set to 0, the GPFS dead-lock detection feature is disabled. The default value of this parameters is 300 seconds.

Important: You must enable the GPFS dead-lock detection feature to collect debug data and resolve dead lock issue in a cluster. If dead-lock events occur frequently, fix the problem instead of disabling the feature.

defaultHelperNodes

The nodes that are added to defaultHelperNodes are used in running certain commands, such as mmrestripefs. Running the GPFS command on partial nodes in a cluster, such as running the mmrestripefs command on all NSD server nodes, might have a better performance. The default value of this parameter is all nodes in cluster.

Important: Set the –N option for GPFS management commands or change the value of defaultHelperNodes before running the GPFS management commands.

maxFilesToCache

The maxFilesToCache parameter specifies the number of files that can be cached by each node. The value of the maxFilesToCache parameter can be any number ranging from 1 to 100,000,000. The default value of this parameter is 4000.

The value of this parameter must be large to handle the number of concurrently open files and allow the caching of recently used files. Changing the value of maxFilesToCache affects the amount of memory used on the node. In a large cluster, a change in the value of maxFilesToCache is greatly magnified. Increasing maxFilesToCache in a large cluster with hundreds of nodes increases the number of tokens a token manager needs to store. Ensure that the manager node has enough memory and tokenMemLimit is increased when running GPFS version 4.1.1 and earlier. Therefore, increasing the value of maxFilesToCache on large clusters usually happens on a subset of nodes that are used as log-in nodes, SMB and NFS exporters, email servers, and other file servers. For systems on which applications use a large number of files, increasing the value of maxFilesToCache might be beneficial, especially where a large number of small files are accessed.

maxBlockSize

The value of maxBlockSize must be equal or larger than the maximum block size of all file systems in the local and remote clusters. Before changing this parameter, ensure that the IBM Spectrum Scale™ daemon on each node in the cluster is shut down. The default value is 1 MB.

maxMBpS

The maxMBpS parameter indicates the maximum throughput in megabytes per second that GPFS can submit into or out of a single node. GPFS calculates from this variable how many prefetch/writebehind threads to schedule for sequential file access.

In GPFS version 3.5 and earlier, the default value is 2048. But if the node has faster interconnect, such as InfiniBand or 40GigE or multiple links) you can set the parameter to a higher value. As a general rule, try setting maxMBpS to twice the I/O throughput that the node can support. For example, if the node has 1 x FDR link and the GPFS configuration parameter verbRdma has been enabled, then the expected throughput of the node is 6000 MB/s. In this case, set maxMBpS to 12000.

Setting maxMBpS does not guarantee the desired GPFS sequential bandwidth on the node. All the layers of the GPFS stack, including the node, the network, and the storage subsystem, must be designed and tuned to meet the I/O performance requirements.

maxStatCache

If the value of maxFilesToCache is changed, the value of maxStatCache must also be changed. If it is not, maxStatCache defaults to 4 * maxFilesToCache. The value of the maxStatCache parameter can be any number ranging from 0 to 10,000,000. The default value of this parameter is 1000.

The maxStatCache parameter sets aside the pageable memory to cache attributes of files that are not currently in the regular file cache. This improves the performance of stat() calls for applications with a working set that does not fit in the regular file cache. For systems where applications test the existence of files, or the properties of files, without actually opening them as backup applications do, increasing the value for maxStatCache can be beneficial. On a system where maxFilesToCache is increased to a very high value, set the value to a number less than 4 * maxFilesToCache.

Important: The maxStatCache is ineffective on Linux inodes. Therefore, on Linux systems, the value of maxStatCache must be set to the default of 0, and the value of maxFilesToCache can be modified.

nsdMaxWorkerThreads

NSD server tuning. For more information about GPFS NSD server design and tuning, see NSD Server Design and Tuning.

Pagepool

The Pagepool parameter is used to change the size of the data cache on each node. The default value is either one-third of the physical memory of the node or 1G, whichever is smaller. This value applies to new installations only. On upgrades, the existing default value is maintained.

The maximum GPFS pagepool size depends on the value of the pagepoolMaxPhysMemPct parameter and the amount of physical memory on the node. Unlike local file systems that use the operating system page cache to cache file data, GPFS allocates its own cache called the page pool. The GPFS page pool is used to cache user file data and file system metadata. Along with file data, the page pool supplies memory for various types of buffers such as prefetch and write behind. The default page pool size might be sufficient for sequential IO workloads. The default page pool size might not be sufficient for Random IO or workloads that involve a large number of small files.

In some cases, allocating 4GB, 8GB, or more memory can improve the workload performance. For database applications that use Direct IO, the page pool is not used for any user data. The main purpose in this case is for system metadata and caching the indirect blocks for the files. For NSD server, if no applications or file system manager services are running on NSD server, the page pool is only used transiently by the NSD worker threads to gather data from client nodes and write the data to disk. The NSD server does not cache any of the data.

readReplicaPolicy

The readReplicaPolicy parameter specifies the location from which the disk must read the replicas. The valid values are default, local and fastest. The default value is default.

By default, GPFS reads the first replica even when there is no replica on the local disk. When the value of this parameter is set to local, the policy reads replicas from the local disk only if the local disk has data. For performance considerations, this is the recommended setting for FPO environments. When the value of this parameter is set to fastest, the policy reads replicas from the disk considering the fastest based on the read I/O statistics of the disk. In a system with SSD and regular disks, the value of fastestPolicyCmpThreshold can be set to a greater number, such as 100, to let GPFS refresh the slow disk speed statistics less frequently.

restripeOnDiskFailure

The restripeOnDiskFailure specifies if GPFS attempts to automatically recover from certain common disk failure situations. The default value of this parameter is no.

Important: While deploying FPO or when the HAWC feature is enabled, set the restripeOnDiskFailure parameter to yes.

tiebreakerDisks

For a small cluster with up to eight nodes that have SAN-attached disk systems, define all nodes as quorum nodes and use tiebreaker disks. With more than eight nodes, use only node quorum. While defining the tiebreaker disks, you can use the SAN-attached NSD in the file system. The default value of this parameter is null, which means no tiebreaker disk has been defined.

unmountOnDiskFail

The unmountOnDiskFail specifies how the GPFS daemon responds when a disk failure is detected. The valid values of this parameter are yes, no, and meta. The default value is no.

Important: Set the value of unmountOnDiskFail to meta for FPO deployment or when the file system metadata and data replica are more than one.

workerThreads

The workerThreads parameter controls an integrated group of variables that tune the file system performance in environments that are capable of high sequential and random read and write workloads and small file activity.

The default value of this parameter is 48 for a base IBM Spectrum Scale cluster and 512 for a cluster with protocols installed. A valid value can be any number ranging from 1 to 8192. The -N flag is valid with this variable. This variable controls both internal and external variables. The internal variables include maximum settings for concurrent file operations, for concurrent threads that flush dirty data and metadata, and for concurrent threads that prefetch data and metadata. You can adjust the following external variables with the mmchconfig command:

• logBufferCount
• preFetchThreads
• worker3Threads