Suboptimal performance due to maintenance commands in progress

When in progress, long-running GPFS maintenance operations like mmrestripefs, mmapplypolicy, mmadddisk, and mmdeldisk, consume some percentage of the system resources. Significant consumption of the system resources can impact the I/O performance of the application.

Problem identification

Check the GPFS log file /var/adm/ras/mmfs.log.latest on the File System Manager node mmlsmgr to verify whether any GPFS maintenance operations are in progress.

The following sample output shows that the mmrestripefs operation was initiated on Jan 19 at 14:32:41, and the operation was successfully completed at 14:45:42. The I/O performance of the application is impacted during this time frame due to the execution of the mmrestripefs command.

Tue Jan 19 14:32:41.625 2016: [I] Command: mmrestripefs /dev/gpfs2 -r -N all
Tue Jan 19 14:45:42.975 2016: [I] Command: successful mmrestripefs /dev/gpfs2 -r -N all

Problem resolution and verification

The Quality of Service (QoS) feature for I/O operations is used to allocate appropriate maintenance IOPS to reduce the impact of the maintenance operation on the application. In the following sample output, the file system consists of a single storage pool – the default ‘system’ pool. The QoS feature is disabled and inactive.

 mmlsqos gpfs1a
QOS config::     disabled
QOS status::     throttling inactive, monitoring inactive

You can use the mmchqos command to allocate appropriate maintenance IOPS to the IBM Storage Scale system. For example, consider that the storage system has 100 K IOPS. If you want to allocate 1000 IOPS to the long running GPFS maintenance operations for the system storage pool, use the mmchqos command to enable the QoS feature, and allocate the IOPS as shown:

# mmchqos gpfs1a --enable pool=system,maintenance=1000IOPS

Adjusted QOS Class specification: pool=system,other=inf,maintenance=1000Iops
QOS configuration has been installed and broadcast to all nodes.

Verify the QoS setting and values on a file system by using the mmlsqos command.

# mmlsqos gpfs1aQOS config:: enabled --

pool=system,other=inf,maintenance=1000IopsQOS status:: throttling active, monitoring active

Note: Allocating a small share of IOPS, for example 1000 IOPS, to the long running GPFS maintenance operations can increase the maintenance command execution times. So, depending on the operation's needs, the IOPS assigned to the ‘other’ and ‘maintenance’ class must be adjusted by using the mmchqos command. This balances the application as well as the I/O requirements for the GPFS maintenance operation.

For more information on setting the QoS for I/O operations, see the mmlsqos command and the Setting the Quality of Service for I/O operationssection.