Suboptimal performance due to frequent invocation or execution of maintenance commands

When the GPFS maintenance operations like mmbackup, mmapplypolicy, mmdf, mmcrsnapshot, mmdelsnapshot, and others are in progress, they can consume some percentage of system resources. This can impact the I/O performance of applications. If these maintenance operations are scheduled frequently, for example within every few seconds or minutes, the performance impact can be significant, unless the I/O subsystem is sized adequately to handle both the application and the maintenance operation I/O load.

Problem identification

Check the GPFS log file /var/adm/ras/mmfs.log.latest on the file system manager node mmlsmgr to verify whether any GPFS maintenance operations are being invoked frequently by a cron job or other cluster management software like Nagios.

In the sample output below, the mmdf command is being invoked periodically every 3-4 seconds.

Tue Jan 19 15:13:47.389 2016: [I] Command: mmdf /dev/gpfs2
Tue Jan 19 15:13:47.518 2016: [I] Command: successful mmdf /dev/gpfs2
Tue Jan 19 15:13:51.109 2016: [I] Command: mmdf /dev/gpfs2
Tue Jan 19 15:13:51.211 2016: [I] Command: successful mmdf /dev/gpfs2
Tue Jan 19 15:13:54.816 2016: [I] Command: mmdf /dev/gpfs2
Tue Jan 19 15:13:54.905 2016: [I] Command: successful mmdf /dev/gpfs2
Tue Jan 19 15:13:58.481 2016: [I] Command: mmdf /dev/gpfs2
Tue Jan 19 15:13:58.576 2016: [I] Command: successful mmdf /dev/gpfs2
Tue Jan 19 15:14:02.164 2016: [I] Command: mmdf /dev/gpfs2
Tue Jan 19 15:14:02.253 2016: [I] Command: successful mmdf /dev/gpfs2
Tue Jan 19 15:14:05.850 2016: [I] Command: mmdf /dev/gpfs2
Tue Jan 19 15:14:05.945 2016: [I] Command: successful mmdf /dev/gpfs2
Tue Jan 19 15:14:09.536 2016: [I] Command: mmdf /dev/gpfs2
Tue Jan 19 15:14:09.636 2016: [I] Command: successful mmdf /dev/gpfs2
Tue Jan 19 15:14:13.210 2016: [I] Command: mmdf /dev/gpfs2
Tue Jan 19 15:14:13.299 2016: [I] Command: successful mmdf /dev/gpfs2
Tue Jan 19 15:14:16.886 2016: [I] Command: mmdf /dev/gpfs2
Tue Jan 19 15:14:16.976 2016: [I] Command: successful mmdf /dev/gpfs2
Tue Jan 19 15:14:20.557 2016: [I] Command: mmdf /dev/gpfs2
Tue Jan 19 15:14:20.645 2016: [I] Command: successful mmdf /dev/gpfs2

Problem resolution and verification

Adjust the frequency of the GPFS maintenance operations so that it does not impact the applications performance. The I/O subsystem must be designed in such a way that it is able to handle both the application and the maintenance operation I/O load.

You can also use the mmchqos command to allocate appropriate maintenance IOPS, which can reduce the impact of the maintenance operations on the application. For more information on how to allocate appropriate maintenance IOPS commands, see Suboptimal performance due to maintenance commands in progress.