We have a 22.214.171.124 GPFS cluster with (4) NSD servers (quorum-managers) and (1) quorum node (Xcat cluster manager) all running RHEL 6.1. I noticed the quorum node took a long time to restart GPFS (mmstartup) after a reboot, it does not autoload. So I tested mmshutdown and it took 18 minutes on the node. The node is also the Ganglia collection server so it is running Gmetad and gmond. If I shutdown gmetad and gmond then mmshutdown takes 45 seconds.
So I thought to fix this problem I would just have to setup /etc/rc.d so that gmetad and gmond shutdown before GPFS. Well there are K## and S## links for gmetad and gmond but none for GPFS (ls -l */*gpfs* in /etc/rc.d). The only thing for GPFS is init.d/gpfs -> /usr/lpp/mmfs/bin/gpfsrunlevel, none of the NSD servers have K## or S## links either and mmlsconfig shows autoload = yes.
This GPFS cluster was recently rebuilt (3.3 to 3.4 and upgraded (RHEL 5 to RHEL 6 and (2) more NSD servers and more DS5000 arrays). What I want to accomplish is fixing the cluster so if someone reboots or shutsdown one or all of the quorum nodes that we don't create a problem with GPFS from not being shutdown graciously. Do I add the K## and S## links to the NSD servers and add the adjusted K## to the quorum servers to work around Ganglia daemons running?
Thoughts or suggestions?