Topic
  • 4 replies
  • Latest Post - ‏2012-09-13T06:28:52Z by Tiya_
Tiya_
Tiya_
6 Posts

Pinned topic what affect the speed of mmstartup on client node

‏2012-09-12T08:59:32Z |
Hi, gpfs expert

I have a gpfs cluster with 4 nodes, one as gpfs master and provide nsd, the other 3 as a gpfs client with no nsd. I put all the operations for setup a gpfs client in a bash script. This bash is used as xcat post script so I can add the nodes to my cluster on each provisioning.

In the bash script, 1.First, client node ssh to gpfs master and execute "mmaddnode", and 2. do mmstartup locally on client node 3. wait until client is active and mmmount all filesystem.
It looks like this:

# first mmaddnode remotely on master node
State=`ssh root@$MasterNode "/usr/lpp/mmfs/bin/mmaddnode -N $ClinetNode:client"`

# wait the asynchronous process
sleep 5

#Accept gpfs license locally
State=`/usr/lpp/mmfs/bin/mmchlicense client --accept -N $ClinetNode 2>&1`

#Startup gpfs Deamon
State=`/usr/lpp/mmfs/bin/mmstartup 2>&1`
if [; then
# Polling status until node is active. The longest period is 3X3 seconds
Count=3
NodeState=""
State=""
while (( $Count ))
do
State=`/usr/lpp/mmfs/bin/mmgetstate`
NodeState=`echo $State | grep "$ClientNode" | awk '{print $NF}'`
if [;then
break
else
sleep 3
Count=$$Count-1
fi
done
else
echo "Error!!! mmstartup failure on $ClientNode!!!" 1>&2
echo "Please check $ClientNode:$Logfile for trouble shooting if gpfs is not mounted!!!" 1>&2
fi

It runs well when there are two clients provisioned synchronously. But when 3 nodes, the node state remains done after polling 3 times on some node some times. When I logon the node polling failed, the node state get active.
So what affect the speed of gpfs startup? As far as my understanding, mmstartup just startup gpfs daemon locally and exchange some configurations with gpfs master. It seems the larger the cluster, the slower to startup. I guess there might be some lock in gpfs, so the client have to startup one by one.
How can I speed up the process if a large cluster need to mmstartup at the same time?
Highly appreciated.~~~:)
Tiya
Updated on 2012-09-13T06:28:52Z at 2012-09-13T06:28:52Z by Tiya_
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: what affect the speed of mmstartup on client node

    ‏2012-09-12T14:16:24Z  
    Please describe better what you are trying to accomplish.

    mmaddnode is meant to be used only when you roll in some new hardware -- not on every reboot.

    Normally GPFS is configured so it automagically starts up and mounts all of your production filesystems just after the OS is booted.

    If for some reason, some node is not up or some filesystem is not mounted, just issuing ...

    mmstartup -a

    ... and then

    mmmount whatever-filesystem -a
    Will get everything you need going.
  • truongv
    truongv
    81 Posts

    Re: what affect the speed of mmstartup on client node

    ‏2012-09-12T18:37:54Z  
    It looks like you have a typo in your script. You mmadded node ClinetNode but grep for ClientNode later in the script.
  • Tiya_
    Tiya_
    6 Posts

    Re: what affect the speed of mmstartup on client node

    ‏2012-09-13T06:18:12Z  
    Please describe better what you are trying to accomplish.

    mmaddnode is meant to be used only when you roll in some new hardware -- not on every reboot.

    Normally GPFS is configured so it automagically starts up and mounts all of your production filesystems just after the OS is booted.

    If for some reason, some node is not up or some filesystem is not mounted, just issuing ...

    mmstartup -a

    ... and then

    mmmount whatever-filesystem -a
    Will get everything you need going.
    Sorry, I did not made that clear. This script is not used during reboot but on each re-install. Each time the node install different image of linux or setup as diskless machine, the script will run to install gpfs automatically.
    So when the installation and network is ready, the script will run to install gpfs packages and start it up on each nodes. To do so, the script will first visit gpfs master node to do mmaddnode -N, and then do mmchlicence and mmstartup locally on the re-install node.
  • Tiya_
    Tiya_
    6 Posts

    Re: what affect the speed of mmstartup on client node

    ‏2012-09-13T06:28:52Z  
    • truongv
    • ‏2012-09-12T18:37:54Z
    It looks like you have a typo in your script. You mmadded node ClinetNode but grep for ClientNode later in the script.
    Yes.:)something wrong when paste the script because I simplified it,it comes from a long script. I post the original part in attachment.

    Attachments