So, I've created a GPFS callback:
command = /opt/StorageTools/bin/nodeDownCallback.pl
event = nodeLeave
node = server5.domain.com
parms = %eventName %clusterName %downNodes %myNode %fsName
This works well when local nodes leave the cluster, via explusion, etc. However, this callback also is called whenever a remote cluster node leaves it's own cluster. Since if I recall, remote cluster nodes are somehow also a part of the local cluster (I forgot the official GPFS reason; I might be wrong here). The problem is, I don't want to have the callback called on the local cluster when nodes leave.
For instance, our situation now is that we have one local NSD node down to an IB cable issue, and it's been down for two days, server8.domain.com
However, whenever a remote node leaves the cluster (this happens sometimes due to our remote cluster architecture), the callback is still called. From the server5.domain.com mmfslog:
Fri Jan 4 02:32:05.441 2013: VERBS RDMA closed connection to 172.23.100.86 vs86 on mlx4_0 port 2
Fri Jan 4 02:32:05.941 2013: Calling User Exit Script nodeDownCallback: event nodeLeave, Async command /opt/StorageTools/bin
Fri Jan 4 02:32:06.066 2013: VERBS RDMA closed connection to 172.23.247.207 R00-ID-J05 on mlx4_0 port 2
Fri Jan 4 02:32:06.528 2013: VERBS RDMA closed connection to 172.23.149.169 Q1H-I2-J07 on mlx4_0 port 2
Fri Jan 4 02:32:06.858 2013: VERBS RDMA closed connection to 172.23.53.238 Q0H-I0-J04 on mlx4_0 port 2
Fri Jan 4 02:32:07.801 2013: Calling User Exit Script nodeDownCallback: event nodeLeave, Async command /opt/StorageTools/bin
Fri Jan 4 02:32:07.855 2013: Calling User Exit Script nodeDownCallback: event nodeLeave, Async command /opt/StorageTools/bin
Fri Jan 4 02:32:07.909 2013: Calling User Exit Script nodeDownCallback: event nodeLeave, Async command /opt/AStorageTools/bin
Worse off, the callback script does not actually list the above nodes as the node that left. Here is some example output from the nodeDownCallback.pl script:
Down Node(s): server8.domain.com
Reporting Node: serve5.domain.com
This was triggered by the nodeLeave callback event.
Please check the status of the cluster, and the down nodes.
So when a remote cluster node leaves, the callback is run on server5.domain.com, and it lists server8.domain.com (which is right). We basically got 4 emails within one second telling us something we've known for 2 days because a non-local cluster node left.
Basically, are we doing something wrong with the callback here? All I would like is that if a local node leaves a cluster for any reason, the nodeLeave callback is called, and that is it. I don't care if remote cluster nodes leave, since even if I did, only the local %downNodes% are printed anyways.
We are running GPFS 188.8.131.52
This topic has been locked.
4 replies Latest Post - 2013-01-04T19:22:03Z by dlmcnabb
Pinned topic GPFS callbacks - nodeLeave - remote cluster nodes
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
Updated on 2013-01-04T19:22:03Z at 2013-01-04T19:22:03Z by dlmcnabb
FelipeKnop 120000G51H18 PostsACCEPTED ANSWER
Re: GPFS callbacks - nodeLeave - remote cluster nodes2013-01-04T16:30:36Z in response to SystemAdminThe behavior you describe matches the implementation. That is, the event will be triggered even when a remote node leaves the cluster. In addition, the list of nodes provided via
%downNodes will only include the local nodes in the cluster.
The %eventNode specifier can be used to retrieve the set of nodes involved in a given
event, and that should display the name of the remote node. Using %eventNode will not
reduce the number of events but will at least provide the name of a (possibly remote)
node which just went down.
SystemAdmin 110000D4XK2092 PostsACCEPTED ANSWER
Re: GPFS callbacks - nodeLeave - remote cluster nodes2013-01-04T17:15:35Z in response to FelipeKnopThanks Felipe.
I can probably throw something together to use the eventNodes, and correlate whether or not it part of the local cluster. This functionality is a bit odd though, but I'll manage.
However, the nodeLeave for some of the examples I pasted above is technically wrong. The nodes in question never went down. They only closed their RDMA QPs, since there wasn't any data between that client and the NSD servers for awhile. I don't remember the exact amount of time, but if GPFS doesn't send any data over an RDMA QP, it will close it eventually, and reopen if there is a data transfer.
user@sn1:~ ]$ ssh Q1H-I2-J07
17:10:35 up 17 days, 19:22, 2 users, load average: 3.11, 3.08, 3.08
The GPFS daemon has up since Dec 17th.
FelipeKnop 120000G51H18 PostsACCEPTED ANSWER
Re: GPFS callbacks - nodeLeave - remote cluster nodes2013-01-04T19:03:12Z in response to SystemAdminThe GPFS daemon will periodically scan the clusters to which the local node is joined, and then will cause the local node to leave remote clusters which have not been used "in a while" (I believe the interval is 5 mins). When the node leaves a cluster, that appears to the remaining nodes in that cluster as a node leave/death. Such behavior only happens for remote clusters: the daemon will not leave the local cluster, as long as it can communicate with the cluster manager.
dlmcnabb 120000P4JT985 PostsACCEPTED ANSWER
Re: GPFS callbacks - nodeLeave - remote cluster nodes2013-01-04T19:22:03Z in response to FelipeKnopA remote node will only leave the main cluster if it has unmounted all filesystems from that main cluster. If there is any filesystem mounted on the client from the main cluster, it will not leave due to inactivity. However, if you are using autofs on the client to automatically mount a filesystem, autofs may unmount automatically due to inactivity. Then the node will eventually leave the cluster after another period times out.