Topic
  • 4 replies
  • Latest Post - ‏2013-01-04T19:22:03Z by dlmcnabb
SystemAdmin
SystemAdmin
2092 Posts

Pinned topic GPFS callbacks - nodeLeave - remote cluster nodes

‏2013-01-04T04:31:42Z |
So, I've created a GPFS callback:

nodeDownCallback
command = /opt/StorageTools/bin/nodeDownCallback.pl
event = nodeLeave
node = server5.domain.com
parms = %eventName %clusterName %downNodes %myNode %fsName
This works well when local nodes leave the cluster, via explusion, etc. However, this callback also is called whenever a remote cluster node leaves it's own cluster. Since if I recall, remote cluster nodes are somehow also a part of the local cluster (I forgot the official GPFS reason; I might be wrong here). The problem is, I don't want to have the callback called on the local cluster when nodes leave.

For instance, our situation now is that we have one local NSD node down to an IB cable issue, and it's been down for two days, server8.domain.com

However, whenever a remote node leaves the cluster (this happens sometimes due to our remote cluster architecture), the callback is still called. From the server5.domain.com mmfslog:

Fri Jan 4 02:32:05.441 2013: VERBS RDMA closed connection to 172.23.100.86 vs86 on mlx4_0 port 2
Fri Jan 4 02:32:05.941 2013: Calling User Exit Script nodeDownCallback: event nodeLeave, Async command /opt/StorageTools/bin
/nodeDownCallback.pl.
Fri Jan 4 02:32:06.066 2013: VERBS RDMA closed connection to 172.23.247.207 R00-ID-J05 on mlx4_0 port 2
Fri Jan 4 02:32:06.528 2013: VERBS RDMA closed connection to 172.23.149.169 Q1H-I2-J07 on mlx4_0 port 2
Fri Jan 4 02:32:06.858 2013: VERBS RDMA closed connection to 172.23.53.238 Q0H-I0-J04 on mlx4_0 port 2
Fri Jan 4 02:32:07.801 2013: Calling User Exit Script nodeDownCallback: event nodeLeave, Async command /opt/StorageTools/bin
/nodeDownCallback.pl.
Fri Jan 4 02:32:07.855 2013: Calling User Exit Script nodeDownCallback: event nodeLeave, Async command /opt/StorageTools/bin
/nodeDownCallback.pl.
Fri Jan 4 02:32:07.909 2013: Calling User Exit Script nodeDownCallback: event nodeLeave, Async command /opt/AStorageTools/bin
/nodeDownCallback.pl.
Worse off, the callback script does not actually list the above nodes as the node that left. Here is some example output from the nodeDownCallback.pl script:
===========
Down Node(s): server8.domain.com
Reporting Node: serve5.domain.com

This was triggered by the nodeLeave callback event.

Please check the status of the cluster, and the down nodes.
============

So when a remote cluster node leaves, the callback is run on server5.domain.com, and it lists server8.domain.com (which is right). We basically got 4 emails within one second telling us something we've known for 2 days because a non-local cluster node left.

Basically, are we doing something wrong with the callback here? All I would like is that if a local node leaves a cluster for any reason, the nodeLeave callback is called, and that is it. I don't care if remote cluster nodes leave, since even if I did, only the local %downNodes% are printed anyways.

We are running GPFS 3.5.0.4
Updated on 2013-01-04T19:22:03Z at 2013-01-04T19:22:03Z by dlmcnabb
  • FelipeKnop
    FelipeKnop
    25 Posts

    Re: GPFS callbacks - nodeLeave - remote cluster nodes

    ‏2013-01-04T16:30:36Z  
    The behavior you describe matches the implementation. That is, the event will be triggered even when a remote node leaves the cluster. In addition, the list of nodes provided via
    %downNodes will only include the local nodes in the cluster.

    The %eventNode specifier can be used to retrieve the set of nodes involved in a given
    event, and that should display the name of the remote node. Using %eventNode will not
    reduce the number of events but will at least provide the name of a (possibly remote)
    node which just went down.

    Felipe
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: GPFS callbacks - nodeLeave - remote cluster nodes

    ‏2013-01-04T17:15:35Z  
    The behavior you describe matches the implementation. That is, the event will be triggered even when a remote node leaves the cluster. In addition, the list of nodes provided via
    %downNodes will only include the local nodes in the cluster.

    The %eventNode specifier can be used to retrieve the set of nodes involved in a given
    event, and that should display the name of the remote node. Using %eventNode will not
    reduce the number of events but will at least provide the name of a (possibly remote)
    node which just went down.

    Felipe
    Thanks Felipe.

    I can probably throw something together to use the eventNodes, and correlate whether or not it part of the local cluster. This functionality is a bit odd though, but I'll manage.

    However, the nodeLeave for some of the examples I pasted above is technically wrong. The nodes in question never went down. They only closed their RDMA QPs, since there wasn't any data between that client and the NSD servers for awhile. I don't remember the exact amount of time, but if GPFS doesn't send any data over an RDMA QP, it will close it eventually, and reopen if there is a data transfer.

    user@sn1:~ ]$ ssh Q1H-I2-J07
    -bash-4.1$ uptime
    17:10:35 up 17 days, 19:22, 2 users, load average: 3.11, 3.08, 3.08
    -bash-4.1$

    The GPFS daemon has up since Dec 17th.
  • FelipeKnop
    FelipeKnop
    25 Posts

    Re: GPFS callbacks - nodeLeave - remote cluster nodes

    ‏2013-01-04T19:03:12Z  
    Thanks Felipe.

    I can probably throw something together to use the eventNodes, and correlate whether or not it part of the local cluster. This functionality is a bit odd though, but I'll manage.

    However, the nodeLeave for some of the examples I pasted above is technically wrong. The nodes in question never went down. They only closed their RDMA QPs, since there wasn't any data between that client and the NSD servers for awhile. I don't remember the exact amount of time, but if GPFS doesn't send any data over an RDMA QP, it will close it eventually, and reopen if there is a data transfer.

    user@sn1:~ ]$ ssh Q1H-I2-J07
    -bash-4.1$ uptime
    17:10:35 up 17 days, 19:22, 2 users, load average: 3.11, 3.08, 3.08
    -bash-4.1$

    The GPFS daemon has up since Dec 17th.
    The GPFS daemon will periodically scan the clusters to which the local node is joined, and then will cause the local node to leave remote clusters which have not been used "in a while" (I believe the interval is 5 mins). When the node leaves a cluster, that appears to the remaining nodes in that cluster as a node leave/death. Such behavior only happens for remote clusters: the daemon will not leave the local cluster, as long as it can communicate with the cluster manager.

    Felipe
  • dlmcnabb
    dlmcnabb
    1012 Posts

    Re: GPFS callbacks - nodeLeave - remote cluster nodes

    ‏2013-01-04T19:22:03Z  
    The GPFS daemon will periodically scan the clusters to which the local node is joined, and then will cause the local node to leave remote clusters which have not been used "in a while" (I believe the interval is 5 mins). When the node leaves a cluster, that appears to the remaining nodes in that cluster as a node leave/death. Such behavior only happens for remote clusters: the daemon will not leave the local cluster, as long as it can communicate with the cluster manager.

    Felipe
    A remote node will only leave the main cluster if it has unmounted all filesystems from that main cluster. If there is any filesystem mounted on the client from the main cluster, it will not leave due to inactivity. However, if you are using autofs on the client to automatically mount a filesystem, autofs may unmount automatically due to inactivity. Then the node will eventually leave the cluster after another period times out.