Topic
  • 8 replies
  • Latest Post - ‏2013-11-15T14:16:25Z by FelipeKnop
SystemAdmin
SystemAdmin
2092 Posts

Pinned topic GPFS Network Down issues

‏2012-11-24T14:14:57Z |
my GPFS 2 node Cluster with TIE Breaker disks fails to failover for Network Unavailability.

Any help would be highly appreciated

Configuration data for cluster PROD_EMS.EMS_NODE_1:

myNodeConfigNumber 1
clusterName PROD_EMS.EMS_NODE_1
clusterId 771243877078702085
autoload no
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
tiebreakerDisks GBS_TIE_DISK_1;NEC_TIE_DISK_1
adminMode allToAll

File systems in cluster PROD_EMS.EMS_NODE_1:

/dev/emsfs

GPFS cluster information
========================
GPFS cluster name: PROD_EMS.EMS_NODE_1
GPFS cluster id: 771243877078702085
GPFS UID domain: PROD_EMS.EMS_NODE_1
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp

GPFS cluster configuration servers:

Primary server: EMS_NODE_1
Secondary server: EMS_NODE_2

Node Daemon node name IP address Admin node name Designation

1 EMS_NODE_1 X.X.X.X EMS_NODE_1 quorum-manager
2 EMS_NODE_2 Y.Y.Y.Y EMS_NODE_2 quorum-manager

  1. mmlsmgr
file system manager node

------------------
emsfs X.X.X.X (EMS_NODE_1)

Cluster manager node: X.X.X.X (EMS_NODE_1)

For Network failover testing I had brought down the Cluster Manager IP.

ifconfig en0 down

After some time, the other node doesnt take up the Cluster Manager & File System Manager role. It just unmount the GPFS FS with the following message

Wed 21 Nov 12:07:40 2012: finished mounting /dev/emsfs
Sat Nov 24 14:08:21.814 2012: Node X.X.X.X (EMS_NODE_1) lease renewal is overdue. Pinging to check if it is alive
Sat Nov 24 14:08:21.815 2012: Lease is overdue. Probing cluster PROD_EMS.EMS_NODE_1
Sat Nov 24 14:08:22.188 2012: Waiting for challenge to be responded during disk election
Sat Nov 24 14:08:53.190 2012: Challenge response received; canceling disk election.
Sat Nov 24 14:08:53.294 2012: Waiting for challenge to be responded during disk election
Sat Nov 24 14:09:24.299 2012: Challenge response received; canceling disk election.
Sat Nov 24 14:09:24.394 2012: Waiting for challenge to be responded during disk election
Sat Nov 24 14:09:55.404 2012: Challenge response received; canceling disk election.
Sat Nov 24 14:09:55.407 2012: Lost membership in cluster PROD_EMS.EMS_NODE_1. Unmounting file systems.
Sat Nov 24 14:09:55.569 2012: Close connection to X.X.X.X EMS_NODE_1 <c0n0> (Node failed)

LABEL: MMFS_PHOENIX
IDENTIFIER: AB429E38

Date/Time: Tue 20 Nov 19:28:14 2012
Sequence Number: 22250
Machine Id: 00F764A04C00
Node Id: gbsttap078
Class: S
Type: UNKN
WPAR: Global
Resource Name: mmfs

Description
DIAGNOSTIC REPORT

Probable Causes
SOFTWARE PROGRAM

User Causes
INFORMATION ONLY

Recommended Actions
NO ACTION NECESSARY

Detail Data
EVENT CODE
11245001
REASON CODE
668
ERROR DATA
Lost membership in cluster PROD_EMS.EMS_NODE_1. Unmounting file systems.

While rebooting the server, The failover happens properly

Tue Nov 20 19:28:56.755 2012: Node X.X.X.X (EMS_NODE_1) is being expelled due to expired lease.
Tue Nov 20 19:28:56.757 2012: Recovering nodes: X.X.X.X
Tue Nov 20 19:28:56.808 2012: Recovered 1 nodes for file system emsfs.
Tue Nov 20 19:29:11.858 2012: Responding to disk challenge: response: 197. Error code: 0.
Tue Nov 20 19:29:43.561 2012: Accepted and connected to X.X.X.X EMS_NODE_1 <c0n0>
Tue Nov 20 19:31:13.440 2012: Connection from 127.0.0.1 timed out
Wed Nov 21 11:21:28.680 2012: Command: unmount emsfs forced
Wed Nov 21 11:21:28.690 2012: Command: err 0: unmount emsfs
Wed Nov 21 11:21:33.696 2012: Node Y.Y.Y.Y (EMS_NODE_2) resigned as manager for emsfs.
Wed Nov 21 11:21:33.701 2012: Node X.X.X.X (EMS_NODE_1) appointed as manager for emsfs.
Wed Nov 21 11:21:33.890 2012: Node X.X.X.X (EMS_NODE_1) completed take over for emsfs.
Wed Nov 21 11:21:33.897 2012: mmfsd64 is shutting down.
Wed Nov 21 11:21:33.899 2012: Reason for shutdown: Normal shutdown
Wed 21 Nov 11:21:35 2012: mmcommon mmfsdown invoked. Subsystem: mmfs Status: down
Wed 21 Nov 11:21:35 2012: mmcommon: Unmounting file systems ...
Updated on 2012-12-27T13:01:35Z at 2012-12-27T13:01:35Z by SystemAdmin
  • FelipeKnop
    FelipeKnop
    33 Posts

    Re: GPFS Network Down issues

    ‏2012-11-25T21:51:39Z  
    This is normal behavior when tiebreaker disks are used. That is, network failure at the cluster manager node will not trigger cluster manager failover.

    The behavior can be altered by writing a tiebreaker callback. The callback is activated via the

    mmaddcallback tiebreakerCheck other options

    command.

    According to the man page:

    http://...
    tiebreakerCheck
    Triggered when a quorum node detects loss of network connectivity but before GPFS runs the algorithm that decides if the node will remain in the cluster. This event is generated only in configurations that use quorum nodes with tiebreaker disks. It is also triggered on the cluster manager periodically by a challenge-response thread to verify that the node can still continue as cluster manager.
    http://...
    Felipe
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: GPFS Network Down issues

    ‏2012-11-26T01:03:33Z  
    This is normal behavior when tiebreaker disks are used. That is, network failure at the cluster manager node will not trigger cluster manager failover.

    The behavior can be altered by writing a tiebreaker callback. The callback is activated via the

    mmaddcallback tiebreakerCheck other options

    command.

    According to the man page:

    http://...
    tiebreakerCheck
    Triggered when a quorum node detects loss of network connectivity but before GPFS runs the algorithm that decides if the node will remain in the cluster. This event is generated only in configurations that use quorum nodes with tiebreaker disks. It is also triggered on the cluster manager periodically by a challenge-response thread to verify that the node can still continue as cluster manager.
    http://...
    Felipe
    Thanks for the response Felipe...

    Which do you think as better for my configuration? Having a third node or writing mmaddcallback

    I am a newbie and just followed the best steps available in the net to configure so far.... I have presented 2 data & 2 tie breaker disks from 2 different Storage hosted in 2 different DC. Also I have replicated both Data&Metadata having 2 copies

    flag value description

    ------------------------
    -f 16384 Minimum fragment size in bytes
    -i 512 Inode size in bytes
    -I 16384 Indirect block size in bytes
    -m 2 Default number of metadata replicas
    -M 2 Maximum number of metadata replicas
    -r 2 Default number of data replicas
    -R 2 Maximum number of data replicas
    -j cluster Block allocation type
    -D nfs4 File locking semantics in effect
    -k all ACL semantics in effect
    -n 32 Estimated number of nodes that will mount file system
    -B 524288 Block size
    -Q user;group;fileset Quotas enforced
    none Default quotas enabled
    --filesetdf no Fileset df enabled?
    -V 12.10 (3.4.0.7) File system version
    --create-time Fri Oct 19 18:17:33 2012 File system creation time
    -u yes Support for large LUNs?
    -z no Is DMAPI enabled?
    -L 4194304 Logfile size
    -E yes Exact mtime mount option
    -S no Suppress atime mount option
    -K whenpossible Strict replica allocation option
    --fastea yes Fast external attributes enabled?
    --inode-limit 133120 Maximum number of inodes
    -P system Disk storage pools in file system
    -d GBSDISK_1;NECDISK_1 Disks in file system
    -A yes Automatic mount option
    -o none Additional mount options
    -T /data/tibco Default mount point
    --mount-priority 0 Mount priority
  • FelipeKnop
    FelipeKnop
    33 Posts

    Re: GPFS Network Down issues

    ‏2012-11-26T03:27:00Z  
    If a 3rd node is available, and if it's reasonably guaranteed that at least two of them will always be available then reverting to a normal node quorum (that is, removing the tiebreaker
    disk configuration) is probably the recommended approach. When using a normal node quorum,
    you should no longer see a network-isolated node remaining the cluster manager.

    Felipe
  • renarg
    renarg
    163 Posts

    Re: GPFS Network Down issues

    ‏2012-11-26T17:27:17Z  
    If a 3rd node is available, and if it's reasonably guaranteed that at least two of them will always be available then reverting to a normal node quorum (that is, removing the tiebreaker
    disk configuration) is probably the recommended approach. When using a normal node quorum,
    you should no longer see a network-isolated node remaining the cluster manager.

    Felipe
    Hallo Felipe,

    one question, the callback you mentioned is only for 3.5? Is these callback also valid for 3.4.0.17? I don't find these in the manpage in 3.4 release. Thanks for your answer.
    Regards Renar
  • FelipeKnop
    FelipeKnop
    33 Posts

    Re: GPFS Network Down issues

    ‏2012-11-27T14:05:42Z  
    • renarg
    • ‏2012-11-26T17:27:17Z
    Hallo Felipe,

    one question, the callback you mentioned is only for 3.5? Is these callback also valid for 3.4.0.17? I don't find these in the manpage in 3.4 release. Thanks for your answer.
    Regards Renar
    I believe the callback should be available on 3.4.0.17 as well.

    Felipe
  • SystemAdmin
    SystemAdmin
    2092 Posts

    Re: GPFS Network Down issues

    ‏2012-12-27T13:01:35Z  
    I believe the callback should be available on 3.4.0.17 as well.

    Felipe
    Thanks Felipe for your help...

    The Callback scenarios were not helping basically. Hence, declared a Quorom-Manager new node from a different hardware to avoid localized network Problems. This solved the issue

    Thanks
    Ram
  • sdas.aix
    sdas.aix
    1 Post

    Re: GPFS Network Down issues

    ‏2013-11-13T01:56:46Z  
    I believe the callback should be available on 3.4.0.17 as well.

    Felipe

    Hi Felipe,

    Our two node GPFS cluster is facing exactly the same problem. During network outage of cluster manager node, the surviving node is unable to take over. Unfortunately, we cannot have a third node to mitigate the problem. So the only option I see is the second option you have explained above, that is, to add a callback.

    In the callback command, I ping a third DC server so that if command is unsuccessful, the node can resign as cluster manager. I have added the callback as below:

    mmaddcallback pingD31 --event tiebreakerCheck --command /tmp/gpfs/pingD31 --sync --onerror quorumLoss

    (Above /tmp/gpfs/pingD31 is script that pings the third DC server 5 times; return 0 if successful and 1 if unsuccessful)

    The problem is, now I get below messages in the log during mmstartup and the startup does not go ahead from this:

    Wed Nov 13 01:48:15.010 2013: GPFS: 6027-2742 CallExitScript: exit script /tmp/gpfs/pingD31 on event tiebreakerCheck returned code 0X0, quorumloss.
    Wed Nov 13 01:49:46.569 2013: GPFS: 6027-2742 CallExitScript: exit script /tmp/gpfs/pingD31 on event tiebreakerCheck returned code 0X0, quorumloss.
    Wed Nov 13 01:51:18.126 2013: GPFS: 6027-2742 CallExitScript: exit script /tmp/gpfs/pingD31 on event tiebreakerCheck returned code 0X0, quorumloss.

    It would be very helpful if you can throw some more light on how to mitigate the problem using this method.

     

    Many Thanks,

    Das

     

    Updated on 2013-11-13T01:59:23Z at 2013-11-13T01:59:23Z by sdas.aix
  • FelipeKnop
    FelipeKnop
    33 Posts

    Re: GPFS Network Down issues

    ‏2013-11-15T14:16:25Z  
    • sdas.aix
    • ‏2013-11-13T01:56:46Z

    Hi Felipe,

    Our two node GPFS cluster is facing exactly the same problem. During network outage of cluster manager node, the surviving node is unable to take over. Unfortunately, we cannot have a third node to mitigate the problem. So the only option I see is the second option you have explained above, that is, to add a callback.

    In the callback command, I ping a third DC server so that if command is unsuccessful, the node can resign as cluster manager. I have added the callback as below:

    mmaddcallback pingD31 --event tiebreakerCheck --command /tmp/gpfs/pingD31 --sync --onerror quorumLoss

    (Above /tmp/gpfs/pingD31 is script that pings the third DC server 5 times; return 0 if successful and 1 if unsuccessful)

    The problem is, now I get below messages in the log during mmstartup and the startup does not go ahead from this:

    Wed Nov 13 01:48:15.010 2013: GPFS: 6027-2742 CallExitScript: exit script /tmp/gpfs/pingD31 on event tiebreakerCheck returned code 0X0, quorumloss.
    Wed Nov 13 01:49:46.569 2013: GPFS: 6027-2742 CallExitScript: exit script /tmp/gpfs/pingD31 on event tiebreakerCheck returned code 0X0, quorumloss.
    Wed Nov 13 01:51:18.126 2013: GPFS: 6027-2742 CallExitScript: exit script /tmp/gpfs/pingD31 on event tiebreakerCheck returned code 0X0, quorumloss.

    It would be very helpful if you can throw some more light on how to mitigate the problem using this method.

     

    Many Thanks,

    Das

     

    Das,

    Though it may not be totally obvious from the output you include, I believe the /tmp/gpfs/pingD31 script returned a non-zero value, and that is triggering the quorum loss.

    Could you check:

    - if the /tmp/gpfs/pingD31 script has appropriate execute permissions

    - if the script starts with the

                  #!/usr/bin/ksh

    line? (this line may be needed)

    - if the script is indeed returning 0 on success?

     

    To get a bit more information on the problem, one suggestion is to add debug statements into the script and redirect them into a temporary file, especially indicating what the exit value is about to be returned by the script. Then, once the script runs, the debug statements can be examined in the temporary file. (I suggest saving the current date, so we'll know when the script was run)

    Thanks,

    Felipe