IV97240: CAA:SLOW GOSSIP TRANSMISSION ON BOOT MAY CAUSE PARTIONED CLUSTER

APAR status

• Closed as program error.

Error description

• After rebooting, one of the cluster nodes was not able
  to join  the cluster environment.
  - lscluster from each node did not show the other node as
    UP.
  - After system reboot, the syslog.caa log file showed
    a delay of over 2 minutes in getting the first
  multicast
    gossip packet.
    When this delay occurs, the node creates its own
  cluster
    ignoring the other node which is already up.
    This leads to a split-brain / partitioned cluster
    in the CAA environment.
  

Local fix

• n/a
  

Problem summary

• If a node is rebooted and, due to network issues, fails to
  receive a gossip from other UP nodes within twice of
  node_down_delay, could join by itself, causing a split-brain
  (partitioned cluster).
  

Problem conclusion

• There is a gate in which all initial clusterwide lock requests
  should consider the count of nodes heartbeating to the
  repository in addition to those gossiping over network.
  There was a hole in the gate and the fix closes it.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  IV97148

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  AIX V7.1

• Fixed component ID

  5765H4000

Applicable component levels

• R710 PSY U872165

     UP17/10/13 I 1000