ZooKeeper watchdog

ZooKeeper watchdog monitors the health of the Apache ZooKeeper ensemble, and when quorum is lost, it restarts the ZooKeeper process with a reduced ensemble of the surviving nodes, thereby restoring the quorum.

ZooKeeper always operates in the Cassandra equivalent of QUORUM consistency, because consistency levels cannot be configured in ZooKeeper. A ZooKeeper ensemble must always have N/2+1 nodes operating, where N is the total number of nodes. For example, if the total number of nodes is 6, then 6/2+1=4, that is, four ZooKeeper nodes must be operational in the ensemble.

ZooKeeper watchdog is a process, which ensures that ZooKeeper limitation of not having the provision to configure consistency levels does not hamper protocol transactions and administrator user interface tasks in another data center. The watchdog monitors the health of the ZooKeeper ensemble, and when quorum is lost, it restarts the ZooKeeper process with a reduced ensemble of the surviving nodes, thereby restoring the quorum. If for some reason, nodes in the reduced ensemble are lost, it restarts the ZooKeeper process again with a further reduced ensemble. The watchdog manages the ZooKeeper ensemble until there is at least one operational ZooKeeper node.

When quorum loss is detected, the watchdog waits for a configurable threshold time (maxQuorumLossTolerance property) before restarting the ZooKeeper process with a reduced ensemble. During this threshold time, ZooKeeper operations fail, because there is no quorum. After the threshold time elapses, the ZooKeeper process is restarted with a reduced ensemble of surviving nodes, and ZooKeeper operations start to succeed shortly thereafter.

After starting a reduced ensemble, the watchdog monitors the failed ZooKeeper nodes, and as the failed nodes come back online, the watchdog restarts the ZooKeeper process with a new ensemble that includes the nodes that came back online. The watchdog waits for the time specified for the ensembleRecoveryDelay property in the watchdog.properties file before starting the ZooKeeper process with the new ensemble.

When operating with a reduced ensemble, there are fewer ZooKeeper nodes to handle the same load that the full ensemble used to handle. So, when operating with a reduced ensemble, ZooKeeper operations might take longer or timeout, especially if the surviving ZooKeeper nodes are on a remote data center, and there is network latency.

There might be situations where the network link between data centers goes down, but the ZooKeeper processes continue running. In such cases, the watchdog starts multiple separate ZooKeeper ensembles, one on each data center, consisting of the ZooKeeper nodes of each data center. If this situation arises, the lock for the same ZooKeeper path might potentially be granted simultaneously on each data center, allowing mutual exclusion of global locks, and resulting in events getting posted to the messaging queue twice.

The Installation Manager deploys ZooKeeper watchdog during the installation of ZooKeeper. The watchdog is deployed on the servers where ZooKeeper is installed. When the watchdog is deployed, the <install_Dir>/zookeeper/watchdog directory is created in the ZooKeeper installation directory. The watchdog is started when you run the startGM.sh script in the <install_Dir>/MailboxUtilities/bin directory. In case you need to exclusively start or stop the watchdog process, you must run the startGMCoordinateWatchdog.sh and stopGMCoordinateWatchdog.sh scripts in the <install_Dir>/MailboxUtilities/bin directory.

The Installation Manager modifies the watchdog.properties file to set the fully qualified path to the ZooKeeper startup and shutdown scripts. The other watchdog properties must be kept as defaults. The watchdog.properties file is in the <install_Dir>/zookeeper/watchdog/conf directory. To edit the properties, you must open the properties file in a text editor and make the necessary changes.