ZooKeeper cannot create a lock

Edit online

In some situations, ZooKeeper cannot successfully create a lock because the ZooKeeper ensemble is offline. You can recover from this situation by troubleshooting the messages.log file for error messages that identify a ZooKeeper lock error, and verify the online status of the ZooKeeper nodes in your ensemble.

Symptoms

Any action in the Global Mailbox management tool that involves mailboxes or messages results in the following message:

A system error has occurred. Please contact your system administrator.

Causes

The ZooKeeper ensemble might be offline because of a configuration error, a network error, or because the ensemble is unable to maintain quorum:

Configuration error
An instance of ZooKeeper is not configured correctly.
Network connection error
A network connection to the ZooKeeper ensemble cannot be established.
ZooKeeper ensemble is unable to maintain quorum
The ZooKeeper ensemble is unable to maintain the minimum number of ZooKeeper nodes to coordinate distributed processes. To maintain quorum, a majority of the total number of ZooKeeper nodes in the ensemble must be online.

Environment

Windows and Linux®.

Diagnosing the problem

If you suspect that ZooKeeper was unable to create a lock, you can search the messages.log file for the CBXMB0005E error code or the CBXMB0011E error code at the time of the failure:
  1. Go to the <install_directory>/usr/servers/defaultServer/logs directory.
  2. Open the messages.log file.
  3. Examine the log for events with the following error message:

    • CBXMB0005E: Could not acquire a lock on "{0}"

      There is either a configuration issue or network connectivity issue with the ZooKeeper ensemble.

    • CBXMB0011E: Could not release a lock on "{0}"

      The distributed lock was unable to be released.

The format of the error message is as follows:

[time stamp] [thread ID] [logging class] [logging level] [error ID]: [error message]

The following example event includes the information that is logged by the messages.log file for an event that identifies when ZooKeeper cannot successfully create a lock:

Resolving the problem

If any error messages in the messages.log file indicate a distributed lock error, you can determine whether each ZooKeeper node in the ensemble is online or offline:
  1. From the command line, type the host name of the ZooKeeper node.
  2. Type ruok. If the ZooKeeper node is online and functioning properly, the ZooKeeper node returns the response imok.

    If the ZooKeeper node is not functioning properly, the ZooKeeper node does not return a response to the ruok command.