Clock skew

Edit online

Understand and troubleshoot clock skew.

A Ceph Monitor is out of quorum, and the ceph health detail command output contains error messages similar to the following examples:
mon.a (rank 0) addr 127.0.0.1:6789/0 is down (out of quorum)
mon.a addr 127.0.0.1:6789/0 clock skew 0.08235s > max 0.05s (latency 0.0045s)
In addition, Ceph logs contain error messages similar to the following examples:
2022-05-04 07:28:32.035795 7f806062e700 0 log [WRN] : mon.a 127.0.0.1:6789/0 clock skew 0.14s > max 0.05s
2022-05-04 04:31:25.773235 7f4997663700 0 log [WRN] : message from mon.1 was stamped 0.186257s in the future, clocks not synchronized

What this means

The clock skew error message indicates that Ceph Monitors’ clocks are not synchronized. Clock synchronization is important because Ceph Monitors depend on time precision and behave unpredictably if their clocks are not synchronized.

The mon_clock_drift_allowed parameter determines what disparity between the clocks is tolerated. By default, this parameter is set to 0.05 seconds.
Important: Do not change the default value of mon_clock_drift_allowed without previous testing. Changing this value might affect the stability of the Ceph Monitors and the Ceph Storage Cluster in general.

Possible causes of the clock skew error include network problems or problems with chrony Network Time Protocol (NTP) synchronization if that is configured. In addition, time synchronization does not work properly on Ceph Monitors deployed on virtual machines.

For more information, see:

Troubleshooting this problem

Edit online
Note: Ceph evaluates time synchronization every five minutes, resulting in delays between fixing the problem and clearing the clock skew messages.
  1. Verify that your network works correctly. For more information, see Troubleshooting networking issues. If you use chrony for NTP, see Basic chrony NTP troubleshooting.
  2. If you use a remote NTP server, consider deploying your own chrony NTP server on your network. For more information, see Using the Chrony Suite to Configure NTP chapter within the Configuring basic system settings guide within the Product Documentation for Red Hat Enterprise Linux for your OS version, on the Red Hat Customer Portal.