Clock skew
A Ceph Monitor is out of quorum, and the ceph health detail command output
contains error messages similar to these:
mon.a (rank 0) addr 127.0.0.1:6789/0 is down (out of quorum)
mon.a addr 127.0.0.1:6789/0 clock skew 0.08235s > max 0.05s (latency 0.0045s)In addition, Ceph logs contain error messages similar to these:
2022-05-04 07:28:32.035795 7f806062e700 0 log [WRN] : mon.a 127.0.0.1:6789/0 clock skew 0.14s > max 0.05s
2022-05-04 04:31:25.773235 7f4997663700 0 log [WRN] : message from mon.1 was stamped 0.186257s in the future, clocks not synchronizedWhat This Means
The ‘clock skew` error message indicates that Ceph Monitors’ clocks are not synchronized. Clock synchronization is important because Ceph Monitors depend on time precision and behave unpredictably if their clocks are not synchronized.
mon_clock_drift_allowed parameter determines what disparity between the
clocks is tolerated. By default, this parameter is set to 0.05 seconds.mon_clock_drift_allowed without previous testing.
Changing this value might affect the stability of the Ceph Monitors and the Ceph Storage Cluster in
general.Possible causes of the clock skew error include network problems or problems
with chrony Network Time Protocol (NTP) synchronization if that is configured. In addition, time
synchronization does not work properly on Ceph Monitors deployed on virtual machines.
To Troubleshoot This Problem
-
Verify that your network works correctly. For more information, see Troubleshooting networking issues. If you use chrony for NTP, see Basic chrony NTP troubleshooting.
-
If you use a remote NTP server, consider deploying your own chrony NTP server on your network. For more information, see Using the Chrony Suite to Configure NTP chapter within the Configuring basic system settings guide within the Product Documentation for Red Hat Enterprise Linux for your OS version, on the Red Hat Customer Portal.
clock skew messages.Reference