Configuring time synchronization across Cassandra nodes
Cassandra is sensitive to time stamps in synchronizing operations across nodes. The time on machines where Cassandra nodes are installed must be kept in tight synchronization with all nodes that communicate with or are referenced by Cassandra. This includes Sterling B2B Integrator nodes, Global Mailbox nodes, and ZooKeeper nodes. It is important to sync time across all node machines versus the absolute time.
Before you begin
The configuration uses a pool of two dedicated NTP servers that are synchronized with higher stratum NTP servers. The organizational Cassandra nodes are all configured to use this NTP pool. This is a minimal acceptable configuration for maintaining time synchronization across the Cassandra nodes.
There are other valid NTP configurations that might provide higher levels of synchronization accuracy but at potentially greater expense. These methodologies are well documented on the web.
You must set up an NTP server pool to continue with this configuration. The servers in this pool must be physical machines as virtual machines tend to have issues with clock accuracy in NTP applications. For more information, see Setting up NTP server pool.
About this task
To configure NTP on each machine where Cassandra is installed:
- Log in as root user to the machine where Cassandra is installed.
- To stop the NTPD service, type service ntpd stop.
- Open the /etc/ntp.conf file for editing.
If the Cassandra node is installed on a virtual machine, add the text,
tinker panic 0, as the first line in the ntp.conf file.For example,
tinker panic 0Tip:
tinker panic 0is an NTP directive that instructs the NTPD to not panic and exit if the original clock behaves randomly. It must always appear first in the ntp.conf file.
If the node is installed on a virtual machine, check if any section in the
ntpd.conf file contains the comment
undisciplined local clockand remove the section.Important: NTP must not be configured to use the hardware clock of the virtual machine as a fallback due to the potential inaccuracies of a virtual clock.
Verify whether the ntp.conf file has server entries similar to the
server <pool server1> iburst prefer
server <pool server2> iburstIf the file does not have such entries, add the list of servers (in the preceding format), which are in your NTP server pool and are configured to draw and synchronize with your highest strata NTP servers.Important: The
preferparameter ensures that all the Cassandra node machines use the same time server, thereby staying relatively synchronized with each other.
- Save the ntp.conf file.
To force an initial sync up, type ntpd -gq.
You can also use the ntpd -gq command to force an immediate sync up, in other situations. In such cases, you must stop the NTPD service before using the command, and restart the service after running the command.
- To restart the NTPD service, type service ntpd start.
To ensure that the service runs on start up, type chkconfig ntpd on.
Attention: Periodically check all nodes and make sure that the time on each node is in sync to ensure that the NTPD service is working properly.