Configuring time synchronization across Cassandra nodes

Cassandra is sensitive to time stamps in synchronizing operations across nodes. The time on machines where Cassandra nodes are installed must be kept in tight synchronization with all nodes that communicate with or are referenced by Cassandra. This includes Sterling B2B Integrator nodes, Global Mailbox nodes, and ZooKeeper nodes. It is important to sync time across all node machines versus the absolute time.

Before you begin

The configuration uses a pool of two dedicated NTP servers that are synchronized with higher stratum NTP servers. The organizational Cassandra nodes are all configured to use this NTP pool. This is a minimal acceptable configuration for maintaining time synchronization across the Cassandra nodes.

There are other valid NTP configurations that might provide higher levels of synchronization accuracy but at potentially greater expense. These methodologies are well documented on the web.

You must set up an NTP server pool to continue with this configuration. The servers in this pool must be physical machines as virtual machines tend to have issues with clock accuracy in NTP applications. For more information, see Setting up NTP server pool.

About this task

To configure NTP on each machine where Cassandra is installed:

Procedure

  1. Log in as root user to the machine where Cassandra is installed.
  2. To stop the NTPD service, type service ntpd stop.
  3. Open the /etc/ntp.conf file for editing.
  4. If the Cassandra node is installed on a virtual machine, add the text, tinker panic 0, as the first line in the ntp.conf file. For example,

    # ntpd.conf

    tinker panic 0

    Tip: tinker panic 0 is an NTP directive that instructs the NTPD to not panic and exit if the original clock behaves randomly. It must always appear first in the ntp.conf file.
  5. If the node is installed on a virtual machine, check if any section in the ntpd.conf file contains the comment undisciplined local clock and remove the section.
    Important: NTP must not be configured to use the hardware clock of the virtual machine as a fallback due to the potential inaccuracies of a virtual clock.
  6. Verify whether the ntp.conf file has server entries similar to the following entries:

    server <pool server1> iburst prefer

    server <pool server2> iburst

    If the file does not have such entries, add the list of servers (in the preceding format), which are in your NTP server pool and are configured to draw and synchronize with your highest strata NTP servers.
    Important: The prefer parameter ensures that all the Cassandra node machines use the same time server, thereby staying relatively synchronized with each other.
  7. Save the ntp.conf file.
  8. To force an initial sync up, type ntpd -gq.

    You can also use the ntpd -gq command to force an immediate sync up, in other situations. In such cases, you must stop the NTPD service before using the command, and restart the service after running the command.

  9. To restart the NTPD service, type service ntpd start.
  10. To ensure that the service runs on start up, type chkconfig ntpd on.
    Attention: Periodically check all nodes and make sure that the time on each node is in sync to ensure that the NTPD service is working properly.
Parent topic: Configuring time synchronization