QRadar: Troubleshooting chrony errors and "Time Synchronization to a primary host or Console has failed"

Troubleshooting

Problem

In QRadar®, the chrony daemon is used to synchronize time on QRadar manged hosts to the Console. The article instructs users how to force the Console to time synchronize in that latest QRadar versions.

Symptom

The QRadar Dashboard displays repeated System Notification messages: "Time Synchronization to a primary host or Console has failed".
QRadar logs report a chrony error in /var/log/qradar.log.
Administrators see offenses created with an end time that occurs in the future.
Search results that are erroneous or do not return results from the managed host when you know the event exists.

Cause

The Console appliance is responsible for maintaining time synchronization for all managed hosts in the deployment. Every 10 minutes, managed hosts request a synchronization to the Console time and if chronyd (UDP 123) request for time update is unsuccessful, then a system notification is generated to administrators. If the next request for an update succeeded, the time is synchronized and the previous alert can be ignored.

Consecutive notifications last 10 minutes each and sustained notifications can last over an hour. Consecutive and sustained notifications for unsuccessful updates are most likely caused by network issues or high network load and not hardware issues.

Time synchronization to the Console is critical to QRadar. Without time synchronization searches, reports, and offenses might not complete successfully or return the expected data. Time synchronization can cause the user interface to display incomplete or erroneous search results.

Diagnosing The Problem

Administrators who receive multiple system notifications related to time synchronization can review the firewall port or look for errors on managed host to determine whether errors are reported in the logs.

If the system notification does not list the IP address of the affected managed host, administrators can review /var/log/qradar.log on the managed hosts for error messages. For example,

HOSTNAME [time_sync]: [ERROR] [NOT:0150003100][IPADDRESS]   
Time Synchronization to Console has failed - chrony error

To identify managed hosts with time synchronization issues:

Use SSH to log in to the Console as root user.
Type the command:
```
/opt/qradar/support/all_servers.sh -k 'grep -i "chrony" /var/log/qradar.log'
```
For example, error messages from managed host display any chrony errors from /var/log/qradar.log.

Results
A report with hostname and IP address of affected managed hosts is generated for each appliance experiencing a time synchronization issue.

Resolving The Problem

Administrators can use these steps to troubleshoot time synchronization. Hosts with encryption might require verification that TCP port 12500 is listening between the Console and the managed host. Port 323 is an internal command port used by the chrony daemon and does not need to be open between hosts.

Unencrypted Managed Host

Use SSH to log in to the Console as root user.

Confirm the chronyd service is running on the Console by typing:

systemctl status chronyd

● chronyd.service - NTP client/server     Loaded: loaded 
(/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)    
Drop-In: /etc/systemd/system/chronyd.service.d             └─qradar.conf     Active: active (running) 
since Mon 2020-08-03 13:22:26 EDT; 31min ago       Docs: man:chronyd(8)             man:chrony.conf(5)   
Main PID: 2980 (chronyd)      Tasks: 1     Memory: 4.0K     CGroup: /system.slice/chronyd.service             
└─2980 /usr/sbin/chronyd

Force the managed host with the error to synchronize with the Console by typing:

/opt/qradar/bin/time_sync.sh

If the time sync is successful, the system clock offset is displayed.

2020-05-27T15:55:46Z chronyd version 3.2 starting (+CMDMON +NTP +REFCLOCK
 +RTC +PRIVDROP +SCFILTER +SECHASH +SIGND +ASYNCDNS +IPV6 +DEBUG)  2020-05-27T15:55:50Z System clock wrong 
by -0.005335 seconds (step)  2020-05-27T15:55:50Z chronyd exiting

If the time sync is unsuccessful, administrators can see a similar message:
Note: If the sync is unsuccessful, you need to use the all_servers.sh command so the command can run all appliances in the deployment.

/opt/qradar/support/all_servers.sh -k "/opt/qradar/bin/time_sync.sh"

2020-06-26T17:31:20Z chronyd version
  3.2 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SECHASH +SIGND +ASYNCDNS +IPV6 +DEBUG)  
  2020-06-26T17:31:30Z No suitable source for synchronisation  2020-06-26T17:31:30Z chronyd exiting

Check for chronyd errors in the logs.

journalctl -xe -u chronyd

-- Logs begin at Wed 2020-08-19 11:28:56 EDT, end at Wed 2020-08-19 12:20:55 EDT.
 --  Aug 19 11:29:20 QRadar732Console.example.com systemd[1]: Starting NTP client/server...  -- Subject: Unit 
chronyd.service has begun start-up  -- Defined-By: systemd  -- Support: http://lists.freedesktop.org/mailman/listinfo/
systemd-devel  --  -- Unit chronyd.service has begun starting up.  Aug 19 11:29:20 QRadar732Console.example.com 
chronyd[2964]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS 
+S  Aug 19 11:29:20 QRadar732Console.example.com chronyd[2964]: commandkey directive is no longer supported  Aug 
19 11:29:20 QRadar732Console.example.com chronyd[2964]: generatecommandkey directive is no longer supported  Aug 19
 11:29:20 QRadar732Console.example.com chronyd[2964]: Frequency 0.000 +/- 1000000.000 ppm read from /var/lib/chrony/drift
  Aug 19 11:29:21 QRadar732Console.example.com systemd[1]: Started NTP client/server.  -- Subject: Unit chronyd.service has
 finished start-up  -- Defined-By: systemd  -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
  --  -- Unit chronyd.service has finished starting up.  --  -- The start-up result is done.

Verify that the time synchronization crojobs are running on all hosts by typing:

/opt/qradar/support/all_servers.sh -Ck "crontab -l | grep time_sync"

 x.x.0.84 -> 732APPhost.example.com
  Appliance Type: 4000    Product Version: 2020.3.0.20200716115107   14:27:50 up  5:31,  0 users,  load average
: 4.68, 3.88, 3.39  ------------------------------------------------------------------------  
*/10 * * * * /opt/qradar/bin/time_sync.sh    x.x.0.80 -> QRadar732Console.example.com  Appliance Type: 3199    
Product Version: 2020.3.0.20200716115107   14:27:51 up  5:32,  1 user,  load average: 14.10, 17.59, 16.88  
------------------------------------------------------------------------  */10 * * * * /opt/qradar/bin/time_sync.sh

Verify that that UDP port 123 can connect to the Console from the managed hosts by typing.

/opt/qradar/support/all_servers.sh "nc -z -u -v <Console_IP_address> 123"

x.x.0.84 -> 732APPhost.example.com
  Appliance Type: 4000    Product Version: 2020.3.0.20200716115107   16:38:34 up  3:16,  0 users,  load average:
 1.86, 1.36, 1.27  ------------------------------------------------------------------------  Ncat: Version 7.50 
( https://nmap.org/ncat )  Ncat: Connected to x.x.0.80:123.  Ncat: UDP packet sent successfully  
Ncat: 1 bytes sent, 0 bytes received in 2.01 seconds.

If this test fails to verify that the port is not blocked by a firewall rule.

Use the iptables command to verify UDP port 123 is open on the Console for the managed host.

iptables -L -n | grep 123

ACCEPT     udp  --  x.x.0.84         0.0.0.0/0            udp dpt:123

Verify that the Console can connect to the NTP server.

 ntpdate -q <NTP_server_IP_address>

server NTP_server_IP_address, stratum 1, offset 0.908444, delay 0.26669
  19 Aug 12:16:39 ntpdate[15214]: step time server NTP_server_IP_address offset 0.908444 sec

Verify that UDP port 123 is listening on the Console, type:

netstat -nap | egrep ':123'

0 0 0.0.0.0:123 0.0.0.0:* 1624/chronyd udp6 0 0 :::123 :::*

Another command to check chrony status is "watch chronyc tracking", which you can use on the managed hosts to see whether there is any blocked communications.

Every 2.0s: chronyc tracking
Reference ID    : 7F7F0101 ()
Stratum         : 10
Ref time (UTC)  : Mon Sep 13 14:31:26 2021
System time     : 0.000000118 seconds slow of NTP time
Last offset     : +0.000000000 seconds
RMS offset      : 0.000000000 seconds
Frequency       : 0.757 ppm slow
Residual freq   : +0.000 ppm
Skew            : 0.000 ppm
Root delay      : 0.000000000 seconds
Root dispersion : 0.000000000 seconds
Update interval : 0.0 seconds
Leap status     : Normal

Encrypted Managed Host

For managed hosts that use encryption, TCP port 12500 must be open between the Console and the encrypted managed host.

Use SSH to log in to the Console as root user.

Verify that the connection between the managed host and Console is encrypted.

Option 1, use the command :

/opt/qradar/support/all_servers.sh "/opt/qradar/bin/myver -tunnel"

x.x.0.67 -> qrm.example.com  
 Appliance Type: 700     Product Version: 2019.18.4.20200629201233   11:42:13 up 9 days, 19:01,  0 users, 
 load average: 0.27, 0.44, 0.54  ------------------------------------------------------------------------ 
 true    x.x.0.81 -> qflow.example.com  Appliance Type: 1299    Product Version: 2019.18.4.20200629201233
 11:42:13 up 9 days, 18:25,  0 users,  load average: 0.27, 0.28, 0.38  
 ------------------------------------------------------------------------  false

Results Encrypted managed hosts return a value true.

Option 2, Use the UI to verify that the connection is encrypted.
1. Log in to the QRadar UI as admin user.
2. Click Admin tab > System and License Management.
3. Select a Managed Host from the list of hosts.
4. Click Deployment Actions > Edit Host.
  Note: Do not check Encrypt Host Connections on a Console.
  
  Results The Encrypt Host Connections box is checked.

After confirming that the ports are not blocked, force the managed host with the error to synchronize with the Console by typing:

/opt/qradar/bin/time_sync.sh

If the time sync is successful, the system clock offset is displayed.

2020-05-27T15:55:46Z chronyd version 3.2 starting (+CMDMON +NTP +REFCLOCK
 +RTC +PRIVDROP +SCFILTER +SECHASH +SIGND +ASYNCDNS +IPV6 +DEBUG)  2020-05-27T15:55:50Z System clock wrong 
by -0.005335 seconds (step)  2020-05-27T15:55:50Z chronyd exiting

If the time sync is unsuccessful, administrators see a similar message.
Note: If the sync is unsuccessful, you need to use the all_servers.sh command so the command can run all appliances in the deployment.

/opt/qradar/support/all_servers.sh -k "/opt/qradar/bin/time_sync.sh"

2020-06-26T17:31:20Z chronyd version
 3.2 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SECHASH +SIGND +ASYNCDNS +IPV6 +DEBUG)  
2020-06-26T17:31:30Z No suitable source for synchronisation  2020-06-26T17:31:30Z chronyd exiting

Check for chronyd errors in the logs.

journalctl -xe -u chronyd

-- Logs begin at Wed 2020-08-19 11:28:56 EDT, end at Wed 2020-08-19 12:20:55 
EDT. --  Aug 19 11:29:20 QRadar732Console.example.com systemd[1]: Starting NTP client/server...  -- Subject: 
Unit chronyd.service has begun start-up  -- Defined-By: systemd  -- Support: http://lists.freedesktop.org/
mailman/listinfo/systemd-devel  --  -- Unit chronyd.service has begun starting up.  Aug 19 11:29:20 
QRadar732Console.ibm.com chronyd[2964]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP
 +SCFILTER +SIGND +ASYNCDNS +S  Aug 19 11:29:20 QRadar732Console.example.com chronyd[2964]: commandkey directive
 is no longer supported  Aug 19 11:29:20 QRadar732Console.example.com chronyd[2964]: generatecommandkey directive
 is no longer supported  Aug 19 11:29:20 QRadar732Console.example.com chronyd[2964]: Frequency 0.000 +/- 1000000.00
0 ppm read from /var/lib/chrony/drift  Aug 19 11:29:21 QRadar732Console.example.com systemd[1]: Started NTP client/server.
  -- Subject: Unit chronyd.service has finished start-up  -- Defined-By: systemd  
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel  --  -- 
Unit chronyd.service has finished starting up.  --  -- The start-up result is done.

Confirm the chronyd service is running on the Console:

systemctl status chronyd

● chronyd.service - NTP client/server     Loaded: loaded (/usr/lib/systemd/system
/chronyd.service; enabled; vendor preset: enabled)    Drop-In: /etc/systemd/system/chronyd.service.d             
└─qradar.conf     Active: active (running) since Mon 2020-08-03 13:22:26 EDT; 31min ago       Docs: man:chronyd(8)
  man:chrony.conf(5)   Main PID: 2980 (chronyd)      Tasks: 1     Memory: 4.0K     CGroup: /system.slice/chronyd.service
             └─2980 /usr/sbin/chronyd

Verify that the time synchronization cron jobs are running on all hosts:

/opt/qradar/support/all_servers.sh -Ck "crontab -l | grep time_sync"

 x.x.0.84 -> 732APPhost.ibm.com  
  Appliance Type: 4000    Product Version: 2020.3.0.20200716115107   14:27:50 up  5:31,  0 users,  load average
: 4.68, 3.88, 3.39  ------------------------------------------------------------------------  
*/10 * * * * /opt/qradar/bin/time_sync.sh    x.x.0.80 -> QRadar732Console.example.com  Appliance Type: 3199    
Product Version: 2020.3.0.20200716115107   14:27:51 up  5:32,  1 user,  load average: 14.10, 17.59, 16.88  
------------------------------------------------------------------------  */10 * * * * /opt/qradar/bin/time_sync.sh

Verify that that UDP port 123 can connect to the Console from the managed hosts.

/opt/qradar/support/all_servers.sh "nc -z -u -v <Console_IP_address> 123"

x.x.0.84-> 732APPhost.example.com  
Appliance Type: 4000    Product Version: 2020.3.0.20200716115107   16:38:34 up  3:16,  0 users,  load average:
 1.86, 1.36, 1.27  ------------------------------------------------------------------------  Ncat: Version 7.50
 ( https://nmap.org/ncat )  Ncat: Connected to x.x.0.80:123.  Ncat: UDP packet sent successfully  
Ncat: 1 bytes sent, 0 bytes received in 2.01 seconds.

If this test fails to verify that ports are not blocked by a firewall rule.

Use the iptables command to verify that UDP port 123 is open on the Console for the managed host.
```
iptables -L -n | grep 123
```
```
ACCEPT     udp  --  x.x.0.84         0.0.0.0/0            udp dpt:123
```

Use the ntpdate command to verify that the Console can connect to the NTP server.

ntpdate -q <NTP_server_IP_address>

server NTP_server_IP_address, stratum 1, offset 0.908444, delay 0.26669
  19 Aug 12:16:39 ntpdate[15214]: step time server NTP_server_IP_address offset

When you use encryption between the Console and managed hosts, UDP port 123 is listening only on the Console. TCP port 12500 is listening on both the Console and managed hosts. To verify that UDP port 123 and TCP port 12500 are listening type:

/opt/qradar/support/all_servers.sh -Ck "netstat -nap | egrep ':123|:12500'"

x.x.0.62 -> console.example.com
  Appliance Type: 500     Product Version: 2019.18.4.20200629201233   11:23:04 up 9 days, 19:06,  0 users,  
  load average: 2.16, 2.11, 2.09  ------------------------------------------------------------------------  tcp 
  0      0 127.0.0.1:12500         0.0.0.0:*               LISTEN      1593/socat  udp        0      0 0.0.0.0:123 
   0.0.0.0:*                           1639/chronyd  udp6       0      0 :::123                  :::*   
  1639/chronyd      x.x.0.67 -> qrm.example.com  Appliance Type: 700     
  Product Version: 2019.18.4.20200629201233   11:23:05 up 9 days, 18:42,  0 users,  load average: 0.42, 
  0.44, 0.48   ------------------------------------------------------------------------  tcp        0      
 0 127.0.0.1:12500         0.0.0.0:*               LISTEN      30403/sshd: root  tcp6       0      0 ::1:12500
               :::*                    LISTEN      30403/sshd: root

Results
If you use the time_sync.sh utility and the command fails or you continue to experience time synchronization error messages in your logs, contact IBM QRadar Support.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.4.3;7.5.0"}]

Tips