Node time not in sync

Follow this procedure if the system nodes time is drifting apart.

Time sync issues might break the upgrade process.

If the time is not in sync on the system nodes, the following alert might also be raised:

412: Node time not in sync

If you suspect time sync issues:

Run the following command on all three control nodes:

/opt/ibm/appliance/platform/apos-comms/tools/aposTimeCheck.py

Example healthy output:

[root@e1n1 ansible]# /opt/ibm/appliance/platform/apos-comms/tools/aposTimeCheck.py
No upstream time sources defined
Time currently syncing from e1n1
Time offset healthy. Value:0.000000013

Example unhealthy output:

[root@e1n2 ~]# /opt/ibm/appliance/platform/apos-comms/tools/aposTimeCheck.py
Checking corp-dc6.swg.usma.ibm.com
Configured timesource corp-dc6.swg.usma.ibm.com has not been accepted as a time source by chrony. It is NOT used for time source
Time currently syncing from e1n2.fbond
Time offset healthy. Value:0.000001082

If the time server is unreachable, or having issues, revert the time_servers entry in System_Name.yml to the template default, as in the following example:
```
vars:
    [...]
    time_servers: ["<OPTIONAL>"]
```

Rerun the time setup playbook:

cd /opt/ibm/appliance/platform/apos-comms/customer_network_config/ansible

ANSIBLE_HASH_BEHAVIOUR=merge ANSIBLE_LOG_PATH=/var/log/appliance/platform/apos-comms/house_setup.log ansible-playbook -i <System_name.yml> playbooks/set_upstream_time.yml

For more information on setting up time servers in the yml file, see Common network configuration values. Note that setting up external time source is strongly discouraged unless it is required in your system setup.