VMware ATS heartbeating

VMware ESXi uses the SCSI Compare and Write command (VMware refers to this command as Atomic Test and Set - ATS) to "heartbeat" periodically to datastores.

Note: The use of ATS heartbeating is not supported on ESXi hosts that run version 5.5 update 2 or later or ESXi version 6.0 before update 3.

Due to the low timeout value for heartbeat I/O that uses ATS, host disconnects can occur if delays of 8 seconds or longer are experienced in completing individual heartbeat I/Os on backend storage systems or the SAN infrastructure. Under certain circumstances, such as if the ATS heartbeats are ended due to delays of 8 seconds or more, the ESXi host might retry the ATS heartbeat command and it might fail with a miscompare or lost access to datastore message that is logged in the vmkernel.log on the host. In many cases, ATS heartbeat miscomparisons during verification and lost access to datastore messages are results of ATS heartbeats that work as expected when I/O delays are present.

ATS heartbeating increases the load on the system and can lead to access issues on busy systems, particularly during maintenance procedures. To reduce this load, ATS heartbeats can be disabled.

For VMFS5 and VMFS6 datastores, run these commands:
  • To disable ATS heartbeats, run the following CLI command:
    # esxcli system settings advanced set -i 0 -o /VMFS3/UseATSForHBOnVMFS5
  • To enable ATS heartbeats, run the following CLI command:
    # esxcli system settings advanced set -i 1 -o /VMFS3/UseATSForHBOnVMFS5
For VMFS3 datastores, run the following command to disable ATS heartbeats:
# esxcli system settings advanced set -i 0 -o /VMFS3/useATSForHBOnVMFS3