Resolving The Problem

What';s Happening

vCenter triggers a virtual machine total disk latency alert.

Troubleshooting

Poor storage performance is generally the result of high I/O latency. To view I/O latency, use vCenter or esxtop to understand the following latencies:

• Guest Average Latency (GAVG) - total latency as seen from vSphere, the sum of KAVG + QABG + DAVG.
• Kernel Average Latency (KAVG) - time an I/O request spent waiting inside the vSphere storage stack. KAVG includes QAVG.
• Queue Average latency (QAVG) - time spent waiting in a queue inside the vSphere Storage Stack.
• Device Average Latency (DAVG) - latency coming from the physical hardware, HBA and Storage device.

vCenter or esxtop cannot provide information for the latency seen by the application since that includes the latency in the VM OS and the application, and these are not visible to vCenter or esxtop. Refer to the VMware vSphere blog Troubleshooting storage performance in vSphere for additional information.

Some applications are much more sensitive to I/O latency than others. For guidance only, GAVG should be less than 25 ms and KAVG should be less than 2ms. Anything greater should be investigated as a potential storage performance issue. Poor storage performance can include the following:

• NFS storage unable to provide the required performance - often storage is based on capacity required rather than IOPs/latency/throughput needed.
• I/O stack queue congestion - as the I/O stack requires memory for its transfer buffers and CPU to process the IO requests, ensure the host has enough memory and CPU resources available.
• Network bandwidth saturation - NFS and vSAN requires network bandwidth, ensure that the network is not congested.

If the virtualized infrastructure layer is operating as expected, but the application is performing poorly, then review the following inside the VM:

• Guest level driver and queuing interaction.
• Incorrectly tuned application.

If required you can benchmark the storage, and there are several free and open source tools like IOmeter that can be used. There is also the VMware I/O Analyzer tool.