Configuring the ESXi operating system

Before you begin

Procedure

After the prerequisite tasks are complete, use the following general steps to configure your host.

1. Define the host object by using the worldwide port name identifiers (WWPNs) for each physical ESXi host in the VMware vSphere cluster.
2. If more than one physical ESXi host is present in the vSphere cluster, create a host cluster and add each ESXi host object as a member.

   When you map volumes to ESXi hosts, ensure that you create shared host-cluster mappings for all shared storage to ensure consistent SCSI IDs across the ESXi hosts for the same volume. If the ESXi host is SAN booted from a volume on the system, use private mappings for these volumes.

3. Review the multipath settings by running the following CLI command:

   # esxcli storage nmp device list -d naa.<UUID of volume>

   The output is similar to the following example:

   naa.6005076400840110700000000000008c
      Device Display Name: IBM Fibre Channel Disk (naa.6005076400840110700000000000008c)
      Storage Array Type: VMW_SATP_ALUA
      Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=on; {TPG_id=16,TPG_state=AO}{TPG_id=17,TPG_state=ANO}}
      Path Selection Policy: VMW_PSP_RR
      Path Selection Policy Device Config: {policy=rr,iops=1,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
      Path Selection Policy Device Custom Config:
      Working Paths: vmhba4:C0:T0:L250, vmhba3:C0:T0:L250

   Verify the following settings:

   1. The Storage Array type plug-in for IBM® volumes is set to VMW_SATP_ALUA.
   2. The path selection policy is set to RoundRobin.
   3. The Round Robin IOPS is set to 1 to evenly distribute I/Os across as many ports on the system as possible. For information about how to make this change, see the VMware information at this website: Adjusting Round Robin IOPS limit from default 1000 to 1 (2069356) .
4. Enable ATS hardware accelerated locking and set to ATS-only public.To verify the current settings, use the following CLI command:

   # vmkfstools -Ph -v1 /vmfs/volumes/<datastore_name> | grep Mode
   Mode: public ATS-only

   If the datastore Mode is not reporting public ATS-only, consider either of the following two options:

   1. Creating a LUN and creating a VMFS5 or VMFS6 datastore and use Storage vMotion to migrate virtual machines to the new datastore.
   2. Enforcing ATS-only on all affected datastores. This process requires stopping all I/O to the datastore first. For more information, see Change VMFS Locking to ATS-Only. This CLI command provides the UUID of the device:

      # vmkfstools -Ph -v1 /vmfs/volumes/<datastore_name>  | grep -A 1 Partitions
      Partitions spanned (on "lvm"):        naa.6005076400840110700000000000008b:1

      Using the UUID of the device, the following CLI commands can be used to change the Lock Mode to ATS-only:

      # vmkfstools --configATSOnly 1 /vmfs/devices/disks/naa.6005076400840110700000000000008b:1

5. Set the XCopy transfer size to 4096.

   # Get-VMHost  | Get-AdvancedSetting -Name DataMover.MaxHWTransferSize | select Entity, name, value
   
   
   Entity                        Name                        Value
   ------                        ----                        -----
   vmlab11c2.ssd.hursley.ibm.com DataMover.MaxHWTransferSize  4096

6. Use VAAI, which is a set of SCSI commands that are used by VMware ESXi hosts to offload operations to the storage array. Enable HardwareAcceleratedMove, HardwareAcceleratedLocking, and HardwareAcceleratedInit.

   # Get-VMHost  | Get-AdvancedSetting -name *HardwareAccelerated* | select Name, value
   Name                              Value
   ----                              -----
   DataMover.HardwareAcceleratedMove     1
   VMFS3.HardwareAcceleratedLocking      1
   DataMover.HardwareAcceleratedInit     1

7. Wherever possible, set the guest virtual machines to have SCSI timeouts set to at least 60 seconds although 120 seconds is recommended. Refer to the host vendor documentation for instructions on which settings to change.
8. The number of paths per volume is recommended not to exceed 8.

   However, during operations such as a nondisruptive volume move, if the Fibre Channel target port mode is transitional for the I/O group, up to 16 paths per volume is supported. See the VMware documentation for the maximum number of devices and paths for the version of VMware ESXi that you are using.

9. If you use Datastore Clusters where vSphere Storage vMotion is used to migrate virtual machines based on I/O latency, it is important to understand the system requirements.
   1. Ensure that each LUN has independent and isolated performance characteristics:
      1. Where possible, present each volume from a different pool, especially when you use arrays or managed disks that are created from traditional spinning (non-SSD or flash) media.
      2. Review the preferred node on each LUN in the Datastore Cluster to ensure an even distribution between the nodes within the I/O group.
   2. Given VMs can migrate between datastores within the Datastore Cluster. Be aware of dynamic I/O workloads and how Easy Tier will respond. Do one of the following tasks:
      1. Disable automatic storage migrations when you use Easy Tier, or
      2. Use a single tier of storage between all LUNs that are used in a Datastore Cluster.
10. If you have dual-site topologies, consider the following requirements:
    • If the system topology is set to stretched, consider using Host and Virtual Machine DRS Groups to restrict a given VM to move only to other hosts within the same site.
    • VMware ESXi host objects should have a site set to avoid unnecessary intersite traffic.

VMware ATS heartbeating

VMware ESXi uses the SCSI Compare and Write command (VMware refers to this command as Atomic Test and Set - ATS) to "heartbeat" periodically to datastores.

Due to the low timeout value for heartbeat I/O that uses ATS, host disconnects can occur if delays of 8 seconds or longer are experienced in completing individual heartbeat I/Os on backend storage systems or the SAN infrastructure. Under certain circumstances, such as if the ATS heartbeats are ended due to delays of 8 seconds or more, the ESXi host might retry the ATS heartbeat command and it might fail with a miscompare or lost access to datastore message that is logged in the vmkernel.log on the host. In many cases, ATS heartbeat miscomparisons during verification and lost access to datastore messages are results of ATS heartbeats that work as expected when I/O delays are present.

ATS heartbeating increases the load on the system and can lead to access issues on busy systems, particularly during maintenance procedures. To reduce this load, ATS heartbeats can be disabled.

# esxcli system settings advanced set -i 0 -o /VMFS3/useATSForHBOnVMFS3

SAN boot support for hosts running ESXi operating systems

The system can be used as a boot device for the ESXi operating system.

See the software restrictions page on the following IBM support website for any other restrictions for SAN boot support: www.ibm.com/support.