VMware ESXi lists hard drives in mixed order with LSI-SAS controller - IBM BladeCenter and System x

Resolving The Problem

Source

RETAIN tip: H196649

Symptom

This tip only impacts Unified Extensible Firmware Interface (UEFI) based systems that also use the on-board LSI SAS controller. Depending on the system, this could be LSI1064e, ServeRAID-BR10i, or ServeRAID-BR10il.

The symptom is most often seen with VMware's ESXi, "Classic" ESXi versions and Legacy Operating Systems (OSes) as well. Windows Server 2003 and 2008 are Legacy OSes.

This symptom only occurs when more than one disk or array is in the server or blade.

Note: This symptom has only been observed consistently in ESXi.

Note: This issue affects all versions of ESXi "Installable" that are supported on IBM System x Servers and Blades.

The symptom will not be seen in the following scenarios:

1. When all drives are included in a single array.
   
   
2. When using Windows Server 2008 R2 because it is UEFI aware.
   
   
3. With ESXi Embedded (that resides on a Universal Serial Bus (USB) key).
   

While installing VMware ESXi, the user is asked to choose the boot disk. If the prerequisite UEFI settings have not been enabled, then the drives might appear in a mixed or reverse order during ESXi installation.

For example:

• The drive, which actually sits in physical Bay0, would be shown by ESXi as disk 2, while the drive in physical Bay1 would be shown by ESXi as disk 1.
• In systems with multiple drives, the disk in Bay0 might actually be shown by ESXi as the first disk, but the other drives might appear in a mixed order.

Because drives are sometimes presented in a mixed order, this will impact scripted installs, which depend upon the boot drive being called "HardDisk0" during installation of the OS.

Affected configurations

The system may be any of the following IBM servers:

• BladeCenter HS22, type 1936, any model
• BladeCenter HS22, type 7870, any model
• BladeCenter HS22V, type 1949, any model
• BladeCenter HS22V, type 7871, any model
• BladeCenter HX5, type 7872, any model
• System x3250 M2, type 4190, any model
• System x3250 M2, type 4191, any model
• System x3250 M2, type 4194, any model
• System x3250 M3, type 4251, any model
• System x3250 M3, type 4252, any model
• System x3250 M3, type 4261, any model
• System x3400 M2, type 7836, any model
• System x3400 M2, type 7837, any model
• System x3400 M3, type 7378, any model
• System x3400 M3, type 7379, any model
• System x3500 M2, type 7839, any model
• System x3500 M3, type 7380, any model
• System x3550 M2, type 4198, any model
• System x3550 M2, type 7946, any model
• System x3550 M3, type 4254, any model
• System x3550 M3, type 7944, any model
• System x3650 M2, type 4199, any model
• System x3650 M2, type 7947, any model
• System x3650 M3, type 4255, any model
• System x3650 M3, type 7945, any model
• System x3690 X5, type 7148, any model
• System x3690 X5, type 7149, any model
• System x3850 X5, type 7145, any model
• System x3850 X5, type 7146, any model
• iDataPlex dx360 M2 Server, type 6380, any model
• iDataPlex dx360 M2 Server, type 7321, any model
• iDataPlex dx360 M2 Server, type 7323, any model

The system is configured with at least one of the following:

• Microsoft Windows 2003 Server, any service pack
• Microsoft Windows 2003 Server for 32-bit Servers, any service pack
• Microsoft Windows 2003 Server for 64-bit Servers, any service pack
• Microsoft Windows 2003 Server, Datacenter Edition, any service pack
• Microsoft Windows 2003 Server, EE x64, any service pack
• Microsoft Windows 2003 Server, x64 Edition, any service pack
• Microsoft Windows Server 2003/2003 R2, Datacenter x64 Edition, any service pack
• Microsoft Windows Server 2008, Datacenter 32-bit, any service pack
• Microsoft Windows Server 2008, Datacenter 64-bit, any service pack
• Microsoft Windows Server 2008, Enterprise 32-bit, any service pack
• Microsoft Windows Server 2008, Enterprise 64-bit, any service pack
• Microsoft Windows Server 2008, Standard 32-bit, any service pack
• Microsoft Windows Server 2008, Standard 64-bit, any service pack
• VMware ESXi 3.5 Embedded, Update 2
• VMware ESXi 3.5 Installable, Update 1
• VMware ESXi 4.0 Installable, any Update

The system is configured with one or more of the following IBM Options:

• Unified Extensible Firmware Interface (UEFI)

Note: This does not imply that the network operating system will work under all combinations of hardware and software.

Please see the compatibility page for more information: http://www.ibm.com/servers/eserver/serverproven/compat/us/

Solution

A legacy OS such as ESXi will depend upon the UEFI server to indicate which disk is supposed to be the boot drive. In turn, the server depends upon the user to make this designation.

The following two steps are expected from the user:

1. In UEFI systems, set "Legacy Only" as the first boot item.
   
   
2. Use the LSI setup utility to indicate which disk is to be the boot drive.
   To be sure that step 2 works properly, it would be best to update the server's LSI firmware, which is downloadable from the following URL:
   
   

   http://www.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-65666

Follow the instructions in the text file, which can be downloaded along with the firmware update file.

1. To enter the UEFI\Basic Input/Output System (BIOS) menu, power on the server and press F1 during the IBM splash screen.
   
   
2. Using the arrow keys to navigate, scroll down to highlight "Boot Manager" and press Enter.
   
   
3. “Add Boot Option” is highlighted by default. Press Enter.
   
   
4. Using the down arrow key, scroll down until "Legacy Only" is highlighted. Press Enter.
   
   
5. Select “Change Boot Order” and press Enter.
   
   
6. Press Enter to begin changing the boot order.
   
   
7. Highlight Legacy Only and press Shift + to move it to the top. Press Enter.
   
   
8. Press the down arrow once to highlight "Commit Changes" and press Enter.
   
   1. Indicate which disk is to be the boot drive
      Note: If Redundant Array of Independent Disks (RAID) arrays are to be used, these steps assume that the arrays have already been created.
      
   
   
9. To enter the LSI Serial Attached SCSI (SAS) BIOS menu, reboot the server. During Legacy Power On Self Test (POST), some of the system devices will be enumerated on screen. When LSI is shown, press Ctrl+C.
   
   
10. The LSI on-board controller is highlighted by default. Press Enter.
    
    
11. Highlight "SAS Topology" and press Enter.
    
    
12. Highlight "Direct Attach Devices" and press Enter.
    
    
13. Arrow down to the drive (or array) that should be made bootable.
    
    
14. Press "Alt+B" to mark the drive (or array) bootable.
    
    

    Note: If "Alt+B" is not shown at the bottom of the screen, then the system has downlevel firmware. If the firmware cannot be updated at this time, follow the workaround steps in this RETAIN tip.

    
    
15. Press Esc then arrow down to "Save changes then exit this menu." Press Enter.
    

Now, when ESXi is installed, the correct drive (or array) will be presented by the installation process as Hard Disk 1 (the first one) and the rest of the drives will be enumerated in the correct sequence after that.

Workaround

Workaround 1:
Reconfigure the drives so that all disks are in a single array.

Workaround 2:
Add ALL hard disks into the Boot Order in UEFI.

When following the steps above for adding Legacy to the boot order, add HardDisk1 as well. The boot order will then be as follows:

Additional information

ESX and ESXi are Legacy OSes and are not UEFI aware. By default, IBM servers are set up to first check for UEFI-based boot-code. If none is found, the server then reverts to Legacy POST. A standard practice for servers running a Legacy OS should be to change the boot order to Legacy first.