Topic
  • 9 replies
  • Latest Post - ‏2012-12-13T23:29:32Z by SystemAdmin
SystemAdmin
SystemAdmin
9111 Posts

Pinned topic ESXi 5.1 / IBM x3550M3 strange issues

‏2012-12-05T22:17:30Z |
Good day,

I installed ESXi 5.1 on two IBM x3550M3. The installation went fine and the hypervisor detects all resources e.g. local SATA datastore etc. When creating VM (Win2008R2 x64), the VM won't boot properly. There seems to be some low level communication issues. It either takes like 30 mins to boot then I can't connect to the console to continue installation etc. To power off takes same amount of time. I have tried using IBM's ESXi 5.1 ISO but no help. I tried using both direct host cd conneciton and client IDE emulation to boot of installation cd in VM. No luck. It has connected after hours of leaving it and I reached Windows install but it sticks during install, it's no way acceptable.

I tried on a different server type (Asus) mother board with the same ESXi 5.1 and it works like it should, connect to both host cd or emulation to boot installation cd and VM creation easily. It seems like an issue with ESXi 5.1 on the IBM x3550M3 because ESXi 4.1 works like it should on this server.
Current server info
UEFI Firmware
D6E154AUS , v 1.13
Machine Model:7944AC1
SAS Controller
LSI SAS2 MPT Controller SAS 2004
Intel Xeon E5620 2.40 GHz x 2, 4 core

On vmware HCL for ESXi 5.1 and x3550M3 states Firmware D6E150CUS-1.11 but the current D6E154AUS-1.13 seems newer than this?

I am a little confused. Am I missing something?
Updated on 2012-12-13T23:29:32Z at 2012-12-13T23:29:32Z by SystemAdmin
  • skyron
    skyron
    99 Posts

    Re: ESXi 5.1 / IBM x3550M3 strange issues

    ‏2012-12-06T05:44:12Z  
    Hi,

    Could you try updating all the server firmware first, using IBM BoMC tool?

    BoMC Tool: www-947.ibm.com/support/entry/portal/docdisplay?lndocid=tool-bomc
  • SystemAdmin
    SystemAdmin
    9111 Posts

    Re: ESXi 5.1 / IBM x3550M3 strange issues

    ‏2012-12-10T14:48:32Z  
    • skyron
    • ‏2012-12-06T05:44:12Z
    Hi,

    Could you try updating all the server firmware first, using IBM BoMC tool?

    BoMC Tool: www-947.ibm.com/support/entry/portal/docdisplay?lndocid=tool-bomc
    Hi,
    I updated all firmware with BoMC. Same problem.
    I believe it's the internal local disk attached/LSI SAS MPT 2004 controller, it is now confirmed. All other resources on the server work well. Browsing and uploading to the data store has extremely major lags and I/O errors from the VI Client level also High Disk latency reaching 43560 milliseconds and this is from uploading pdf files.

    See vmkwarning repeated logs below after upload attempt.

    cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...
    2012-12-08T20:13:30.630Z cpu4:4510)WARNING: LinScsi: SCSILinuxQueueCommand:1193:queuecommand failed with status = 0x1055 Host Busy vmhba2:0:0:0 (driver name: Fusion MPT SAS Host) - Message repeated 1 time
    2012-12-08T20:13:41.910Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...
    2012-12-08T20:13:53.204Z cpu0:4112)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...
    2012-12-08T20:14:04.500Z cpu6:4102)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...

    cpu8:4271)WARNING: LinScsi: SCSILinuxAbortCommands:1816:Failed, Driver Fusion MPT SAS Host, for vmhba2

    Note it's a SATA drive on the SAS connector.

    On vmware HCL for (LSI2004: LSI Logic Fusion-MPT 6GSAS SAS2004 PCI-Express) and ESXi 5.1 it shows "mpt2sas version 06.00.00.00-5vmw" Inbox. When I checked the version in use on the server it's version 10.00.00.00.5vmw (looks newer). VID / DID numbers match the HCL.

    Note one page on the HCL shows no ESX 5.1 support while another shows the driver support mentioned. I am wondering if this is indeed supported with ESXi 5.1 as 4.1 works just fine.

    Any advice?
  • SystemAdmin
    SystemAdmin
    9111 Posts

    Re: ESXi 5.1 / IBM x3550M3 strange issues

    ‏2012-12-12T17:38:39Z  
    Hi,
    I updated all firmware with BoMC. Same problem.
    I believe it's the internal local disk attached/LSI SAS MPT 2004 controller, it is now confirmed. All other resources on the server work well. Browsing and uploading to the data store has extremely major lags and I/O errors from the VI Client level also High Disk latency reaching 43560 milliseconds and this is from uploading pdf files.

    See vmkwarning repeated logs below after upload attempt.

    cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...
    2012-12-08T20:13:30.630Z cpu4:4510)WARNING: LinScsi: SCSILinuxQueueCommand:1193:queuecommand failed with status = 0x1055 Host Busy vmhba2:0:0:0 (driver name: Fusion MPT SAS Host) - Message repeated 1 time
    2012-12-08T20:13:41.910Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...
    2012-12-08T20:13:53.204Z cpu0:4112)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...
    2012-12-08T20:14:04.500Z cpu6:4102)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...

    cpu8:4271)WARNING: LinScsi: SCSILinuxAbortCommands:1816:Failed, Driver Fusion MPT SAS Host, for vmhba2

    Note it's a SATA drive on the SAS connector.

    On vmware HCL for (LSI2004: LSI Logic Fusion-MPT 6GSAS SAS2004 PCI-Express) and ESXi 5.1 it shows "mpt2sas version 06.00.00.00-5vmw" Inbox. When I checked the version in use on the server it's version 10.00.00.00.5vmw (looks newer). VID / DID numbers match the HCL.

    Note one page on the HCL shows no ESX 5.1 support while another shows the driver support mentioned. I am wondering if this is indeed supported with ESXi 5.1 as 4.1 works just fine.

    Any advice?
    Hi,
    The controller is a ServeRAID H1110. Looking at the IBM site http://www.redbooks.ibm.com/abstracts/tips0831.html I don't see ESXi 5.0 or 5.1 It has up to 4.1 and 4.1 works fine on the server and detects the controller just as the ESXi 5.x except a different driver version relating to the version OS.
    Does this controller indeed not support ESXi 5.x?
  • DaveQ
    DaveQ
    132 Posts

    Re: ESXi 5.1 / IBM x3550M3 strange issues

    ‏2012-12-12T21:09:55Z  
    Hi,
    The controller is a ServeRAID H1110. Looking at the IBM site http://www.redbooks.ibm.com/abstracts/tips0831.html I don't see ESXi 5.0 or 5.1 It has up to 4.1 and 4.1 works fine on the server and detects the controller just as the ESXi 5.x except a different driver version relating to the version OS.
    Does this controller indeed not support ESXi 5.x?
    According to ServerProven, the x3550 M3 w/ the H1110 controller and vSphere 5 is supported.
    http://www-03.ibm.com/systems/info/x86servers/serverproven/compat/us/xseries/7944/ibm_81Y4492.html
    Do you know what firmware the H1110 is operating? Reason I ask is firmware 1.10.04 from March 23rd doesn't list VMware 5, but the most current firmware 1.10.06 from August 14th does. It's a little vague, stating "VMware 5 x64", assuming that means vSphere 5. Since the earlier one doesn't even mention version 5, chances are controller support wasn't intended for 5 and may have been added later.
  • SystemAdmin
    SystemAdmin
    9111 Posts

    Re: ESXi 5.1 / IBM x3550M3 strange issues

    ‏2012-12-12T21:48:15Z  
    • DaveQ
    • ‏2012-12-12T21:09:55Z
    According to ServerProven, the x3550 M3 w/ the H1110 controller and vSphere 5 is supported.
    http://www-03.ibm.com/systems/info/x86servers/serverproven/compat/us/xseries/7944/ibm_81Y4492.html
    Do you know what firmware the H1110 is operating? Reason I ask is firmware 1.10.04 from March 23rd doesn't list VMware 5, but the most current firmware 1.10.06 from August 14th does. It's a little vague, stating "VMware 5 x64", assuming that means vSphere 5. Since the earlier one doesn't even mention version 5, chances are controller support wasn't intended for 5 and may have been added later.
    I have updated the UEFI using BoMC. Updated UEFI and NIC's and DSA. See below for firmware info
    UEFI Firmware
    Version 1.15
    Build D6E157AUS

    Controller Info
    LSI SAS2 MPT Controller SAS 2004
    v 07.18.01.05
    Firmware Type IR
    Firmware version 10.0.3.0

    I have slipped many mpt2sas drives from vmware to the ESXi 5.0 ISO and confirmed loaded by OS but same issues. Haven't scene any mpt2sas ESXi 5.1 drivers though. Although ESXi 5.1 does install and load a driver. There is connection to the disk but clearly shoes in vmkernel logs, write issues and errors. ESXi 4.1 however loads and works like it should. I want to believe it's not supported fully by ESXi 5.x but am not 100% sure and I would think it should be?
  • DaveQ
    DaveQ
    132 Posts

    Re: ESXi 5.1 / IBM x3550M3 strange issues

    ‏2012-12-13T00:45:10Z  
    I have updated the UEFI using BoMC. Updated UEFI and NIC's and DSA. See below for firmware info
    UEFI Firmware
    Version 1.15
    Build D6E157AUS

    Controller Info
    LSI SAS2 MPT Controller SAS 2004
    v 07.18.01.05
    Firmware Type IR
    Firmware version 10.0.3.0

    I have slipped many mpt2sas drives from vmware to the ESXi 5.0 ISO and confirmed loaded by OS but same issues. Haven't scene any mpt2sas ESXi 5.1 drivers though. Although ESXi 5.1 does install and load a driver. There is connection to the disk but clearly shoes in vmkernel logs, write issues and errors. ESXi 4.1 however loads and works like it should. I want to believe it's not supported fully by ESXi 5.x but am not 100% sure and I would think it should be?
    Well, according to the ServerProven link I sent, it clearly says 5.1 is covered by standard support for the combination of server and storage controller you have (if it's truly an H1110). To me that says yes, it is supported. Just out of curiousity, what kind of SCSI driver are you using in the VM's?

    Not saying any of the below is your problem, but about the vSphere install itself, I would keep using the IBM customized vSphere 5.1 installer; otherwise you'll wind up missing some fairly important drivers which are hardware specific (such as for the IMM). Sorry for this long post, just copying contents of the readme for reference below. Can you compare your lsiprovider version too? Note the different sections, 5.1 customization, then update 4 which applies to 5.0 and 5.1 base installs, lsiprovider is updated in update 4.


    This is the content that is included in VMware vSphere Hypervisor 5.1 with IBM Customization:

    Vendor Provider/Driver Version



    Broadcom brcm 500.1.1-000000
    Broadcom misc-cnic-register 1.72.1.v50.1-1OEM.500.0.0.472560
    Broadcom net-bnx2 2.2.1l.v50.1-1OEM.500.0.0.472560
    Broadcom net-bnx2x 1.72.54.v50.2-1OEM.500.0.0.472560
    Broadcom net-cnic 1.72.50.v50.1-1OEM.500.0.0.472560
    Broadcom net-tg3 3.123b.v50.1-1OEM.500.0.0.472560
    Broadcom scsi-bnx2fc 1.72.51.v50.1-1OEM.500.0.0.406165
    Broadcom scsi-bnx2i 2.72.10.v50.2-1OEM.500.0.0.472560

    Brocade brcdprovider 3.0.3.0-0
    Brocade net-bna 3.0.3.0-1OEM.500.0.0.472560
    Brocade scsi-bfa 3.0.3.0-1OEM.500.0.0.472560

    Emulex ima-be2iscsi 4.1.334.3-1OEM.500.0.0.472629
    Emulex net-be2net 4.1.334.48-1OEM.500.0.0.472560
    Emulex scsi-be2iscsi 4.1.334.3-1OEM.500.0.0.472629
    Emulex scsi-lpfc820 8.2.2.126.50-1OEM.500.0.0.472560
    Emulex emulex-cim-provider 3.5.19.1-01

    LSI lsiprovider 500.04.V0.30-4000000

    Qlogic fchba 1.3.14-469512
    Qlogic net-qlge 1.0.0.47-1vmw.500.0.0.406165
    Qlogic ima-qla4xxx 500.2.01.31-1vmw.0.0.011509
    Qlogic net-qlcnic 5.0.741-1OEM.500.0.0.472560
    Qlogic scsi-qla2xxx 911.k1.1-26OEM.500.0.0.472560
    Qlogic scsi-qla4xxx 624.01.43-1OEM.500.0.0.472560

    IBM immpassthru 500-2ACE15AUS
    IBM concretejob 500-2ACE14DUS
    IBM filetrans 500-2ACE14DUS
    IBM fwupdate 500-2ACE14DUS
    IBM fupb 5-7.250
    IBM hwckvm 500-ESXI01ACN
    IBM pciinfo 500-PRIVATE
    IBM soibms 500-ESXI01ACN
    IBM brcmfwup 500-1.00.20120507
    IBM ilfu 500-1.01.20120504
    IBM ibmcustomization 3.0-0
    IBM ibmlic-esx-license 1.0-03
    VMware vSphere ESXi 5.x with IBM Customization Patch 4 change file (also applies to 5.1, indicated in file)

    fupb 5.01-7.280
    concretejob 500-2ACE16AUS
    filetrans 500-2ACE16AUS
    fwupdate 500-2ACE16AUS
    immpassthru 500-2ACE16AUS
    pciinfo 500-I03ACN
    Brcmfwup (facade) 500-1.00.20121017
    Brcm 500.2.0.3-000000
    ilfu (facade) 500-1.01.20121012
    lsiprovider 500.04.v0.32-0004
    fchba 1.3.14-469512
    qlcnic 5.0.741-1OEM.500.0.0.472560
    scsi-qla2xxx 911.k1.1-26OEM.500.0.0.472560
    net-be2net 4.2.327.0-1OEM.500.0.0.472560
    be2iscsi 4.2.327.0-1OEM.500.0.0.472560
    ima-be2isci 4.2.327.0-1OEM.500.0.0.472560

    Note the following for the update 4 patch:
    "This patch can be applied to the following IBM customized images only:
    * VMware ESXi 5.0 build 474610
    * VMware ESXi 5.0u1 build 623860
    * VMware ESXi 5.1 build 799733 <-- this is the build of the base IBM customized 5.1 installer
    "
  • DaveQ
    DaveQ
    132 Posts

    Re: ESXi 5.1 / IBM x3550M3 strange issues

    ‏2012-12-13T01:09:20Z  
    I have updated the UEFI using BoMC. Updated UEFI and NIC's and DSA. See below for firmware info
    UEFI Firmware
    Version 1.15
    Build D6E157AUS

    Controller Info
    LSI SAS2 MPT Controller SAS 2004
    v 07.18.01.05
    Firmware Type IR
    Firmware version 10.0.3.0

    I have slipped many mpt2sas drives from vmware to the ESXi 5.0 ISO and confirmed loaded by OS but same issues. Haven't scene any mpt2sas ESXi 5.1 drivers though. Although ESXi 5.1 does install and load a driver. There is connection to the disk but clearly shoes in vmkernel logs, write issues and errors. ESXi 4.1 however loads and works like it should. I want to believe it's not supported fully by ESXi 5.x but am not 100% sure and I would think it should be?
    The firmware on the controller is old compared to what is available.

    What you indicated you have:
    LSI SAS2 MPT Controller SAS 2004
    v 07.18.01.05 <-- this is the UEFI driver version mentioned below.
    Firmware Type IR
    Firmware version 10.0.3.0
    What is available:
    BIOS 7.19.00.00
    Firmware 10.00.08.00
    UEFI Driver 7.18.04.03

    You can find the updates in firmware package 1.10.06 for the H1110 (listed under the ServeRAID section for the 3550 M3 (7944) on Fixcentral.

    I still have some questions about the VM's, but I would update the controller first.

    My main VM questions are (and my own opinion):
    1. What SCSI driver are you using in the VM? (LSI Parallel, LSI SAS, VMware Paravirtual) - I use PV for all of my 2008 R2 VMs, stellar performance.

    2. What BIOS mode are you using in the VM? (BIOS or EFI) - I wouldn't build VMs using EFI, too much overhead by allowing it more exposure to the host than it truly needs. A VM should be less not more right?
  • DaveQ
    DaveQ
    132 Posts

    Re: ESXi 5.1 / IBM x3550M3 strange issues

    ‏2012-12-13T01:14:37Z  
    • DaveQ
    • ‏2012-12-13T01:09:20Z
    The firmware on the controller is old compared to what is available.

    What you indicated you have:
    LSI SAS2 MPT Controller SAS 2004
    v 07.18.01.05 <-- this is the UEFI driver version mentioned below.
    Firmware Type IR
    Firmware version 10.0.3.0
    What is available:
    BIOS 7.19.00.00
    Firmware 10.00.08.00
    UEFI Driver 7.18.04.03

    You can find the updates in firmware package 1.10.06 for the H1110 (listed under the ServeRAID section for the 3550 M3 (7944) on Fixcentral.

    I still have some questions about the VM's, but I would update the controller first.

    My main VM questions are (and my own opinion):
    1. What SCSI driver are you using in the VM? (LSI Parallel, LSI SAS, VMware Paravirtual) - I use PV for all of my 2008 R2 VMs, stellar performance.

    2. What BIOS mode are you using in the VM? (BIOS or EFI) - I wouldn't build VMs using EFI, too much overhead by allowing it more exposure to the host than it truly needs. A VM should be less not more right?
    Last follow-up after my carpet bomb responses... the version of firmware for the controller you have is the initial release (now 4 releases old). Definitely would update it. Basically everything in this change list applies since you're still running the initial release.

    http://delivery04.dhe.ibm.com/sar/CMA/XSA/03ase/0/ibm_fw_mpt2sas_h1110-1.10.06_linux_32-64.chg
  • SystemAdmin
    SystemAdmin
    9111 Posts

    Re: ESXi 5.1 / IBM x3550M3 strange issues

    ‏2012-12-13T23:29:32Z  
    • DaveQ
    • ‏2012-12-13T01:14:37Z
    Last follow-up after my carpet bomb responses... the version of firmware for the controller you have is the initial release (now 4 releases old). Definitely would update it. Basically everything in this change list applies since you're still running the initial release.

    http://delivery04.dhe.ibm.com/sar/CMA/XSA/03ase/0/ibm_fw_mpt2sas_h1110-1.10.06_linux_32-64.chg
    Hey Dave,
    Your correct and thanks much for pointing this out. When I used the BoMC, I used UXSP's and the pack was not up to date. Looking in the portal there was indeed many newer updates, including for the controller so I used latest available with BoMC this time and updated to latest. Also used the update patch for customized image and confirmed updates after. I used the customized image without the update during troubleshooting so I would think it's the controller fw was the fix. Now I know not use the UXSP's. I know the SCSI version don't matter now but I usually convert to PV for boot after install from LSI Logic SAS and continue with PV for data drives. Again thanks for the carpet booms ;)