Topic
9 replies Latest Post - ‏2012-12-13T23:29:32Z by SystemAdmin
SystemAdmin
SystemAdmin
9111 Posts
ACCEPTED ANSWER

Pinned topic ESXi 5.1 / IBM x3550M3 strange issues

‏2012-12-05T22:17:30Z |
Good day,

I installed ESXi 5.1 on two IBM x3550M3. The installation went fine and the hypervisor detects all resources e.g. local SATA datastore etc. When creating VM (Win2008R2 x64), the VM won't boot properly. There seems to be some low level communication issues. It either takes like 30 mins to boot then I can't connect to the console to continue installation etc. To power off takes same amount of time. I have tried using IBM's ESXi 5.1 ISO but no help. I tried using both direct host cd conneciton and client IDE emulation to boot of installation cd in VM. No luck. It has connected after hours of leaving it and I reached Windows install but it sticks during install, it's no way acceptable.

I tried on a different server type (Asus) mother board with the same ESXi 5.1 and it works like it should, connect to both host cd or emulation to boot installation cd and VM creation easily. It seems like an issue with ESXi 5.1 on the IBM x3550M3 because ESXi 4.1 works like it should on this server.
Current server info
UEFI Firmware
D6E154AUS , v 1.13
Machine Model:7944AC1
SAS Controller
LSI SAS2 MPT Controller SAS 2004
Intel Xeon E5620 2.40 GHz x 2, 4 core

On vmware HCL for ESXi 5.1 and x3550M3 states Firmware D6E150CUS-1.11 but the current D6E154AUS-1.13 seems newer than this?

I am a little confused. Am I missing something?
Updated on 2012-12-13T23:29:32Z at 2012-12-13T23:29:32Z by SystemAdmin
  • skyron
    skyron
    99 Posts
    ACCEPTED ANSWER

    Re: ESXi 5.1 / IBM x3550M3 strange issues

    ‏2012-12-06T05:44:12Z  in response to SystemAdmin
    Hi,

    Could you try updating all the server firmware first, using IBM BoMC tool?

    BoMC Tool: www-947.ibm.com/support/entry/portal/docdisplay?lndocid=tool-bomc
    • SystemAdmin
      SystemAdmin
      9111 Posts
      ACCEPTED ANSWER

      Re: ESXi 5.1 / IBM x3550M3 strange issues

      ‏2012-12-10T14:48:32Z  in response to skyron
      Hi,
      I updated all firmware with BoMC. Same problem.
      I believe it's the internal local disk attached/LSI SAS MPT 2004 controller, it is now confirmed. All other resources on the server work well. Browsing and uploading to the data store has extremely major lags and I/O errors from the VI Client level also High Disk latency reaching 43560 milliseconds and this is from uploading pdf files.

      See vmkwarning repeated logs below after upload attempt.

      cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...
      2012-12-08T20:13:30.630Z cpu4:4510)WARNING: LinScsi: SCSILinuxQueueCommand:1193:queuecommand failed with status = 0x1055 Host Busy vmhba2:0:0:0 (driver name: Fusion MPT SAS Host) - Message repeated 1 time
      2012-12-08T20:13:41.910Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...
      2012-12-08T20:13:53.204Z cpu0:4112)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...
      2012-12-08T20:14:04.500Z cpu6:4102)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.5000c5003fc3bd4b" state in doubt; requested fast path state update...

      cpu8:4271)WARNING: LinScsi: SCSILinuxAbortCommands:1816:Failed, Driver Fusion MPT SAS Host, for vmhba2

      Note it's a SATA drive on the SAS connector.

      On vmware HCL for (LSI2004: LSI Logic Fusion-MPT 6GSAS SAS2004 PCI-Express) and ESXi 5.1 it shows "mpt2sas version 06.00.00.00-5vmw" Inbox. When I checked the version in use on the server it's version 10.00.00.00.5vmw (looks newer). VID / DID numbers match the HCL.

      Note one page on the HCL shows no ESX 5.1 support while another shows the driver support mentioned. I am wondering if this is indeed supported with ESXi 5.1 as 4.1 works just fine.

      Any advice?
      • SystemAdmin
        SystemAdmin
        9111 Posts
        ACCEPTED ANSWER

        Re: ESXi 5.1 / IBM x3550M3 strange issues

        ‏2012-12-12T17:38:39Z  in response to SystemAdmin
        Hi,
        The controller is a ServeRAID H1110. Looking at the IBM site http://www.redbooks.ibm.com/abstracts/tips0831.html I don't see ESXi 5.0 or 5.1 It has up to 4.1 and 4.1 works fine on the server and detects the controller just as the ESXi 5.x except a different driver version relating to the version OS.
        Does this controller indeed not support ESXi 5.x?
        • DaveQ
          DaveQ
          96 Posts
          ACCEPTED ANSWER

          Re: ESXi 5.1 / IBM x3550M3 strange issues

          ‏2012-12-12T21:09:55Z  in response to SystemAdmin
          According to ServerProven, the x3550 M3 w/ the H1110 controller and vSphere 5 is supported.
          http://www-03.ibm.com/systems/info/x86servers/serverproven/compat/us/xseries/7944/ibm_81Y4492.html
          Do you know what firmware the H1110 is operating? Reason I ask is firmware 1.10.04 from March 23rd doesn't list VMware 5, but the most current firmware 1.10.06 from August 14th does. It's a little vague, stating "VMware 5 x64", assuming that means vSphere 5. Since the earlier one doesn't even mention version 5, chances are controller support wasn't intended for 5 and may have been added later.
          • SystemAdmin
            SystemAdmin
            9111 Posts
            ACCEPTED ANSWER

            Re: ESXi 5.1 / IBM x3550M3 strange issues

            ‏2012-12-12T21:48:15Z  in response to DaveQ
            I have updated the UEFI using BoMC. Updated UEFI and NIC's and DSA. See below for firmware info
            UEFI Firmware
            Version 1.15
            Build D6E157AUS

            Controller Info
            LSI SAS2 MPT Controller SAS 2004
            v 07.18.01.05
            Firmware Type IR
            Firmware version 10.0.3.0

            I have slipped many mpt2sas drives from vmware to the ESXi 5.0 ISO and confirmed loaded by OS but same issues. Haven't scene any mpt2sas ESXi 5.1 drivers though. Although ESXi 5.1 does install and load a driver. There is connection to the disk but clearly shoes in vmkernel logs, write issues and errors. ESXi 4.1 however loads and works like it should. I want to believe it's not supported fully by ESXi 5.x but am not 100% sure and I would think it should be?
            • DaveQ
              DaveQ
              96 Posts
              ACCEPTED ANSWER

              Re: ESXi 5.1 / IBM x3550M3 strange issues

              ‏2012-12-13T00:45:10Z  in response to SystemAdmin
              Well, according to the ServerProven link I sent, it clearly says 5.1 is covered by standard support for the combination of server and storage controller you have (if it's truly an H1110). To me that says yes, it is supported. Just out of curiousity, what kind of SCSI driver are you using in the VM's?

              Not saying any of the below is your problem, but about the vSphere install itself, I would keep using the IBM customized vSphere 5.1 installer; otherwise you'll wind up missing some fairly important drivers which are hardware specific (such as for the IMM). Sorry for this long post, just copying contents of the readme for reference below. Can you compare your lsiprovider version too? Note the different sections, 5.1 customization, then update 4 which applies to 5.0 and 5.1 base installs, lsiprovider is updated in update 4.


              This is the content that is included in VMware vSphere Hypervisor 5.1 with IBM Customization:

              Vendor Provider/Driver Version



              Broadcom brcm 500.1.1-000000
              Broadcom misc-cnic-register 1.72.1.v50.1-1OEM.500.0.0.472560
              Broadcom net-bnx2 2.2.1l.v50.1-1OEM.500.0.0.472560
              Broadcom net-bnx2x 1.72.54.v50.2-1OEM.500.0.0.472560
              Broadcom net-cnic 1.72.50.v50.1-1OEM.500.0.0.472560
              Broadcom net-tg3 3.123b.v50.1-1OEM.500.0.0.472560
              Broadcom scsi-bnx2fc 1.72.51.v50.1-1OEM.500.0.0.406165
              Broadcom scsi-bnx2i 2.72.10.v50.2-1OEM.500.0.0.472560

              Brocade brcdprovider 3.0.3.0-0
              Brocade net-bna 3.0.3.0-1OEM.500.0.0.472560
              Brocade scsi-bfa 3.0.3.0-1OEM.500.0.0.472560

              Emulex ima-be2iscsi 4.1.334.3-1OEM.500.0.0.472629
              Emulex net-be2net 4.1.334.48-1OEM.500.0.0.472560
              Emulex scsi-be2iscsi 4.1.334.3-1OEM.500.0.0.472629
              Emulex scsi-lpfc820 8.2.2.126.50-1OEM.500.0.0.472560
              Emulex emulex-cim-provider 3.5.19.1-01

              LSI lsiprovider 500.04.V0.30-4000000

              Qlogic fchba 1.3.14-469512
              Qlogic net-qlge 1.0.0.47-1vmw.500.0.0.406165
              Qlogic ima-qla4xxx 500.2.01.31-1vmw.0.0.011509
              Qlogic net-qlcnic 5.0.741-1OEM.500.0.0.472560
              Qlogic scsi-qla2xxx 911.k1.1-26OEM.500.0.0.472560
              Qlogic scsi-qla4xxx 624.01.43-1OEM.500.0.0.472560

              IBM immpassthru 500-2ACE15AUS
              IBM concretejob 500-2ACE14DUS
              IBM filetrans 500-2ACE14DUS
              IBM fwupdate 500-2ACE14DUS
              IBM fupb 5-7.250
              IBM hwckvm 500-ESXI01ACN
              IBM pciinfo 500-PRIVATE
              IBM soibms 500-ESXI01ACN
              IBM brcmfwup 500-1.00.20120507
              IBM ilfu 500-1.01.20120504
              IBM ibmcustomization 3.0-0
              IBM ibmlic-esx-license 1.0-03
              VMware vSphere ESXi 5.x with IBM Customization Patch 4 change file (also applies to 5.1, indicated in file)

              fupb 5.01-7.280
              concretejob 500-2ACE16AUS
              filetrans 500-2ACE16AUS
              fwupdate 500-2ACE16AUS
              immpassthru 500-2ACE16AUS
              pciinfo 500-I03ACN
              Brcmfwup (facade) 500-1.00.20121017
              Brcm 500.2.0.3-000000
              ilfu (facade) 500-1.01.20121012
              lsiprovider 500.04.v0.32-0004
              fchba 1.3.14-469512
              qlcnic 5.0.741-1OEM.500.0.0.472560
              scsi-qla2xxx 911.k1.1-26OEM.500.0.0.472560
              net-be2net 4.2.327.0-1OEM.500.0.0.472560
              be2iscsi 4.2.327.0-1OEM.500.0.0.472560
              ima-be2isci 4.2.327.0-1OEM.500.0.0.472560

              Note the following for the update 4 patch:
              "This patch can be applied to the following IBM customized images only:
              * VMware ESXi 5.0 build 474610
              * VMware ESXi 5.0u1 build 623860
              * VMware ESXi 5.1 build 799733 <-- this is the build of the base IBM customized 5.1 installer
              "
            • DaveQ
              DaveQ
              96 Posts
              ACCEPTED ANSWER

              Re: ESXi 5.1 / IBM x3550M3 strange issues

              ‏2012-12-13T01:09:20Z  in response to SystemAdmin
              The firmware on the controller is old compared to what is available.

              What you indicated you have:
              LSI SAS2 MPT Controller SAS 2004
              v 07.18.01.05 <-- this is the UEFI driver version mentioned below.
              Firmware Type IR
              Firmware version 10.0.3.0
              What is available:
              BIOS 7.19.00.00
              Firmware 10.00.08.00
              UEFI Driver 7.18.04.03

              You can find the updates in firmware package 1.10.06 for the H1110 (listed under the ServeRAID section for the 3550 M3 (7944) on Fixcentral.

              I still have some questions about the VM's, but I would update the controller first.

              My main VM questions are (and my own opinion):
              1. What SCSI driver are you using in the VM? (LSI Parallel, LSI SAS, VMware Paravirtual) - I use PV for all of my 2008 R2 VMs, stellar performance.

              2. What BIOS mode are you using in the VM? (BIOS or EFI) - I wouldn't build VMs using EFI, too much overhead by allowing it more exposure to the host than it truly needs. A VM should be less not more right?
              • DaveQ
                DaveQ
                96 Posts
                ACCEPTED ANSWER

                Re: ESXi 5.1 / IBM x3550M3 strange issues

                ‏2012-12-13T01:14:37Z  in response to DaveQ
                Last follow-up after my carpet bomb responses... the version of firmware for the controller you have is the initial release (now 4 releases old). Definitely would update it. Basically everything in this change list applies since you're still running the initial release.

                http://delivery04.dhe.ibm.com/sar/CMA/XSA/03ase/0/ibm_fw_mpt2sas_h1110-1.10.06_linux_32-64.chg
                • SystemAdmin
                  SystemAdmin
                  9111 Posts
                  ACCEPTED ANSWER

                  Re: ESXi 5.1 / IBM x3550M3 strange issues

                  ‏2012-12-13T23:29:32Z  in response to DaveQ
                  Hey Dave,
                  Your correct and thanks much for pointing this out. When I used the BoMC, I used UXSP's and the pack was not up to date. Looking in the portal there was indeed many newer updates, including for the controller so I used latest available with BoMC this time and updated to latest. Also used the update patch for customized image and confirmed updates after. I used the customized image without the update during troubleshooting so I would think it's the controller fw was the fix. Now I know not use the UXSP's. I know the SCSI version don't matter now but I usually convert to PV for boot after install from LSI Logic SAS and continue with PV for data drives. Again thanks for the carpet booms ;)