Topic
  • 12 replies
  • Latest Post - ‏2010-09-30T18:07:13Z by tkutil
jgionta
jgionta
9 Posts

Pinned topic Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

‏2010-06-15T17:24:04Z |
Hello,

I have recently had many issues updating the firmware for both the IMM and UEFI of our Hs22 blade. Now I am very concerned about the state my server is in. Here is the story:

Yesterday I updated the IMM to version 1.10 (the latest version) After updating, the server rebooted after which I began having problems. The server at this point would fail to boot up through the UEFI environment. The error was:"3808003 Error Retrieving System Configuration from IMM DEBUG ERROR: C80000002:V3808003 I file not found". I saw that an UEFI driver was having an issue getting IMM information during the boot.

The system failed to boot 3x times so it took the default values and was able to boot. After looking through IBM readme docs I saw that my UEFI firmware was old and that I should not have a UEFI version less than 1.07 (which I had 1.05) for the IMM version 1.10. So I flashed the Hs22 with the latest UEFI 1.08. It was successful however now I am getting the following when I try to boot up my server: (this error is provided in the AMM server status. The blade's console output just shows it hanging)

06/15/10, 12:54:19 FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption
06/15/10, 12:53:19 Recovery FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

Looking at the Hs22 Problem/Determination Service Guide, it says I should update my uefi firmware to the latest. However:

1. The machine has the latest version of firmware
2. To flash the firmware you need to have a running Windows or Linux environment (as outline by the installation read me) but I can't boot the machine

Documentation said that the firmwares are triple redundant. Is there anyway to do a factory restore of the flash from AMM?

Thanks,

Jason
Updated on 2010-09-30T18:07:13Z at 2010-09-30T18:07:13Z by tkutil
  • Suzypco
    Suzypco
    71 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-06-15T19:14:37Z  
    Yes, you can update through AMM.
    The procedure is buried somewhere in one of the many documents on updating firmware.

    Here it is:
    1. Make sure AMM "sees" blade.
    In our shop we do this:
    Blade should have been plugged in and communicating with AMM for at least 15 minutes to ensure IMM on Blade and AMM are in sync. We view blade hardware VPD from the AMM. If the VPD shows the right # of processors (with or without their VPD, and lines for every DIMM (with or without their VPD), then it seems it is safe to proceed.
    2. Power up blade and hit F1 to enter Setup.
    3. With blade sitting at top screen in Setup
    4. Choose 'Update Firmware' under blade tasks.
    5. Give the filename of the linux.bin file for updating to the level you want
    6. Make the obvious choices on buttons to push
    In our shop we do these steps (not specifically called for, but an interpretation of requirements)
    a. Once firmware update from AMM is successful, exit out of Setup at the blade.
    b. Blade will continue boot process. Let it do this.
    c. Once blade has booted, do a reboot so that it will implement new UEFI image
    (NOTE: depending on the state of your blade, we have seen it take as long as 7 (Seven) minutes to get to the IBM splash screen containing the F-key prompts.
    If this does not work, call IBM Service.
  • jgionta
    jgionta
    9 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-06-15T20:11:31Z  
    • Suzypco
    • ‏2010-06-15T19:14:37Z
    Yes, you can update through AMM.
    The procedure is buried somewhere in one of the many documents on updating firmware.

    Here it is:
    1. Make sure AMM "sees" blade.
    In our shop we do this:
    Blade should have been plugged in and communicating with AMM for at least 15 minutes to ensure IMM on Blade and AMM are in sync. We view blade hardware VPD from the AMM. If the VPD shows the right # of processors (with or without their VPD, and lines for every DIMM (with or without their VPD), then it seems it is safe to proceed.
    2. Power up blade and hit F1 to enter Setup.
    3. With blade sitting at top screen in Setup
    4. Choose 'Update Firmware' under blade tasks.
    5. Give the filename of the linux.bin file for updating to the level you want
    6. Make the obvious choices on buttons to push
    In our shop we do these steps (not specifically called for, but an interpretation of requirements)
    a. Once firmware update from AMM is successful, exit out of Setup at the blade.
    b. Blade will continue boot process. Let it do this.
    c. Once blade has booted, do a reboot so that it will implement new UEFI image
    (NOTE: depending on the state of your blade, we have seen it take as long as 7 (Seven) minutes to get to the IBM splash screen containing the F-key prompts.
    If this does not work, call IBM Service.
    Thanks for the quick reply. I see what you are getting. The documentation outlines the ability to enter the IMM mode by hitting F1 at boot up. Unfortunately, the IMM screen doesn't display at boot up. On power up the blade goes directly to the UEFI initialization and I don't see an option to enter IMM. I can connect to the IMM through an SSH session. Is it possible to flash the firmware from an SSH session? Or force enter the IMM from the SSH session on the next power on?

    Thanks again for your help. It is greatly appreciated.

    Jason
  • jgionta
    jgionta
    9 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-06-15T20:17:29Z  
    • jgionta
    • ‏2010-06-15T20:11:31Z
    Thanks for the quick reply. I see what you are getting. The documentation outlines the ability to enter the IMM mode by hitting F1 at boot up. Unfortunately, the IMM screen doesn't display at boot up. On power up the blade goes directly to the UEFI initialization and I don't see an option to enter IMM. I can connect to the IMM through an SSH session. Is it possible to flash the firmware from an SSH session? Or force enter the IMM from the SSH session on the next power on?

    Thanks again for your help. It is greatly appreciated.

    Jason
    You can update through the IMM ssh session. There is an update command.

    I will try this.

    Thanks again for your help.

    Jason
  • PPriit
    PPriit
    11 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-06-15T23:47:56Z  
    • jgionta
    • ‏2010-06-15T20:17:29Z
    You can update through the IMM ssh session. There is an update command.

    I will try this.

    Thanks again for your help.

    Jason
    Hi,

    You can try UEFI update via AMM web interface.
    Download the latest firmware file to your desktop pc. Run it and choose extract to hard drive.
    Now got to AMM interface and under blade tasks, choose firmware update. Check mark the blade you want to upgrade and click to browse to the fw file. It is a file in your local desktop, there you extracted it before. file with .upd extension. Upload it to AMM and flash. Progress bar will show you how it's going.

    Rgds

    Priit
  • Suzypco
    Suzypco
    71 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-06-16T19:39:41Z  
    • PPriit
    • ‏2010-06-15T23:47:56Z
    Hi,

    You can try UEFI update via AMM web interface.
    Download the latest firmware file to your desktop pc. Run it and choose extract to hard drive.
    Now got to AMM interface and under blade tasks, choose firmware update. Check mark the blade you want to upgrade and click to browse to the fw file. It is a file in your local desktop, there you extracted it before. file with .upd extension. Upload it to AMM and flash. Progress bar will show you how it's going.

    Rgds

    Priit
    CAUTION:
    You left out a very important step.
    I was told by IBM Support that when flashing via AMM, YOU MUST BOOT BLADE, HIT SETUP (F1) KEY, and be at the Setup Main screen before flashing.

    Heads up to everyone -- we just discovered an new v1.08 UEFI firmware released on June 1, which lists a fix which was causing a boot hang after firmware update.
  • jgionta
    jgionta
    9 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-06-16T21:29:36Z  
    • Suzypco
    • ‏2010-06-16T19:39:41Z
    CAUTION:
    You left out a very important step.
    I was told by IBM Support that when flashing via AMM, YOU MUST BOOT BLADE, HIT SETUP (F1) KEY, and be at the Setup Main screen before flashing.

    Heads up to everyone -- we just discovered an new v1.08 UEFI firmware released on June 1, which lists a fix which was causing a boot hang after firmware update.
    Is there a way to get older versions of the UEFI firmwares? It seems the support pages don't have the firmwares versioned and only shows the latest version.

    Thanks,

    Jason
  • Suzypco
    Suzypco
    71 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-06-16T22:55:32Z  
    • jgionta
    • ‏2010-06-16T21:29:36Z
    Is there a way to get older versions of the UEFI firmwares? It seems the support pages don't have the firmwares versioned and only shows the latest version.

    Thanks,

    Jason
    Hi Jason,
    Given how many fixes of very serious problems have occurred, I would strongly advise against using anything but the latest release. (Which also means you need to be at IMM Level 1.10)
    If you read the change lists for UEFI and IMM you will see what I mean.

    Note that if you are using Emulex FC Adapters on your blades, you must be at Emulex FC card boot code level 5.11.x3 to go above UEFI 1.04. Failure to follow this advice will result in a hang during the UEFI Initialization part of the boot (before the splash screen)
  • jgionta
    jgionta
    9 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-06-17T20:18:07Z  
    • Suzypco
    • ‏2010-06-16T22:55:32Z
    Hi Jason,
    Given how many fixes of very serious problems have occurred, I would strongly advise against using anything but the latest release. (Which also means you need to be at IMM Level 1.10)
    If you read the change lists for UEFI and IMM you will see what I mean.

    Note that if you are using Emulex FC Adapters on your blades, you must be at Emulex FC card boot code level 5.11.x3 to go above UEFI 1.04. Failure to follow this advice will result in a hang during the UEFI Initialization part of the boot (before the splash screen)
    Thanks for the advice. However, I flashed the latest UEFI 1.08 and it cause the server to hang at the splash. So I think I need to revert to some earlier version at least for the time being.

    Thanks everyone for your help on this.
  • sturner
    sturner
    4 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-06-30T01:47:39Z  
    Here is a link to HS22 firmware best practices.

    ftp://ftp.software.ibm.com/systems/support/system_x_pdf/hs22_firmware_best_practice_guide.pdf

    Enjoy!
    Stu
  • Matt.Weatherford
    Matt.Weatherford
    8 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-07-19T04:47:32Z  
    • sturner
    • ‏2010-06-30T01:47:39Z
    Here is a link to HS22 firmware best practices.

    ftp://ftp.software.ibm.com/systems/support/system_x_pdf/hs22_firmware_best_practice_guide.pdf

    Enjoy!
    Stu
    throwing in .02 worth...

    We have a bladecenter H and HS22 blades - we've been having trouble booting from the media tray (CD and USB) as well as remote disk with the latest AMM
    firmware --- we actually had to downgrade the AMM to an earlier BPET50G version and things are looking ok again. 1.10 imm on all the blades.
  • H_Chassis
    H_Chassis
    15 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-07-19T14:36:14Z  
    throwing in .02 worth...

    We have a bladecenter H and HS22 blades - we've been having trouble booting from the media tray (CD and USB) as well as remote disk with the latest AMM
    firmware --- we actually had to downgrade the AMM to an earlier BPET50G version and things are looking ok again. 1.10 imm on all the blades.
    We're having similar issues with our HS22's booting from the Media tray. We have the lasted AMM, IMM and UEFI firmware. Remote disk seems to work for us.

    We also tried the HS22's in a different H chassis and it did the same thing....with older AMM firmware.

    IBM: please start working on this issue!
  • tkutil
    tkutil
    12 Posts

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-09-30T18:07:13Z  
    Obviously this issue has not been fixed. I'm having the "FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption" issue on one of my blades. Tech support had me turn on switch 5 on the system board to reset the uefi, but still cannot get the FW updated. Someone mentioned that the blade must be at the startup screen. I'm new to IBM, so what is the startup screen?