Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
12 replies Latest Post - ‏2010-09-30T18:07:13Z by tkutil
jgionta
jgionta
9 Posts
ACCEPTED ANSWER

Pinned topic Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

‏2010-06-15T17:24:04Z |
Hello,

I have recently had many issues updating the firmware for both the IMM and UEFI of our Hs22 blade. Now I am very concerned about the state my server is in. Here is the story:

Yesterday I updated the IMM to version 1.10 (the latest version) After updating, the server rebooted after which I began having problems. The server at this point would fail to boot up through the UEFI environment. The error was:"3808003 Error Retrieving System Configuration from IMM DEBUG ERROR: C80000002:V3808003 I file not found". I saw that an UEFI driver was having an issue getting IMM information during the boot.

The system failed to boot 3x times so it took the default values and was able to boot. After looking through IBM readme docs I saw that my UEFI firmware was old and that I should not have a UEFI version less than 1.07 (which I had 1.05) for the IMM version 1.10. So I flashed the Hs22 with the latest UEFI 1.08. It was successful however now I am getting the following when I try to boot up my server: (this error is provided in the AMM server status. The blade's console output just shows it hanging)

06/15/10, 12:54:19 FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption
06/15/10, 12:53:19 Recovery FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

Looking at the Hs22 Problem/Determination Service Guide, it says I should update my uefi firmware to the latest. However:

1. The machine has the latest version of firmware
2. To flash the firmware you need to have a running Windows or Linux environment (as outline by the installation read me) but I can't boot the machine

Documentation said that the firmwares are triple redundant. Is there anyway to do a factory restore of the flash from AMM?

Thanks,

Jason
Updated on 2010-09-30T18:07:13Z at 2010-09-30T18:07:13Z by tkutil
  • Suzypco
    Suzypco
    71 Posts
    ACCEPTED ANSWER

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-06-15T19:14:37Z  in response to jgionta
    Yes, you can update through AMM.
    The procedure is buried somewhere in one of the many documents on updating firmware.

    Here it is:
    1. Make sure AMM "sees" blade.
    In our shop we do this:
    Blade should have been plugged in and communicating with AMM for at least 15 minutes to ensure IMM on Blade and AMM are in sync. We view blade hardware VPD from the AMM. If the VPD shows the right # of processors (with or without their VPD, and lines for every DIMM (with or without their VPD), then it seems it is safe to proceed.
    2. Power up blade and hit F1 to enter Setup.
    3. With blade sitting at top screen in Setup
    4. Choose 'Update Firmware' under blade tasks.
    5. Give the filename of the linux.bin file for updating to the level you want
    6. Make the obvious choices on buttons to push
    In our shop we do these steps (not specifically called for, but an interpretation of requirements)
    a. Once firmware update from AMM is successful, exit out of Setup at the blade.
    b. Blade will continue boot process. Let it do this.
    c. Once blade has booted, do a reboot so that it will implement new UEFI image
    (NOTE: depending on the state of your blade, we have seen it take as long as 7 (Seven) minutes to get to the IBM splash screen containing the F-key prompts.
    If this does not work, call IBM Service.
    • jgionta
      jgionta
      9 Posts
      ACCEPTED ANSWER

      Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

      ‏2010-06-15T20:11:31Z  in response to Suzypco
      Thanks for the quick reply. I see what you are getting. The documentation outlines the ability to enter the IMM mode by hitting F1 at boot up. Unfortunately, the IMM screen doesn't display at boot up. On power up the blade goes directly to the UEFI initialization and I don't see an option to enter IMM. I can connect to the IMM through an SSH session. Is it possible to flash the firmware from an SSH session? Or force enter the IMM from the SSH session on the next power on?

      Thanks again for your help. It is greatly appreciated.

      Jason
      • jgionta
        jgionta
        9 Posts
        ACCEPTED ANSWER

        Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

        ‏2010-06-15T20:17:29Z  in response to jgionta
        You can update through the IMM ssh session. There is an update command.

        I will try this.

        Thanks again for your help.

        Jason
        • PPriit
          PPriit
          11 Posts
          ACCEPTED ANSWER

          Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

          ‏2010-06-15T23:47:56Z  in response to jgionta
          Hi,

          You can try UEFI update via AMM web interface.
          Download the latest firmware file to your desktop pc. Run it and choose extract to hard drive.
          Now got to AMM interface and under blade tasks, choose firmware update. Check mark the blade you want to upgrade and click to browse to the fw file. It is a file in your local desktop, there you extracted it before. file with .upd extension. Upload it to AMM and flash. Progress bar will show you how it's going.

          Rgds

          Priit
          • Suzypco
            Suzypco
            71 Posts
            ACCEPTED ANSWER

            Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

            ‏2010-06-16T19:39:41Z  in response to PPriit
            CAUTION:
            You left out a very important step.
            I was told by IBM Support that when flashing via AMM, YOU MUST BOOT BLADE, HIT SETUP (F1) KEY, and be at the Setup Main screen before flashing.

            Heads up to everyone -- we just discovered an new v1.08 UEFI firmware released on June 1, which lists a fix which was causing a boot hang after firmware update.
            • jgionta
              jgionta
              9 Posts
              ACCEPTED ANSWER

              Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

              ‏2010-06-16T21:29:36Z  in response to Suzypco
              Is there a way to get older versions of the UEFI firmwares? It seems the support pages don't have the firmwares versioned and only shows the latest version.

              Thanks,

              Jason
              • Suzypco
                Suzypco
                71 Posts
                ACCEPTED ANSWER

                Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

                ‏2010-06-16T22:55:32Z  in response to jgionta
                Hi Jason,
                Given how many fixes of very serious problems have occurred, I would strongly advise against using anything but the latest release. (Which also means you need to be at IMM Level 1.10)
                If you read the change lists for UEFI and IMM you will see what I mean.

                Note that if you are using Emulex FC Adapters on your blades, you must be at Emulex FC card boot code level 5.11.x3 to go above UEFI 1.04. Failure to follow this advice will result in a hang during the UEFI Initialization part of the boot (before the splash screen)
                • jgionta
                  jgionta
                  9 Posts
                  ACCEPTED ANSWER

                  Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

                  ‏2010-06-17T20:18:07Z  in response to Suzypco
                  Thanks for the advice. However, I flashed the latest UEFI 1.08 and it cause the server to hang at the splash. So I think I need to revert to some earlier version at least for the time being.

                  Thanks everyone for your help on this.
  • sturner
    sturner
    4 Posts
    ACCEPTED ANSWER

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-06-30T01:47:39Z  in response to jgionta
    Here is a link to HS22 firmware best practices.

    ftp://ftp.software.ibm.com/systems/support/system_x_pdf/hs22_firmware_best_practice_guide.pdf

    Enjoy!
    Stu
    • Matt.Weatherford
      Matt.Weatherford
      8 Posts
      ACCEPTED ANSWER

      Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

      ‏2010-07-19T04:47:32Z  in response to sturner
      throwing in .02 worth...

      We have a bladecenter H and HS22 blades - we've been having trouble booting from the media tray (CD and USB) as well as remote disk with the latest AMM
      firmware --- we actually had to downgrade the AMM to an earlier BPET50G version and things are looking ok again. 1.10 imm on all the blades.
      • H_Chassis
        H_Chassis
        15 Posts
        ACCEPTED ANSWER

        Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

        ‏2010-07-19T14:36:14Z  in response to Matt.Weatherford
        We're having similar issues with our HS22's booting from the Media tray. We have the lasted AMM, IMM and UEFI firmware. Remote disk seems to work for us.

        We also tried the HS22's in a different H chassis and it did the same thing....with older AMM firmware.

        IBM: please start working on this issue!
  • tkutil
    tkutil
    12 Posts
    ACCEPTED ANSWER

    Re: Hs22: FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption

    ‏2010-09-30T18:07:13Z  in response to jgionta
    Obviously this issue has not been fixed. I'm having the "FW/BIOS, firmware progress (Firmware Error) FW/BIOS ROM corruption" issue on one of my blades. Tech support had me turn on switch 5 on the system board to reset the uefi, but still cannot get the FW updated. Someone mentioned that the blade must be at the startup screen. I'm new to IBM, so what is the startup screen?