Troubleshooting Blades that fail to power on in chassis - IBM BladeCenter

Resolving The Problem

Source

RETAIN tip: H19470

Symptom

One or more of the blades in any BladeCenter Chassis either will not power on or will POST very slowly.

Affected configurations

The system may be any of the following IBM servers:

• BladeCenter Chassis, type 8677, any model
• BladeCenter H, type 7989, any model
• BladeCenter H, type 8852, any model
• BladeCenter HC10, type 7996, any model
• BladeCenter HS20, type 1883, any model
• BladeCenter HS20, type 1884, any model
• BladeCenter HS20, type 7981, any model
• BladeCenter HS20, type 8678, any model
• BladeCenter HS20, type 8832, any model
• BladeCenter HS20, type 8843, any model
• BladeCenter HS21 XM, type 1915, any model
• BladeCenter HS21 XM, type 7995, any model
• BladeCenter HS21, type 1885, any model
• BladeCenter HS21, type 8853, any model
• BladeCenter HS40, type 8839, any model
• BladeCenter HT, type 8740, any model
• BladeCenter HT, type 8750, any model
• BladeCenter JS20, type 8842, any model
• BladeCenter JS21, type 7988, any model
• BladeCenter JS21, type 8844, any model
• BladeCenter LS20, type 8850, any model
• BladeCenter LS21, type 7971, any model
• BladeCenter LS41, type 7972, any model
• BladeCenter QS20, type 0200, any model
• BladeCenter QS21, type 0792, any model
• BladeCenter T, type 8720, any model
• BladeCenter T, type 8730, any model

This tip is not option specific.

This tip is not software specific.

Additional information

Troubleshooting blades that will not power on or have POST problems.

Attention: Always inspect the chassis connectors at the back of the blade before moving it to another slot in a chassis. If blade connector damage is found then immediately inspect the chassis midplane behind that blade slot with a flashlight for midplane connector damage.

For blades that will not power on or POST slowly, start by checking the Management Module (MM) Event Log and VPD for abnormalities. If the MM Event logs shows "SP Communication Error" or "Kernel Mode," refer to the document "troubleshooting SP Comm and kernel mode errors" for the recovery procedure. They must be solved before trying to address other symptoms. Other possible symptoms are as follows:

• MM cannot read VPD of blade and will not power on the blade.
• Blade will not power on and no errors are present or an error indicates unsupported hardware is installed.
• Event log says that there is insufficient power to power on the blade or the blade does not have permission to power on.
• Amber Fault light on blade.
• No light on front panel.
• Blades take a long time to complete POST.

No VPD Data Showing For the Blade(s) in the MM

1. Log into the MM web interface, under "Monitors" select Firmware VPD. Here the option to "Reload VPD" for all blades or a particular blade is available. If the blade or blades can be selected via the drop down arrow, make a selection and click the Reload VPD button. This option allows the firmware VPD for a blade or blades to be re-read.

   Note: If the blade or blades can not be selected due to a SP Comm error, refer to "troubleshooting SP Comm and kernel mode errors" for the recovery procedure.

2. Restart the Service Processor for any blade that does not present complete VPD . To do this, navigate to BladeTasks, then Power/Restart, and then Restart Blade Systems Management Processor and select the blade in question.
3. If there are two MMs in the chassis, fail over to the redundant MM . If there is only one MM in the chassis, move it to the other slot. Allow up to five minutes for the blade discover process to complete.

Blade will not power on and no errors are present, or an error indicates unsupported hardware is installed or an "incompatible configuration" exists

Verify that the version of MM firmware installed supports the blade that will not boot. Refer to the IBM BladeCenter Support site for the latest management module firmware, such as the following example at URL:

http://www.ibm.com/systems/support/supportsite.wss/brandmain?brandind=5000020

Check the change history of the MM firmware to verify support for the blades installed. If the management module firmware does not support the blade(s) installed, upgrade the management module firmware by going to the IBM support web site. The "incompatible configuration" message indicates that a daughter card is connecting to a switch type that it does not support. For example, a blade with an Ethernet daughter card in a BladeCenter (Type 8677) chassis that has fibre switches in I/O slots 3 and 4.

MM Event Log indicates there insufficient power for the blade to power on, or the blade does not have permission to power on

1. If the MM indicates that there is insufficient power to power on the blade, check the status of both power domains in fuel gauge. Verify that enough power is available to power on the blade. If the MM indicates power supply problems, review the document "Troubleshooting power supply problems."

   Note: Power Management policies are configurable per power domain in fuel gauge.

   Going from the most restrictive power policy to the least restrictive, the three policies are:

   • "Redundancy without Performance Impact" is a power management policy where the power limit is fixed at a single power module's output. The maximum power of all blades permitted to power on must be within a single power module's output at all times. Hence, this is true whether there are one or two power modules present in the domain.
   • "Redundancy with Potential Performance Impact" is a power management policy that will allow additional blades to power on even when the power draw has exceeded the limits of a single supply. For this to work, the CPUs in the blade must be capable of throttling. Throttling is a feature which allows CPUs to reduce their power consumption under certain conditions.
   • "Non-redundant" or over-subscription is a power management policy that allows power loads to be shared between two functioning power modules in a power domain. It is also the least restrictive power management policy. Hence, if a power module fails with in a power domain with this policy set, the remaining power module may not be able to fulfill the power demand. Therefore, the remaining power module would shut down unless the blades could reduce their demand for power. Depending on the power domain, not only could blades go down, but the MM's and the I/O modules could go as well.
     
2. If a blade is spanning power domain 1 and 2 (for the BladeCenter (Type 8677) chassis, a blade in slots 6-7 span domains. For the BladeCenter H (Type 8852) chassis, blade occupying slots 7-8 span two domains). Try moving the blade to a set of slots that are fully contained in one domain. If this resolves it, the problem may be a MM or Service process firmware defect. Update the MM and blade service processor code or call IBM support more more assistance.

Amber Fault light on blade

Consult the Problem Determination and Service Guide for the particular blade.

No light on front panel

1. Put a working blade in this slot . If the LED on the front panel comes on, the slot is delivering power.
2. Switch the bezel with another blade that is working . That will verify whether the bezel is working.
3. If the above steps isolate the failure to the blade, it is likely that the Baseboard Management Controller (BMC) is not functioning. If this failure occurred during a BMC flash update or is occurring to several blades at the same time, suspect corrupt BMC firmware and contact IBM support for methods to repair the BMC firmware. If the failure is only occurring one one blade, suspect a BMC hardware failure and replace the planar.

Unable to power the blade on using the front panel power on/off button

1. Log into the MM and navigate to Blade Tasks, then Power/Restart. Power on the blade in question.
2. Select Remote Control and start a remote session to the blade server in question. Verify whether the blade completes POST. If there are no lights on the front bezel and the blade is functional, there is a hardware failure in the bezel. Order the proper Bezel assembly for the blade server in question.
3. If the blade fails to complete POST, note the error or symptom and contact IBM support.

Blades take a long time to complete POST

Note: Characters on the screen printing out one at a time, yet very slowly, is due to Serial Over LAN (SOL) misconfiguration. See the SOL configuration documents for the blade in question to correct this behavior. The steps below assume the characters appear on the screen at normal speed, but the POST process is not completing in the time expected.

1. Restart the MM.
2. Fail-over to redundant MM. If the blades POST normally continue to step 9.
3. MM firmware may be corrupt; re-flash to current levels.
4. BIOS or BMC firmware may be corrupt; re-flash to current levels.
5. Remove the media tray. If the blades POST normally, continue to step 11.
6. Reset blade BIOS to defaults.
7. Reinstall the media tray and reseat the affected blades.
8. Reseat the MM's. If the problem is not solved yet, contact IBM support.
9. Test MM 's, one at a time, in each bay.
10. MM firmware may be corrupt; flash each MM to current levels independently.
11. Reseat all media tray components. Reinstall and test.
12. Remove the diskette drive from the media tray and reinsert the media tray into the chassis. If the blade POSTS normally contact support to have the diskette drive replaced.
13. Remove the CD-ROM/DVD-ROM drive from the media tray and reinsert the media tray into the chassis. If the blade POSTs normally contact support to have the CD-ROM/DVD-ROM replaced.
14. Remove the CD-ROM/DVD-ROM interposer card from the media tray and reinsert the media tray into the chassis. If the blade POSTs normally contact support to have the CD- ROM/DVD-ROM interposer card replaced.
15. Remove the customer interface card from the media tray and reinsert the media tray into the chassis. If the blade POSTs normally contact support to have the the customer interface card replaced.