Troubleshooting Processor faults on AMD and Intel processor - IBM BladeCenter

Resolving The Problem

Source

RETAIN tip: H183268

Issue

Troubleshooting I-Errors and Machine Checks on Intel and AMD blades.

Affected configurations

The system may be any of the following IBM servers:

• BladeCenter Chassis, type 7967, any model
• BladeCenter Chassis, type 8677, any model
• BladeCenter H, type 8852, any model
• BladeCenter HS20, type 1883, any model
• BladeCenter HS20, type 1884, any model
• BladeCenter HS20, type 7981, any model
• BladeCenter HS20, type 8678, any model
• BladeCenter HS20, type 8832, any model
• BladeCenter HS20, type 8843, any model
• BladeCenter HS21, type 8853, any model
• BladeCenter HS40, type 8839, any model
• BladeCenter JS20, type 8842, any model
• BladeCenter JS21, type 8844, any model
• BladeCenter LS20, type 8850, any model
• BladeCenter LS21, type 7971, any model
• BladeCenter LS41, type 7972, any model
• BladeCenter QS20, type 0200, any model
• BladeCenter T, type 8720, any model
• BladeCenter T, type 8730, any model

This tip is not option specific.

This tip is not software specific.

Additional information

For I-Errors and machine checks on Intel CPUs:

An I-Error is a generic error created by the Intel CPU. The error occurs when the CPU sees an error anywhere on the system. This does necessarily mean that the CPU caused the error. In general, the root cause of the error is one of the following (ordered roughly by frequency they have been seen in the field):

1. I/O Device Drivers
2. Hard disk drive failure
3. PCI device time-outs
4. Unterminated SCSI bus
5. Real CPU internal failure
6. Power supply issues

Most CPU I-Errors (>99%) are the result of a fault outside of the CPU, usually a non-responding I/O device. When a real CPU Internal hardware fault occurs, it will be flagged by the LightPath function of the blade and light the appropriate CPU fault LED. BIOS will read the CPU?s Machine Check registers and pass the contents to the Management Module (MM) via the BMC (if a fault was detected). BIOS will also look at the I/O stalled bit and inform the BMC of its results. The BIOS and BMC will execute an "I-Error" algorithm to better fault isolate and restart the system at the detection and a CPU I-Error. If the CPU fault LED is not on, do not replace the CPU.

When an I-Error occurs, the MM Event log should contain data that appears as follows:

Notice that after "Status:", there are 16 hex numbers. If the third hexadecimal character from the right (in this case, the number "4"), has a value of 4, 5, 6, 7, c, d, e or f, the CPU is announcing that it stopped because an I/O stream that it needs has stalled. In this case, the CPU is working correctly, but some other device has caused an issue.

If the "Status:" value above does not indicate an I/O stall, replace the processor. If it indicates an I/O stall, take the following steps to isolate the source of the stall:

1. Check operating system logs for hard disk drive errors. If there are hard disk drive errors at the time of the I/O timeout, compare the time stamp of the operating system log and the time stamp of the Management Module log. If they are at the same time then it is possible that the fault was caused by the hard disk drive. hard disk drive failures have been seen to cause an I/O fault and generate the errors above.
2. Check the on-board Ethernet, Ethernet daughter cards, and Fibre daughter cards, if they are at the current level of firmware and using the latest device drivers. Ensure that the blade is running the current BIOS and Service Processor code, since some blades have had BIOS fixes for I-Errors.
3. Reseat any daughter cards installed, and Blade Storage Expansion unit, if installed.
4. Ensure that if there is not a Blade Storage Expansion (BSE) unit installed, that the BSE terminator that ships from the factory is installed.
5. Run the Diagnostics.
6. Try to understand what is being stressed when the failure occurs. If the failure occurs only with doing heavy I/O over an option card, then replaing the option card may be the next step, if the previous steps have not resolved the issue.
7. If you have done the previous steps and are still seeing an I/O timeout as indicated by the MachineCheck in the MM log, open up a case with the support center. Document what was going on with the system at the time of failure, for example, high network traffic, database indexing, idle, etc.
8. Ensure that all of the blades firmware is at the latest level.

For Machine Checks on AMD blades:

In the AMD architecture, errors are handled in a machine check mechanism internal to the processor. These errors can be either recoverable or non-recoverable and have multiple sources. The source of the machine check can be internal to the processor such as parity or ECC errors in caches and TLB's, ECC or chipkill errors from main memory DRAM, or external bus errors. Machine checks that are not recoverable trigger a machine check exception (MCE). These are often uncorrectable memory errors, but also may rarely be fatal errors internal to the processor or on the HyperTransport bus. Due to the HyperTransport interconnection links used in Opteron processor-based systems, an error reporting state called "Sync Flood" is used to halt the system. Once in this state, the system must get a reset to reboot and begin operation again.

Below is an example of an uncorrectable memory error on an AMD blade server reported in the MM log. In the case of uncorrectable errors, the failing bank (usually a pair) of DIMM's are disabled, since it cannot be determined which particular DIMM actually failed with this memory controller. Both DIMM's are to then be replaced with new DIMM's (not swapped) and then the disabled memory bank can be enabled in POST BIOS CMOS settings (press F1 during bootup).

The blades LED lightpath should also light the two LED associated with the DIMM's identified in the MM logs. Other error types (e.g. a CPU failure) will be documented in the MM Event Log and the appropriate lightpath LEDs will be lit.