System-Management Processor for Node XX communication is offline - IBM Flex System

Resolving The Problem

Source

RETAIN tip: H213671

Symptom

During the Integrated Management Module (IMM) firmware (F/W) update process, the Maintenance Mode notification is being sent to the Chassis Management Module (CMM) indicating the Inter-Integrated Circuit (i2c) listener is offline. However, the mode is being ignored in the existing CMM F/W. All i2c traffic to the impacted bus are blocked and later the bus becomes isolated to prevent system contamination. Thus, all outbound i2c requests (such as MAC Addresses) from node layer would not be transmitted to the IMM and the node discovery process fails.

After the IMM F/W upgrade, the node fails the discovery process with error event 'The system-management processor for Node xx communication to the CMM is offline.' The Checklog LED of the rear and front panels of the Flex Chassis would be illuminated. Also, the graphic of impacted node is marked RED.

Affected configurations

The system is configured with one or more of the following IBM Options:

• Flex System Chassis Management Module, Option part number 68Y7029, any replacement part number

This tip is not system specific.

This tip is not software specific.

Solution

This behavior has been fixed in CMM Build ID: 2PET12T.

The file is or will be available by selecting the appropriate Product Group, type of System, Product name, Product machine type, and Operating system on IBM Support's Fix Central web page, at the following URL:    

• http://www.ibm.com/support/fixcentral/

Chassis Management Module firmware needs to be enhanced to prevent all outgoing i2c traffics to the blade during the Maintenance Mode, but continue to allow the configuration commands on General Purpose Input Output (GPIO) registers which are needed for resetting the peripheral i2c buses in case of system hang.

Workaround

Follow these best practices for firmware updates:

Do not perform the IMM firmware update during the node discovery process. The 'D' on a node in the chassis map view indicates the node is in the discovery process.

Do not perform the 'Virtual Reseat' feature during the IMM Firmware update.

Additional information

The IMM Maintenance Mode is designed to notify other applications (such as CMM) that the IMM is not listening to its I2C port. If ignored, the i2c traffic would be blocked and triggers the i2c bus isolation logic (from CMM Firmware) which leads to failure of node communication.

In the normal operating node cycle, there is almost zero i2c traffic from the CMM to the IMM Service Processor, thus the IMM F/W update where the Maintenance Mode is raised would have little or no impact on i2c bus and bus isolation would never occur. When IMM Firmware update is done, its Service Processor would be automatically restarted and CMM will be notified so it can restart the discovery process.