Intermittent connection loss to BladeCenter Ethernet switch - IBM BladeCenter (Type 8677) and BladeCenter T

Resolving The Problem

RETAIN tip: H182909

Users on remote IP subnets attempting to manage an Ethernet Switch Module (ESM) may experience intermittent loss of connectivity to the switch.
 
The failure is usually seen in the following environment:  

1. Both Management Module (MM) IP interfaces and the ESM or CIGESM IP interface are in the same IP subnet.
2. In-band management of the ESM or CIGESM is enabled (that is, "External Management Over all Ports"=Enabled).
3. MM and ESM or CIGESM ports are connected to the same layer two network.
4. The user managing the ESM or CIGESM is connected across a router.
   

The system may be any of the following IBM eServer servers:  

• BladeCenter, type 8677, any model
• BladeCenter T, type 8730, any model
• BladeCenter T, type 8720, any model

The system is configured with one of the following RSA options:  

• 4-Port Gigabit Ethernet Switch Module option part number 48P7054 or 13N0568, replacement part number (FRU) 26K6482 replaces 59P6620, 13N0557, or 26K6482
• Cisco Gigabit Intelligent Ethernet Switch Module option part number 13N2286, replacement part number (FRU) 13N2285

Root cause has yet to be identified.

IBM strongly recommends disabling "External Management Over all Ports" until the root cause of this problem has been identified:
 
Out-of-band management of ESM or CIGESM: (Default Configuration)  

• "External Management Over all Ports" = Disabled
• Internal MM interface, external MM interface, and ESM or CIGESM management interface must all use the same IP subnet.

Some customers desire in-band management of the ESM. For those customers, configuration guidance for in-band management has also been included below. However, if the failure persists, customers will need to Disable "External Management Over all Ports". In-band management of ESM or CIGESM. (ESM or CIGESM management traffic is isolated)  

• "External Management Over all Ports" = Enabled
• The ESMs or CIGESMs IP management interface must be configured with a unique IP subnet.

That is, the subnet is not used for production traffic or for the MM interfaces. This IP subnet must be isolated on its own VLAN in the upstream switch as well as within the ESM or CIGESM. 802.1Q tagging may be used to achieve this isolation. The MM IP interfaces may be in the same IP subnet as the blades.
 
Notes:

1. When "External Management Over all Ports" = Enabled", the external MM port should not be in the same broadcast domain with the external ESM or CIGESM ports.
2. The MM internal and external interface must always be configured with an IP addresses in the same IP subnet. Do not try to resolve this problem by putting the two MM interfaces in different IP subnets.

The default configuration of the MM provides out-of-band connectivity between devices on the network and ESMs or CIGESMs installed in the chassis. To accomplish this, the MM performs a Proxy ARP function on certain traffic and then passes it along. The MM will only respond to ARPs that are either addressed to the IP address of the ESM or the CIGESM, or originate from the IP address of the ESM or the CIGESM. The MM will not allow production traffic to pass through.
 
When remote management of the ESM or CIGESM is also enabled, both the MM and ESM or CIGESM will respond to ARP requests destined for the ESMs or CIGESMs. On a flat network, there is no way to predict which MAC address will be found in forwarding tables and ARP caches of upstream devices. In fact, these mappings are likely to change with time. This condition has led to connectivity loss when ESMs or CIGESMs and management workstation are separated by a router.