Troubleshooting
Problem
[This abstract has been truncated due to length constraints] Immediately after Cisco 3110G, 3110X or 3012 switch modules are upgraded to IOS 15.0(2)SE6, prior working switch configurations with Link State Tracking (LST) will fail to operate. This behavior is observed immediately after upgrading to 15.0(2)SE6 in both stacked and non-stacked switch configurations. The symptoms of this failure include the error disabling of the applicable LST downstream interfaces when the upstream interface remains stable, High CPU utilization and loss of network connectivity to/from Blade server interfaces along with the potential for switch and or switch-stack crash. Log messages similar to the ones below will be observed continually occurring on the switch: May21 15:34:19.728: %PM-4-ERR_DISABLE: lsgroup error detected on Gi1/0/1, putting Gi1/0/1 in err-disable state May 21 15:34:19.728: %PM-4-ERR_DISABLE: lsgroup error detected on Gi1/0/2, putting
Resolving The Problem
Source
RETAIN tip: H212598
Symptom
Immediately after Cisco 3110G, 3110X or 3012 switch modules are upgraded to IOS 15.0(2)SE6, prior working switch configurations with Link State Tracking (LST) will fail to operate.
This behavior is observed immediately after upgrading to 15.0(2)SE6 in both stacked and non-stacked switch configurations.
The symptoms of this failure include the error disabling of the applicable LST downstream interfaces when the upstream interface remains stable, High CPU utilization and loss of network connectivity to/from Blade server interfaces along with the potential for switch and or switch-stack crash.
Log messages similar to the ones below will be observed continually occurring on the switch:
| May 21 15:34:19.728: %PM-4-ERR_DISABLE: lsgroup error detected on Gi1/0/1, putting Gi1/0/1 in err-disable state May 21 15:34:19.728: %PM-4-ERR_DISABLE: lsgroup error detected on Gi1/0/2, putting Gi1/0/2 in err-disable state May 21 15:34:19.728: %PM-4-ERR_DISABLE: lsgroup error detected on Gi1/0/3, putting Gi1/0/3 in err-disable state May 21 15:34:19.804: %PM-4-ERR_RECOVER: Attempting to recover from lsgroup err-disable state on Gi1/0/1 May 21 15:34:19.804: %PM-4-ERR_RECOVER: Attempting to recover from lsgroup err-disable state on Gi1/0/2 May 21 15:34:19.804: %PM-4-ERR_RECOVER: Attempting to recover from lsgroup err-disable state on Gi1/0/3 May 21 15:34:20.743: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to down May 21 15:34:20.768: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/10, changed state to down May 21 15:34:20.785: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to down |
The switch will attempt to recover the error disabled interface(s) but will fail and the interface instability/flapping will continue.
The affected downstream interface Link and Protocol status will be in a 'down' state during the failure scenario.
Elevated CPU utilization by the 'Link State Group' process will also be observed and will be highly dependent on the number of downstream interfaces are configured in Link State Tracking. This will become an immediate issue with stacked switch configurations.
| Switch#sh proc CPU | ex 0.0 (truncated) CPU utilization for five seconds: 18%/0%; one minute: 21%; five minutes: 31% PID Runtime(ms)    Invoked     uSecs  5Sec  1Min  5Min TTY Process 229     123931       1402     88395 3.67% 6.15% 12.31%  0 Link State Group |
Excessive CPU utilization as a result of the Link State Group process can potentially result in network outage and or switch crash.
Affected configurations
The system is configured with one or more of the following IBM Options:
- Cisco Catalyst Switch Module 3012 for IBM BladeCenter, option part number 43W4395, any replacement part number
- Cisco Catalyst Switch Module 3110G for IBM BladeCenter, option part number 41Y8523, any replacement part number
- Cisco Catalyst Switch Module 3110X for IBM BladeCenter, option part number 41Y8522, any replacement part number
This tip is not software specific.
The 15.0(2)SE6 firmware for the Cisco 3110G, 3110X and 3012 is affected.
The system has the symptom described above.
Solution
This behavior was corrected in Cisco IOS release 15.0(2)SE7.
The file is or will be available by selecting the appropriate Product Group, type of System, Product name, Product machine type, and Operating system on IBM Support's Fix Central web page, at the following URL:Â Â Â Â
Workaround
Currently there are two workarounds available:
- Disable Link State Tracking globally on the switch module
Switch(config)#no link state track
- Downgrade the switches Cisco IOS to an earlier installed
version or level 15.0(2)SE5, if Link State Tracking is required for
network operations. This failure is not observed on level
15.0(2)SE5 release Cisco IOS.
Additional information
If the switches need to run 15.0(2)SE6, then Link State Tracking needs to be disabled to prevent the failure from occurring.
If Link State Tracking function is required by the client configuration for proper failover or High Availability (HA) operations, then the installation of the client's previous version IOS or 15.0(2)SE5 is required.
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
30 January 2019
UID
ibm1MIGR-5095382