How To
Summary
MAC Bit Errors on Cisco Fibre-channel switches are indications that a physical link is unhealthy. This document provides a procedure for troubleshooting these errors to identify the most likely failing component.
Objective
When IBM detects a callhome event on your Cisco MDS Director or switch similar to the following:
Event Description:MODULE_WARNING Module N (serial: XXXXXXXXXXXX) reported
warnings on ports fcN/P (Fibre Channel) due to MAC Bit error exceeded
threshold in device 154 (device error 0xc9a00503)
where the serial number is the serial number of your switch and N/P is a slot and port on the switch, this indicates an unhealthy link for that port. MAC Bit errors are encoding errors. These occur when you have a marginal SFP, cable or more rarely a switch port. Left unfixed the link will degrade further over time.
Follow these steps to troubleshoot the problem to determine the failing component:
1. Collect additional data to try and determine the failing component using this command:
Where N/P is the slot and port from the error message:
show interface fcN/P transceiver details
You will see output similar to this:
— No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68
— SFP Diagnostics Information:
—----------------------------------------------------------------------------
— Alarms Warnings
— High Low High Low
—----------------------------------------------------------------------------
— Temperature 24.40 C 75.00 C -5.00 C 70.00 C 0.00 C
— Voltage 3.34 V 3.63 V 2.97 V 3.46 V 3.13 V
— Current 8.26 mA 11.80 mA 4.00 mA 10.80 mA 5.00 mA
— Tx Power -2.57 dBm 1.70 dBm -13.00 dBm -1.30 dBm -9.00 dBm
— Rx Power -3.69 dBm 3.00 dBm -15.90 dBm 0.00 dBm -11.90 dBm
If the receive (Rx Power in the above output) levels are too low, the cabling is suspect. Valid values are:
| Speed (Gbps) | Minimum Receive Power |
|---|---|
| 8 | -6 dBm |
| 16 | -12 dBm |
| 32 | -11 dBm |
2. Check for CRC errors on the switch interface
show interface fcN/P counters detailed | in CRC
This will show you any CRC errors on the link. If there are CRC errors, the link is certainly bad.
3. Clear the counters by running
clear counters interface all
debug system internal clear-counters all
Wait for some hours - at least 12 or 24 hours and see if the CRC errors increment, and how quickly they increment. This will provide a measure of how severe the problem is.
4. (if possible) move the link to a new port on the switch to see if the problem reoccurs on the new port.
If you move only the cable and the problem occurs again, then the SFP is not the problem, the cable is. If you move both the cable and the SFP and the problem goes away, the switch port is the problem.
5: If you have completed steps 1-4 and the errors disappeared after moving the cable and SFP to the new port, or you require troubleshooting assistance, you can open a ticket against the switch here:
https://www-946.ibm.com/support/servicerequest/Home.action
If you have problems with multiple links you should prioritize fixing the links in this order:
1. ISLs
2. Storage
3. Critical or high-volume hosts
4. Non-critical or low-volume hosts
Document Location
Worldwide
[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"STAPUZ","label":"Cisco MDS 9132T 32G Fabric Switch"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"ST2GRX","label":"Cisco MDS 9718 Multilayer Director"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB26","label":"Storage"}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"STTQW4","label":"Cisco MDS 9706 Multilayer Director"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSU6LN","label":"Cisco MDS 9710 Multilayer Director"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB26","label":"Storage"}},{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSY5QTU","label":"Cisco MDS 9250i Multiservice Fabric Switch"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB26","label":"Storage"}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"STMKQW","label":"Cisco MDS 9148 Multilayer Fabric Switch"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"STTQV4","label":"Cisco MDS 9148S 16G Multilayer Fabric Switch"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"STMKRM","label":"Cisco MDS 9222i Multi-Service Switch"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"ST5PVM","label":"Cisco MDS 9396S 16G Multilayer Fabric Switch"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STTQ3Y","label":"Cisco MDS 9513 Multiplayer Director"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB26","label":"Storage"}}]
Was this topic helpful?
Document Information
Modified date:
11 April 2022
UID
ibm16251337