What conditions can create "ECH_PORT_OUT_OF_SYNC" errors on AIX/VIOS lpar when the attached to a Cisco Switch that is logging "FWM-2-STM_LOOP_DETECT" messages?

This message indicates that the switch receives frames with the same source MAC address on these two ports and that the swtich learns the same MAC address on these ports at a very high speed. The switch detects this as a loop.

For more information about this message, refer to following link.

http://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/116200-qanda-nexus5000-00.pdf

Following symptoms may be observed on AIX/VIOS lpars when "FWM-2-STM_LOOP_DETECT" message is logged in the switch log.

(1) When an etherchannel in 802.3ad mode is set up, netstat/entstat shows the LACP Partner "Synchronization: OUT_OF_SYNC", errpt shows ECH_PORT_OUT_OF_SYNC and the "Received LACPDUs" not incrementing. The etherchannel may or may not be part of a SEA.

(2) netstat/entstat shows very large numbers of "Hypervisor Send Failures" on the Virtual Ethernet Adapters (VEAs) under a Shared Ethernet Adapter (SEA). The send failures normally would be roughly equal to the sum of the "Hypervisor Receive Failures" on all of the virtual clients but in this case, they will not be.

For example:


(3) The packet capture (i.e. iptrace or tcpdump) on the SEA show a large number of conversations at the ethernet level to be one direction. There will be packets in the capture going from MAC address A to MAC address B but zero packets going from MAC address B to MAC address A. Under normal conditions, such conversations would be rare and only a few packets. But, in the case being described, these one-way conversations will far out weight the normal two-way conversations.

(4) When pinging a system, duplicates may be seen:

For example:

Possible Root Cause:

The reason why the Cisco Switch is logging "FWM-2-STM_LOOP_DETECT" messages, disabling MAC address learning on all switch ports, needs to be determined and corrected. For example, this could be the result of another, unrelated, server ports being set-up for etherchannel but using a mode that is not matching what is set-up on the switch ports.  Or the port group network switch may be incorrectly configured.  The number of ports set for the port group do not match the number of ports on the etherchannel adapter.

Miscellaneous Information:
When MAC learning is disabled, LACPDUs are not sent from the switch ports even when configured to do so. When the issue that is causing the FWM-2-STM_LOOP_DETECT has been resolved, LACPDUs still may not start flowing.  A reboot of the system is known to resolve the issue. It may be that disabling and re-enabling the affected switch ports may also resolve the issue.