Troubleshooting
Problem
Why is my Shared Ethernet Adapter (SEA) in LIMBO state?
Symptom
SEA shows in limbo state
$ entstat -all ent#|grep -i state > where ent# is the SEA
State: LIMBO
Device State: Dead
LAN State: Operational
LAN State: Operational
$ entstat -all ent#|grep -i state > where ent# is the SEA
State: LIMBO
Device State: Dead
LAN State: Operational
LAN State: Operational
The following error may be found in the VIOS errlog:
LABEL: VIOS_SEA_ADAP_FAIL
IDENTIFIER: 8D424E06
Date/Time: Mon Jun 24 09:56:59 2015
...
Failure Causes
ADAPTER FAILURE
Recommended Actions
SWITCHING TO LIMBO STATE
...
ABSTRACT
Adapter link down
...
IDENTIFIER: 8D424E06
Date/Time: Mon Jun 24 09:56:59 2015
...
Failure Causes
ADAPTER FAILURE
Recommended Actions
SWITCHING TO LIMBO STATE
...
ABSTRACT
Adapter link down
...
Cause
Limbo Packets is the number of limbo packets received on the control channel. Limbo packets are sent by the primary Shared Ethernet Adapter (SEA) when it detects that its physical network is not operational, or when it cannot ping the specified remote host (to inform the backup SEA that it needs to become active). This is documented in the Shared Ethernet Adapter failover statistics.
The most common causes for a Shared Ethernet Adapter to be in limbo state include (but are not limited to) the following:
- The link status went down for the underlying physical adapter. This is the most common cause.
- The Shared Ethernet Adapter cannot ping the specified remote host--IP address to ping (netaddr)--to inform the backup Shared Ethernet Adapter that it needs to become active. If the Shared Ethernet Adapter netaddr attribute has an IP address to ping, ensure the IP is valid and pingable.
- The Shared Ethernet Adapter may be misconfigured. When using a dedicated control channel adapter and the Shared Ethernet Adapters are created using a control channel adapter with a different PVID, such misconfiguration can lead to a network storm. See "Diagnosing the problem" for more details.
- Note: If the Shared Ethernet Adapter in question is a "Simplified SEA failover" setup (no dedicated control channel adapter is being used), so this does NOT apply. To check, run: 'lsdev -dev SEA_ent# -attr netaddr'
- Problem on the network switch.
Environment
This applies to VIOS 2.2.x with SEA configured in failover (auto or sharing) mode.
To check SEA mode, run:
$ lsdev -dev ent# -attr > ha_mode = auto or sharing
The following example shows SEA failover adapter, ent8, configured in auto mode:
$ lsdev -dev ent8 -attr
attribute value description user_settable
...
ha_mode auto High Availability Mode True
...
Diagnosing The Problem
1. Is the SEA in question a new configuration in failover (auto or sharing) mode?
2. Was the SEA in question previously working but you changed the control channel adapter to a different ent device?
If answer is YES, ensure the control channel adapter specified during SEA creation (mkvdev) on both VIOS is using the same "Port VLAN ID" (PVID). The PVID must be a "unique" PVID that is NOT in use. To verify, run the following on both VIOs: When creating an SEA in failover (auto or sharing) mode without a properly configured control channel adapter, this can lead to a network storm. Such misconfiguration is known to cause loops in the network that lead to an ARP broadcast storm that is then detected by the network switch and as a result, the switch port gets disabled. If the VIOS is then rebooted with a "proper" configuration, it may not detect a link negotiation from the switch, and that causes the SEA state to be put in Limbo. In this case, you need to contact your network administrator, as they may need to reset the switch port.
Note: The lack of or misconfiguration of a control channel adapter would only be an "indirect" cause of Limbo state if the switch port gets shutdown due to network storm.
Note: The lack of or misconfiguration of a control channel adapter would only be an "indirect" cause of Limbo state if the switch port gets shutdown due to network storm.
If answer is NO, continue to #2.
$ entstat -all ent#|grep -i "Port VLAN ID" > where ent# is the SEA
Both VIOs should return the same PVID.
2. Was the SEA in question previously working but you changed the control channel adapter to a different ent device?
If answer is YES, ensure that the new control channel adapter (ent#) specified is configured with the same PVID used by the original control channel.
If answer is NO, see Resolving the Problem for other possible causes
Resolving The Problem
Very often the SEA switches to LIMBO state due to some type of hardware problem where the "Link Status" of the physical network port used by the SEA is DOWN instead of UP. If the criterias described above did not apply to you, check to see if the physical adapter used by the SEA in question logged any errors indicating that its link status went down.
Note: The SEA will not get out of LIMBO state until the Link Status is UP. If the link status is found to be DOWN, that must be corrected first.
1. Find out the physical adapter used by the SEA. Then check the VIOS error log for link errors.
In the following example, ent8 is the SEA, and ent4 is the physical adapter used by the SEA:
$ lsdev -type adapter|grep -i shared > Lists all SEAs
ent8 Available Shared Ethernet Adapter
ent8 Available Shared Ethernet Adapter
$ lsdev -dev ent8 -attr |grep real_adapter
...
real_adapter ent4 Physical adapter associated with the SEA True
...
$ errlog|grep ent4
The VIOS errlog is a key, as sometimes, it clearly reveals the SEA switched to limbo state due to the link status of its physical adapter going down.
In the following example, ent8 is an SEA configured in failover (auto) mode using physical adapter, ent1.
If you see link errors logged by the physical adapter used by the SEA in question, consider the following:
Jun 24 09:56:59 ent8 I VIOS_SEA_ADAP_FAIL
Jun 24 09:56:59 ent4 P LNCENT_HW_ERR
Jun 24 09:56:46 ent4 T LNCENT_TX_ERR
LABEL: VIOS_SEA_ADAP_FAIL
IDENTIFIER: 8D424E06
Date/Time: Mon Jun 24 09:56:59 2015
...
Failure Causes
ADAPTER FAILURE
Recommended Actions
SWITCHING TO LIMBO STATE
...
ABSTRACT
Adapter link down
...
Jun 24 09:56:59 ent4 P LNCENT_HW_ERR
Jun 24 09:56:46 ent4 T LNCENT_TX_ERR
LABEL: VIOS_SEA_ADAP_FAIL
IDENTIFIER: 8D424E06
Date/Time: Mon Jun 24 09:56:59 2015
...
Failure Causes
ADAPTER FAILURE
Recommended Actions
SWITCHING TO LIMBO STATE
...
ABSTRACT
Adapter link down
...
1. Check the environment for possible issues with the physical layer, such as bad GBIC/adapter/switch port, cable problems (check for a loose or defective cable or connection), etc.
2. If a switch or another system is directly attached to the ethernet adapter, verify it is powered up, configured, and functioning correctly. Consult with your network administrator to find out if there was any maintenance on the switch side around the time the link error was logged by the VIOS that might help justify the link going down. Examine the switch logs specifically for the switch port connected to physical network adapter port for which the link status went down to see if the switch port is actually enabled, or perhaps shut down.
If the switch port is enabled and not shut down, try putting the SEA in "standby" mode.
Then, bring the interface down, to change the SEA and its physical adapter into a Defined state and back to Available state by running the following commands:
To put SEA in standby mode:
$ chdev -dev <SEA_ent#> -attr ha_mode=standby
$ chdev -dev <SEA_ent#> -attr ha_mode=standby
To bring the SEA network interface down:
$ chdev -dev <SEA en#> -attr state=down > where en# is the SEA interface
$ chdev -dev <SEA en#> -attr state=detach
$ chdev -dev <SEA en#> -attr state=down > where en# is the SEA interface
$ chdev -dev <SEA en#> -attr state=detach
To change the SEA and its physical adapter into a Defined state:
$ rmdev -dev <SEA_ent#> -ucfg
$ rmdev -dev <SEA_real_adapter_ent#> -ucfg
$ rmdev -dev <SEA_ent#> -ucfg
$ rmdev -dev <SEA_real_adapter_ent#> -ucfg
To make them back into Available state:
$ cfgdev -dev <SEA_real_adapter_ent#>
$ cfgdev -dev <SEA_ent#>
$ cfgdev -dev <SEA_real_adapter_ent#>
$ cfgdev -dev <SEA_ent#>
On the other hand, if the switch port is enabled but the link status shows DOWN, consider the following:
- Try bringing the SEA network interface down, and put the SEA in Defined state as noted above. Then put an IP address on the physical adapter. Sometimes the link status may not change until you try to put an IP on the physical network adapter port.
- As a last resource, try rebooting the VIOS. Then, check if the Link Status shows UP, and if so, the SEA state show no longer be LIMBO.
The following errors are known to be logged on environments where the SEA in question was using a real adapter (ent0 in this example) with unsupported FW level, 00010000020025201905, causing the SEA (ent14) to go into limbo state:
Aug 25 12:50:41 ent14 I VIOS_SEA_ADAP_FAIL
Aug 25 12:49:47 ent0 T LNC2ENT_HW_TMP_ERR
Aug 25 12:47:59 ent0 T LNC2ENT_TX_ERR
Aug 25 12:47:47 ent0 T LNC2ENT_TX_ERR
Aug 25 12:49:47 ent0 T LNC2ENT_HW_TMP_ERR
Aug 25 12:47:59 ent0 T LNC2ENT_TX_ERR
Aug 25 12:47:47 ent0 T LNC2ENT_TX_ERR
To determine the real adapter type, run:
$ lsdev -type adapter|grep ent0
ent0 Available 0B-00 U78CA.001.CSS00TP-P1-C4-C1-T1 PCIe3 4-Port 10GbE SR Adapter (df1020e21410e304)
ent0 Available 0B-00 U78CA.001.CSS00TP-P1-C4-C1-T1 PCIe3 4-Port 10GbE SR Adapter (df1020e21410e304)
To determine the adapter FW level:
$ lsfware -dev ent0
Last, if none of the above apply to you and the SEA state shows LIMBO when the link status is UP, collect a snap from the VIO servers and contact your IBM SupportLine Representative for investigation.
Related Information
[{"Product":{"code":"SSPHKW","label":"PowerVM Virtual I\/O Server"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":null,"Platform":[{"code":"","label":"Other"}],"Version":"2.2.0;2.2.1.0;2.2.2.0;2.2.3.0","Edition":"Enterprise;Express;Standard","Line of Business":{"code":"LOB57","label":"Power"}},{"Product":{"code":"SSPHKW","label":"PowerVM Virtual I\/O Server"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB57","label":"Power"}}]
Was this topic helpful?
Document Information
Modified date:
19 February 2022
UID
isg3T1022635