How To
Summary
Every system admin must be aware of the basic networking troubleshooting. This can often save time by performing some initial troubleshooting before engaging the networking team. In this technote, we'll cover the basics of network troubleshooting on the IBM AIX command line.
Steps
TCP/IP model quick review
- Layer 5: Application Layer
- Layer 4: Transport Layer
- Layer 3: Network/Internet Layer
- Layer 2: Data Link Layer
- Layer 1: Physical Layer
Note: This technote assumes all PowerVM VIOS commands are run in root shell environment. After logging into padmin, $oem_setup_env needs to be run.
# entstat -d entX | grep -i link
# entstat -d ent1 | grep -i link
Physical Port Link Status: Down
Logical Port Link Status: Down
PCIe Link Speed: Unknown
Tip
# netstat -v | awk '/ETHERNET STATISTICS/ {device = $3; getline; if ($0 ~ /Physical Port Link Status/) {print device ": " $0; getline; if ($0 ~ /Link Status:/) {print device ": " $0}} else if ($0 ~ /Shared/) {print device ": " $0} else {while ($0 !~ /ETHERNET STATISTICS/ && getline) {if ($0 ~ /Physical Port Link Status/) {print device ": " $0; getline; if ($0 ~ /Link Status:/) {print device ": " $0}; break}}}}'
(ent2): Physical Port Link Status: Up
(ent2): Logical Port Link Status: Up
(ent1): Physical Port Link Status: Down
(ent1): Logical Port Link Status: Down
(ent4): Device Type: Shared Ethernet Adapter
(ent0): Physical Port Link Status: Up
(ent0): Logical Port Link Status: Up
Note
# errpt
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
F596EFAC 0524110222 T H ent1 Physical link down
# ifconfig -a
en1: flags=1e084863,814c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
inet 192.168.0.10 netmask 0xffff0000 broadcast 192.168.255.255
inet6 fe80::b0ad:34ff:fea3:3402/64
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
sit0: flags=8100041<UP,RUNNING,LINK0>
inet6 ::192.168.0.10/96
en0: flags=1e080822,18c0<BROADCAST,NOTRAILERS,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
inet6 ::1%1/128
tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
# ifconfig en0
en0: flags=1e080863,18c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
inet 192.168.1.20 netmask 0xffffff00 broadcast 192.168.1.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
# entstat -d ent0 | grep -i link
Physical Port Link Status: Up
Logical Port Link Status: Up
PCIe Link Speed: 8.0 GT/s
# arp -an
# arp -an
? (192.168.1.200) at 7a:21:16:25:8f:2 [ethernet] stored in bucket 7
? (192.168.1.31) at 98:be:94:58:cd:a4 [ethernet] stored in bucket 87
bucket: 0 contains: 0 entries
bucket: 1 contains: 0 entries
..
There are 2 entries in the arp table.
# netstat -nr
# netstat -nr
Routing tables
Destination Gateway Flags Refs Use If Exp Groups
Route Tree for Protocol Family 2 (Internet):
default 192.168.1.200 UG 1 291482 en1 - -
# arp -an
? (192.168.1.200) at (incomplete)
A VLAN functions as a logical subnet within a switch or across an entire physical network. In a physically separated network, devices are assigned to specific switch ports.
In contrast, when VLANs are used to separate a network, devices are logically divided using a VLAN TAG in the Ethernet frame. This tag is added to Layer 2 (L2) frames, marking the frame with a specific VLAN ID.
Switches that support VLANs can assign these tagged frames to specific VLANs, allowing the creation of logically distinct networks on a shared infrastructure. These VLANs do not need to be physically contiguous and can span wide-ranging networks and multiple switches.
The principles of VLANs are defined by the IEEE 802.1Q standard.
It's noticed that network disruptions often occur when VLANs are configured virtually on a POWER system for the first time due to the confusion and lack of arrangement between what is already on the network and what needs to be done on the POWER system. This document will go over several scenarios:
- AIX's network is managed by a VIOS (e.g. VEA is used for network connectivity)
- AIX is using dedicated ethernet adapters
- AIX is using vNICs
AIX's network is managed by a VIOS
Network virtualization on POWER systems depend mainly on the configuration of Shared Ethernet Adapters. Often, intertwined concepts regarding configuring a Shared Ethernet Adapter will cause confusion.
Before going ahead an exploring the case when AIX/VIOS has its IP configured on a Virtual Ethernet Adapter (VEA), it will be good to have an understanding of the difference between a VLAN ID (VID) and a Port VLAN ID (PVID).
The VLAN tag information is referred to as VLAN ID (VID). Ports on a switch are configured as being members of a VLAN designated by the VID for that port. The default VID for a port is referred to as the Port VID (PVID). The VID can be added to an Ethernet packet either by a VLAN-aware host, or by the switch in the case of VLAN-unaware hosts. Ports on an Ethernet switch must therefore be configured with information that indicates whether the host connected is VLAN-aware.
Simply put, when a VEA on an AIX/VIOS that is set up on a Virtual Network bridged by a Shared Ethernet Adapter (SEA) has a VID that is identical to the PVID of the SEA, the outgoing traffic of this system will be untagged.
In a scenario where the switch is configured in trunk mode, untagged traffic is usually dropped at the switch level. Further configuration of the system from a VLAN perspective is required.
Checking which VLAN the Virtual Ethernet Adapter is using is done through executing the command:
# entstat -d entX | grep -i vlan
The output will show the PVID of the VEA in addition to VID. Consider the following snippet:
# entstat -d ent1 | grep -i vlan
Invalid VLAN ID Packets: 0
Port VLAN ID: 1
VLAN Tag IDs: None
The traffic that goes out of the client LPAR will be tagged with a VLAN tag of 1. If the SEA bridging the network for this LPAR has a PVID of 1. This will indicate that the traffic is sent untagged. If the PVID of the SEA is different from 1 AND has VLAN 1 included as a VID of the SEA, the traffic will go out of the POWER frame with a tag of 1.
Checking which VIDs and PVIDs are used by each SEA is made through:
# entstat -d entX | egrep "ETHERNET STATISTICS|Port VLAN ID|VLAN Tag IDs" | awk '/ETHERNET STATISTICS/ { adapter=$0 } /Port VLAN ID/ { vlan_id=$0 } /VLAN Tag IDs/ { vlan_tags=$0; if (vlan_id && vlan_tags) { print adapter "\n" vlan_id "\n" vlan_tags; adapter=""; vlan_id=""; vlan_tags="" } }' | awk '!seen[$0]++'
Running the command over a Shared Ethernet Adapter ent4 will give an example output:
# entstat -d ent4 | egrep "ETHERNET STATISTICS|Port VLAN ID|VLAN Tag IDs" | awk '/ETHERNET STATISTICS/ { adapter=$0 } /Port VLAN ID/ { vlan_id=$0 } /VLAN Tag IDs/ { vlan_tags=$0; if (vlan_id && vlan_tags) { print adapter "\n" vlan_id "\n" vlan_tags; adapter=""; vlan_id=""; vlan_tags="" } }' | awk '!seen[$0]++'
ETHERNET STATISTICS (ent2) :
Port VLAN ID: 10
VLAN Tag IDs: 1
This output implies that the SEA ent4 has a virtual trunk adapter ent2. The VEA ent2 has a PVID of 10 and a VID of 1.
AIX is using dedicated ethernet adapters
In cases where AIX has a dedicated ethernet adapter and the switch port isn't configured in access mode, an additional pseudo-VLAN adapter is required.
The pseudo-VLAN adapter is configured by running:
# smitty vlan
AIX is using vNICs
A virtual Network Interface Controller (vNIC) is a type of virtual Ethernet adapter that is configured on client partitions of Power Systems servers. Each vNIC is backed by an SR-IOV logical port that is available in a Virtual I/O Server (VIOS) partition. This type of vNIC is also called dedicated vNIC, as the backing SR-IOV logical port serves the vNIC exclusively. The VLAN settings and MAC settings are applied to both the vNIC and the SR-IOV logical ports.
The vNIC VLAN setting on AIX can be checked through the command:
# entstat -d entX | grep -i vlan
Consider a vNIC adapter ent5 on an AIX LPAR, the command output will be:
# entstat -d ent5 | grep -i vlan
Port VLAN ID Status: Enabled
Port VLAN ID: 0010
Port VLAN Priority: 0
VLAN ACL Status: Disabled
In this example, the VLAN ID used for the tagging is VLAN 10.
For more info on vNIC functionality, refer to the link on vNIC Functionality Guide in Related Information section.
# ifconfig -a
en1: flags=1e084863,80480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>
inet 192.168.1.70 netmask 0xffff0000 broadcast 192.168.255.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
en5: flags=1e080862,18c0<BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
inet6 ::1%1/64
tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
en1: flags=1e084863,80480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>
^^
inet 192.168.1.70 netmask 0xffff0000 broadcast 192.168.255.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
|--- Notice the absence of the UP flag.
v
en5: flags=1e080862,18c0<BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
- The IP address is 192.168.1.70
- The Subnet Mask (netmask) is 0xffff0000 or 255.255.0.0
# netstat -nr
Routing tables
Destination Gateway Flags Refs Use If Exp Groups
Route Tree for Protocol Family 2 (Internet):
default 192.168.0.1 UG 1 313905 en1 - -
192.168.0.0 192.168.1.70 UHSb 0 0 en1 - - =>
192.168/16 192.168.1.70 U 4 24300 en1 - -
192.168.1.70 127.0.0.1 UGHS 0 11037 lo0 - -
192.168.255.255 192.168.1.70 UHSb 0 4 en1 - -
127/8 127.0.0.1 U 3 294858 lo0 - -
Route Tree for Protocol Family 24 (Internet v6):
::1%1 ::1%1 UH 0 69173 lo0 - -
# netstat -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp 0 0 *.* *.* CLOSED
tcp4 0 0 *.* *.* CLOSED
tcp4 0 0 *.13 *.* LISTEN
tcp 0 0 *.21 *.* LISTEN
tcp6 0 0 *.22 *.* LISTEN
tcp4 0 0 *.22 *.* LISTEN
tcp 0 0 *.23 *.* LISTEN
tcp4 0 0 *.25 *.* LISTEN
tcp4 0 0 *.37 *.* LISTEN
tcp 0 0 *.111 *.* LISTEN
tcp 0 0 *.512 *.* LISTEN
tcp 0 0 *.513 *.* LISTEN
tcp 0 0 *.514 *.* LISTEN
tcp4 0 52 192.168.1.70.22 192.168.1.35.48349 ESTABLISHED
tcp 0 0 *.657 *.* LISTEN
tcp4 0 0 *.1334 *.* LISTEN
tcp 0 0 *.3901 *.* LISTEN
tcp 0 0 *.32768 *.* LISTEN
tcp 0 0 *.32769 *.* LISTEN
udp4 0 0 *.* *.*
udp4 0 0 *.13 *.*
udp4 0 0 *.37 *.*
udp 0 0 *.111 *.*
udp 0 0 *.514 *.*
udp4 0 0 *.518 *.*
udp 0 0 *.657 *.*
udp 0 0 192.168.1.70.2279 *.*
udp 0 0 *.2279 *.*
udp 0 0 *.32768 *.*
udp 0 0 *.32772 *.*
udp 0 0 *.32777 *.*
udp 0 0 *.32850 *.*
udp 0 0 *.32864 *.*
# telnet 192.168.1.70 24
Trying...
telnet: connect: Connection refused
Additional Information
| SUPPORT |
|---|
|
If you require more assistance, use the following step-by-step instructions to contact IBM to open a case for software with an active and valid support contract. 1. Document (or collect screen captures of) all symptoms, errors, and messages related to your issue. 2. Capture any logs or data relevant to the situation. 3. Contact IBM to open a case: -For electronic support, see the IBM Support Community: 4. Provide a clear, concise description of the issue. - For more information, see: Working with IBM AIX Support: Describing the problem. 5. If the system is accessible, collect a system snap, and upload all of the details and data for your case. - For more information, see: Working with IBM AIX Support: Collecting snap data |
Related Information
Product Synonym
PowerVM; AIX
Was this topic helpful?
Document Information
Modified date:
10 June 2024
UID
ibm17154661