90
I was working with a customer recently on a Power Blade that was running the Integrated Virtualisation Manager (IVM). They’d installed a VIO partition onto the Blade and had hoped to install a couple of AIX LPARs on the system. However they didn’t get very far.
As soon as they attempted to NIM install the LPARs, they would get stuck at trying to ping the NIM master from the client. Basically, the Shared Ethernet Adapter (SEA) was not working properly and none of the LPARs could communicate with the external network. So they asked for some assistance.
The Blade server name was Server-8406-71Y-SN06BF99Z. The SEA was configured as ent7.
# lsdev -Cc adapter
ent0 Available Logical Host Ethernet Port (lp-hea)
ent1 Available Logical Host Ethernet Port (lp-hea)
ent2 Available Virtual I/O Ethernet Adapter (l-lan)
ent3 Available Virtual I/O Ethernet Adapter (l-lan)
ent4 Available Virtual I/O Ethernet Adapter (l-lan)
ent5 Available Virtual I/O Ethernet Adapter (l-lan)
ent6 Available EtherChannel / IEEE 802.3ad Link Aggregation
ent7 Available Shared Ethernet Adapter
ent9 Available Virtual I/O Ethernet Adapter (l-lan)
fcs0 Available 02-00 8Gb PCIe FC Blade Expansion Card (7710322577107601)
fcs1 Available 02-01 8Gb PCIe FC Blade Expansion Card (7710322577107601)
ibmvmc0 Available Virtual Management Channel
lhea0 Available Logical Host Ethernet Adapter (l-hea)
sissas0 Available 01-08 PCI-X266 Planar 3Gb SAS Adapter
usbhc0 Available 00-08 USB Host Controller (33103500)
usbhc1 Available 00-09 USB Host Controller (33103500)
usbhc2 Available 00-0a USB Enhanced Host Controller (3310e000)
vhost0 Available Virtual SCSI Server Adapter
vsa0 Available LPAR Virtual Serial Adapter
vts0 Available Virtual TTY Server Adapter
The SEA was configured with Port-VLAN ID (PVID) of 68 without any VLAN tags. This was the root cause of the problem.
# lsattr -El ent7
accounting enabled Enable per-client accounting of network statistics True
ctl_chan Control Channel adapter for SEA failover True
gvrp no Enable GARP VLAN Registration Protocol (GVRP) True
ha_mode disabled High Availability Mode True
jumbo_frames no Enable Gigabit Ethernet Jumbo Frames True
large_receive no Enable receive TCP segment aggregation True
largesend 0 Enable Hardware Transmit TCP Resegmentation True
netaddr 0 Address to ping True
pvid 68 PVID to use for the SEA device True
pvid_adapter ent9 Default virtual adapter to use for non-VLAN-tagged packets True
qos_mode disabled N/A True
real_adapter ent6 Physical adapter associated with the SEA True
thread 1 Thread mode enabled (1) or disabled (0) True
virt_adapters ent9 List of virtual adapters associated with the SEA (comma separated) True
$ entstat -all ent8 | grep -i vlan
VLAN Ids :
VLAN Extract: False
VLAN tagged filtering mode: Filter according to VLAN permit array
Max number of VLAN IDs per HEA port: 20
VLAN Extract: False
VLAN tagged filtering mode: Filter according to VLAN permit array
Max number of VLAN IDs per HEA port: 20
Invalid VLAN ID Packets: 5388
Port VLAN ID: 68
VLAN Tag IDs: None
On the network switch port, the native VLAN (PVID), was configured as 11, with VLAN tag 68 added as an allowed VLAN. If the client LPARs tried to access the network using a PVID of 68, instead of a VLAN TAG of 68, they would get stuck at the switch port i.e. the un-tagged packets for 10.1.68.X via PVID 11 would fail. The packets for 10.1.68.X needed to be tagged with VLAN id 68 in order for the switch to pass the traffic.
So the question was, how do we add VLAN tags in the IVM environment? If we’d been using a HMC, then this would be simple to fix. Just add the VLAN tags into the Virtual Ethernet Adapter used by the SEA and we’d be done.
We had to use the lshwres and chhwres commands to resolve this one. First we listed the virtual adapters known to the VIO server (IVM). At slot 12, we found our SEA adapter with port_vlan_id set to 68 and addl_vlan_ids set to none.
$ lshwres -r virtualio --rsubtype eth --level lpar
lpar_name=06-BF99Z,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62803
lpar_name=06-BF99Z,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62804
lpar_name=06-BF99Z,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62805
lpar_name=06-BF99Z,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62806
lpar_name=06-BF99Z,lpar_id=1,slot_num=12,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA6280C
lpar_name=aixlpar1,lpar_id=2,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=0,trunk_priority=0,is_required=0,mac_addr=F67D5345AD04
We needed to change port_vlan_id to 11 and addl_vlan_ids to 68. We also required the ieee_virtual_eth value set to 1.
First we removed the existing SEA adapter, as we would not be able to make changes to it while it was “active”. We then removed the adapter from slot 12 and then re-added it, again at slot 12, with port_vlan_id and addl_vlan_ids set to the desired values.
$ chhwres -m Server-8406-71Y-SN06BF99Z -p 06-BF99Z -r virtualio --rsubtype eth -s 12 -o r
$ chhwres -m Server-8406-71Y-SN06BF99Z -p 06-BF99Z -r virtualio --rsubtype eth -s 12 -o a -a "ieee_virtual_eth=1,port_vlan_id=11,addl_vlan_ids=68,is_trunk=1,trunk_priority=1" -d 5
$ lshwres -r virtualio --rsubtype eth --level lpar
lpar_name=06-BF99Z,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62803
lpar_name=06-BF99Z,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62804
lpar_name=06-BF99Z,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62805
lpar_name=06-BF99Z,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62806
lpar_name=06-BF99Z,lpar_id=1,slot_num=12,state=1,ieee_virtual_eth=1,port_vlan_id=11,addl_vlan_ids=68,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA6280C
lpar_name=aixlpar1,lpar_id=2,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=0,trunk_priority=0,is_required=0,mac_addr=F67D5345AD04
Using the mkvdev command we created the SEA again. Then using the entstat command we found that the PVID and VLAN tags had been configured correctly.
$ mkvdev -sea ent6 -vadapter ent2 -default ent2 -defaultid 11
ent7 Available
en7
et7
$ entstat -all ent7 | grep -i vlan
VLAN Ids :
VLAN Extract: False
VLAN tagged filtering mode: Filter according to VLAN permit array
Max number of VLAN IDs per HEA port: 20
VLAN Extract: False
VLAN tagged filtering mode: Filter according to VLAN permit array
Max number of VLAN IDs per HEA port: 20
Invalid VLAN ID Packets: 5388
Port VLAN ID: 11
VLAN Tag IDs: 68
Once this was done, the client LPARs were able to ping the NIM master. The customer happily started installing AIX onto each of the blades Partitions.
A related link: http://www.aixmind.com/?p=2133
