Why AIX VIO Clients can communicate with each other no matter if they are using the primary or the backup Virtual Ethernet Adapter but Linux VIO Clients can not?

Answer

VIO Clients running AIX with a etherchannel in NIB (Network Interface Backup) mode are set up using two VEAs (Virtual Ethernet Adapter) each going to their own virtual switch (vSwitch). VIO Clients running Linux are also set up in a similar way using the Linux Bond driver.  The AIX VIO Clients can communicate with each other no matter if they are using the primary or the backup VEA. But Linux VIO Clients can communicate with another VIO Client only if the two VIO Clients are both using the primary VEA or both using the backup VEA.

Why can't Linux VIO Client using the primary VEA communicate with another VIO Client using its backup VEA?

Case 1: When the primary VEA on both VIO Clients is active and connected to same virtual switch (vSwitch)

When the active VEA on both VIO Clients goes to the same virtual switch, the packet sent from one VIO Client will travel from the sending VEA to the virtual switch and then to the receiving VEA on another VIO Client. In this case, the packet is sent from ent1 of VIO Client 1 because ent1 is active. It is received by vSwitch 1 and then sent to ent1 on VIO Client 2 where it is received and process since ent1 is active.

Case 2: When the backup VEA on both VIO Clients is active and connected same virtual switch (vSwitch)

When the active VEA on both VIO Clients goes to the same virtual switch, the packet sent from one VIO Client will travel from the sending VEA to the virtual switch and then to the receiving VEA on another VIO Client. In this case, the packet is sent from ent2 of VIO Client 1 because ent2 is active. It is received by vSwitch 2 and then sent to ent2 on VIO Client 2 where it is received and process since ent2 is active.

Case 3: When the primary VEA is active on one VIO Client and backup VEA is active on another VIO Client

Figure A shows the expected packet flow in case AIX or Linux VIO Client. Packet is sent from ent1 on VIO Client 1 to vSwitch 1 because ent1 is active. Then, packet could sent to ent3 on the VIOS and out to the real network via ent5. Once in the real network, the packet could come back in to the VIOS via ent0, ent4, ent1 back to vSwitch 2. From vSwitch 2, the packet could be sent to ent2 on VIO Client 2 where it would be received since ent2 is active. However, this is not what happens. Actually, packets flow as shown in Figure B for Linux VIO Client and Figure C for AIX VIO Client.

As shown in Figure B, in case of a Linux VIO Client, when VIO Client 1 sends the packet out ent1 to vSwitch 1, the switch looks at the destination MAC. Since ent1 on VIO Client 2 has the same MAC, vSwitch 1 sends the packet to ent1 on VIO Client 2. But since ent1 is not active, the packet is ignored.

As shown in Figure C, in case of a AIX VIO Client, when VIO Client 1 sends the packet out ent1 to vSwitch 1, the switch looks at the destination MAC. Since ent1 on VIO Client 2 has the same MAC, vSwitch 1 sends the packet to ent1 on VIO Client 2. Packet goes up to etherchannel ent3 and then to network stack.