SUSE Linux Enterprise Server 11 (SLES11) XEN: Various Network issues in guest operating systems with Intel X5xx 10 Gb Ethernet Adapter Card

Troubleshooting

Problem

Symptoms occur in SUSE Linux Enterprise Server 11 (SLES11) Xen with an Intel X5xx 10 Gb Ethernet adapter card installed. When started in the Xen kernel, if users have bonded the two Network Interface Controller (NIC) ports of the Intel adapter card, thenattached that bond to a bridge, and then passed that bridge to a Windows guest, operations such as Secure Copy (SCP) within the guest will fail to complete and '/var/log/messages' will contain errors. The exact errors in '/var/log/messages' dependon the service pack level of SLES 11 and whether out-of-box or in-box drivers are in use. For SLES 11 SP2 with in-box drivers, there will be no error messages in '/var/log/messages', but various network operations will still fail. ForSLE S 11 SP3 wit h in-box drivers, '/var/log/messages' will contain warnings that state 'bond0: received packets cannot be forwarded while LRO is enabled'. For SLES 11 SP2 and SP3 with out-of-box drivers, '/var/log/messages' will contain the foll owing warnin g

Resolving The Problem

Source

RETAIN tip: H21552

Symptom

The exact errors in '/var/log/messages' depend on the service pack level of SLES11 and whether out-of-box or in-box drivers are in use.

For SLES11 SP2 with in-box drivers, there will be no error messages in '/var/log/messages', but various network operations will still fail.

For SLES11 SP3 with in-box drivers, '/var/log/messages' will contain warnings that state 'bond0: received packets cannot be forwarded while LRO is enabled'.

For SLES11 SP2 and SP3 with out-of-box drivers, '/var/log/messages' will contain the following warnings for every packet sent/received:

	[ 4503.744354] WARNING: at /usr/src/packages/BUILD/kernel-xen-3.0.76/linux-3.0/net/core/dev.c:1908 skb_gso_segment+0x21c/0x310() [ 4503.744360] Hardware name: -[7145AC1]- [ 4503.744365] tun: caps=(0x40004040, 0x0) len=1500 data_len=1460 ip_summed=1 [ 4503.744369] Modules linked in: tun bonding ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables nbd blktap blktap2 pciback usbbk xen_scsibk blkbk domctl blkback_pagemap netbk xenbus_be gntdev evtchn edd af_packet bridge stp llc fuse nls_iso8859_1 nls_cp437 vfat fat loop dm_mod ixgbe(FX) cdc_ether usbnet 8250_pnp i7core_edac be2net iTCO_wdt shpchp edac_core ipv6 ipv6_lib mii iTCO_vendor_support pci_hotplug i2c_i801 ioatdma joydev bnx2 tpm_tis pcspkr serio_raw button wmi sr_mod rtc_cmos tpm tpm_bios sg 8250 serial_core ext3 jbd mbcache usbhid hid ttm drm_kms_helper drm agpgart i2c_algo_bit sysimgblt sysfillrect i2c_core syscopyarea uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif processor thermal_sys hwmon scsi_dh_hp_sw scsi_dh_rdac scsi_dh_alua scsi_dh_emc scsi_dh xenblk cdrom xennet ata_generic ata_piix libata megaraid_sas scsi_mod [ 4503.744483] Supported: Yes, External [ 4503.744489] Pid: 0, comm: swapper Tainted: GF W X 3.0.76-0.9-xen #1 [ 4503.744493] Call Trace: [ 4503.744513] [<ffffffff80008b2e>] dump_trace+0x6e/0x1c0 [ 4503.744525] [<ffffffff8040652d>] dump_stack+0x69/0x6f [ 4503.744536] [<ffffffff800463ab>] warn_slowpath_common+0x7b/0xc0 [ 4503.744544] [<ffffffff800464a5>] warn_slowpath_fmt+0x45/0x50 [ 4503.744553] [<ffffffff803501ac>] skb_gso_segment+0x21c/0x310 [ 4503.744562] [<ffffffff8035042e>] dev_hard_start_xmit+0x18e/0x510 [ 4503.744571] [<ffffffff8036ec4e>] sch_direct_xmit+0x1ae/0x230 [ 4503.744579] [<ffffffff80352007>] dev_queue_xmit+0x267/0x430 [ 4503.744595] [<ffffffffa048ff2f>] br_dev_queue_push_xmit+0x5f/0x80 [bridge] [ 4503.744626] [<ffffffffa04976f0>] br_nf_post_routing+0x240/0x300 [bridge] [ 4503.744656] [<ffffffff8037bc94>] nf_iterate+0x84/0xb0 [ 4503.744664] [<ffffffff8037be19>] nf_hook_slow+0x79/0x120 [ 4503.744677] [<ffffffffa04900bd>] br_forward_finish+0x3d/0x60 [bridge] [ 4503.744695] [<ffffffffa0496cb0>] br_nf_forward_finish+0x140/0x150 [bridge] [ 4503.744730] [<ffffffffa0497318>] br_nf_forward_ip+0x208/0x3a0 [bridge] [ 4503.744775] [<ffffffff8037bc94>] nf_iterate+0x84/0xb0 [ 4503.744782] [<ffffffff8037be19>] nf_hook_slow+0x79/0x120 [ 4503.744795] [<ffffffffa048fe35>] __br_forward+0x75/0xc0 [bridge] [ 4503.744812] [<ffffffffa0490f92>] br_handle_frame_finish+0x192/0x2e0 [bridge] [ 4503.744832] [<ffffffffa0496f88>] br_nf_pre_routing_finish+0x188/0x310 [bridge] [ 4503.744867] [<ffffffffa0496ac0>] br_nf_pre_routing+0x360/0x410 [bridge] [ 4503.744896] [<ffffffff8037bc94>] nf_iterate+0x84/0xb0 [ 4503.744903] [<ffffffff8037be19>] nf_hook_slow+0x79/0x120 [ 4503.744916] [<ffffffffa0490d7f>] br_handle_frame+0x1cf/0x250 [bridge] [ 4503.744931] [<ffffffff8034fbdf>] __netif_receive_skb+0x2cf/0x680 [ 4503.744939] [<ffffffff80353570>] netif_receive_skb+0x90/0xa0 [ 4503.744946] [<ffffffff80353688>] napi_skb_finish+0x38/0x50 [ 4503.744963] [<ffffffffa042fa4f>] ixgbe_clean_rx_irq+0x2cf/0x480 [ixgbe] [ 4503.744986] [<ffffffffa04303fe>] ixgbe_poll+0x8e/0x170 [ixgbe] [ 4503.745003] [<ffffffff80353f7e>] net_rx_action+0x13e/0x2a0 [ 4503.745012] [<ffffffff8004d5ef>] __do_softirq+0xff/0x240 [ 4503.745024] [<ffffffff8041286c>] call_softirq+0x1c/0x30 [ 4503.745033] [<ffffffff80008625>] do_softirq+0x95/0xd0 [ 4503.745040] [<ffffffff8004d3d5>] irq_exit+0xc5/0xe0 [ 4503.745049] [<ffffffff802ea25d>] evtchn_do_upcall+0x2cd/0x310 [ 4503.745059] [<ffffffff804123be>] do_hypervisor_callback+0x1e/0x30 [ 4503.745069] [<ffffffff800033aa>] HYPERVISOR_sched_op_new+0xa/0x20 [ 4503.745086] [<ffffffff8000ddf7>] xen_idle+0xb7/0x180 [ 4503.745094] [<ffffffff80006677>] cpu_idle+0x67/0xb0 [ 4503.745106] [<ffffffff80748baf>] start_kernel+0x350/0x41f [ 4503.745113] ---[ end trace e45850273d114454 ]---

Affected configurations

The system is configured with at least one of the following:

SUSE Linux Enterprise Server 11 (SLES11), any service pack

The system is configured with one or more of the following IBM Option part numbers:

Intel X520 Dual-Port 10 Gb Ethernet SFP+ Embedded Adapter for IBM System x, Option part number 49Y7980, any replacement part number
Intel X520-DA2 Dual-Port 10 Gb Ethernet SFP+ Adapter for IBM System x, Option part number 49Y7960, any replacement part number
Intel X540-T2 Dual-Port 10 Gb Base-T Ethernet Adapter, Option part number 49Y7970, any replacement part number

This tip is not system specific.

The ixgbe device driver for the Intel X520 10 Gb Ethernet Adapter is affected.

Note: This does not imply that the network operating system will work under all combinations of hardware and software.

Please see the compatibility page for more information:

http://www.ibm.com/systems/info/x86servers/serverproven/compat/us/

Solution

Follow LTC Bugzilla #95667 for more information.

Workaround

While started in the Xen kernel, add the following to /etc/sysconfig/network/ifcfg-eth*:

ETHTOOL_OPTIONS='-K iface lro off'

where ifcfg-eth* are all of the files that correspond to interfaces in bond0 (for example, eth0 and eth1).

After editing the files, run 'service network restart' in the terminal. This will interrupt network operations momentarily.

The warnings in '/var/log/messages' will immediately disappear and normal network operations will resume within the Windows guest.

Additional information

Large Receive Offload (LRO) is disabled automatically for ethernet devices that are attached to a bridge (http://lxr.linux.no/#linux+v3.0.76/net/bridge/br_if.c#L380).

However, because bond0 provides an extra layer of abstraction between the actual NIC and the bridge, the bridging driver turns off LRO on bond0 but does not turn off LRO on eth0 and eth1 (the Intel NIC ports). Because LRO stays on, the Windows guest is unable to receive packets from the remote machine.

The workaround simply disables LRO as is intended for bridged interfaces. Eventually, a fix will automatically disable LRO on the base NIC interfaces regardless of whether they are enslaved in a bond.

Document Location

Worldwide

Operating System

System x Hardware Options:SUSE Linux Enterprise Server 11

System x Hardware Options:SUSE Linux Enterprise Server 11 x86-64

[{"Type":"HW","Business Unit":{"code":"BU016","label":"Multiple Vendor Support"},"Product":{"code":"QU01FYW","label":"System x Hardware Options->Ethernet->10 Gb->49Y7970"},"Platform":[{"code":"PF047","label":"SurePOS"}],"Line of Business":{"code":"","label":""}},{"Type":"HW","Business Unit":{"code":"BU016","label":"Multiple Vendor Support"},"Product":{"code":"QUOEAOM","label":"System x Hardware Options->Ethernet->10 Gb->49Y7960"},"Platform":[{"code":"PF047","label":"SurePOS"}],"Line of Business":{"code":"","label":""}},{"Type":"HW","Business Unit":{"code":"BU016","label":"Multiple Vendor Support"},"Product":{"code":"QUOEB07","label":"System x Hardware Options->Ethernet->10 Gb->49Y7980"},"Platform":[{"code":"PF047","label":"SurePOS"}],"Line of Business":{"code":"","label":""}}]

Tips