Question & Answer
Question
How can I configure traditional largesend for SAP HANA on SLES with VIOS?
Answer
TARGET AUDIENCE:
Administrators for VIOS RHEL and SLES
OBJECTIVE :
Document basic setup and verification of RHEL7.3, SLES 11 and 12 Linux Largesend/Largereceive in a VIOS environment
SCOPE :
Basic setup of VIOS/SLES to utilize largesend/largereceive feature.
Specific adapter, network or system tuning is beyond the scope of this document.
PROCEDURES:
-------------------------------------------------------------------------------------------------------------
A. VIOs largesend status
-------------------------------------------------------------------------------------------------------------
1) Check The "largesend" Status Of Your VIOs
First log on to console of the VIO Server over the hardware-management-console (HMC) as user padmin.
Check if jumbo_frames and largesend is already enabled:
$ lstcpip
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en6 1500 link#2 0.a.f7.73.45.80 724070 0 262218 0 0
en6 1500 10.17.192 ish330v1.wdf.sap 7924070 0 262218 0 0
lo0 16896 link#1 2589799 0 2589799 0 0
lo0 16896 127 localhost 2589799 0 2589799 0 0
lo0 16896 localhost 2589799 0 2589799 0 0
$
In our example the value of 1500 for mtu (Maximum transfer unit) for the device en6 shows, that jumbo_frames are not enabled. In case it would be enabled you should see a value of 9000. We assume, if someone has set up Jumbo_frames he will also have set up largesend and large_receive correctly.
If you will only see the "lo0" lines in the output of the "lstcpip" command, you do not have any IP Address set on your SEA. In this case you can check the largesend option setting with the following two commands (#2 and #3)
2) Find out your SEA:
$ lsdev -type adapter | grep -i Shared
ent6 Available Shared Ethernet Adapter
$
3) Check largesend, large_receive and jumbo_frames:
<$ lsdev -dev ent6 -attr largesend,large_receive,jumbo_frames
value
1
no
no
$
This output would indicate, that "largesend" is set but not "large_receive" and "jumbo_frames".
-------------------------------------------------------------------------------------------------------------
B. VIOS Requirements
-------------------------------------------------------------------------------------------------------------
1) Verify VIOS level
If not enabled, first check the OS-version of your VIO to verify that it is possible to enable jumbo_frames and largesend:
Recommended:
IOSLEVEL 2.2.4.10
(contains IV72825 Large receive packets are dropped by Linux Client LPAR)
From padmin shell execute:
$ ioslevel
2.2.4.22
$
*If the result is lower than 2.2.4.10, verify if IV72825 is installed:
$ print "instfix -ik IV72825" | oem_setup_env
All filesets for IV72825 were found.
$
*If your VIOS level is below 2.2.3.52 or equal 2.2.3.52 and the fix is not installed stop your investigations here. Do not change any settings and contact the responsible admin to upgrade the VIOS level first.
2) Verify the physical adapter
Verify the physical adapter used by SEA supports the largereceive capability.
(10Gb adapters have this capability. However, some 1Gb adapters do not.)
3) Change the Setup Of the VIOs
i. List Adapters/Devices of your VIO
First find out which Ethernet devices are located on your VIO as padmin:
$ lsdev | grep ent
ent0 Available PCIe2 4-Port Adapter (10GbE SFP+) (e41xxa123xxx123x)
ent1 Available PCIe2 4-Port Adapter (10GbE SFP+) (e41xxa123xxx123x)
ent2 Available PCIe2 4-Port Adapter (1GbE RJ45) (e41xxa123xxx223x)
ent3 Available PCIe2 4-Port Adapter (1GbE RJ45) (e41xxa123xxx223x)
ent4 Available Virtual I/O Ethernet Adapter (l-lan)
ent5 Available Virtual I/O Ethernet Adapter (l-lan)
ent6 Available Shared Ethernet Adapter
$
ii. Now take the Shared Ethernet Adapter (SEA) and detect to which "physical" Ethernet adapter (NIC) it is connected with. This might be single adapter, adapters combined over Ether-channel or a Link Aggregation:
$ lsdev -dev ent6 -attr | grep adapter
adapter_reset no Reset real adapter on HA takeover True
ctl_chan ent5 Control Channel adapter for SEA failover True
pvid_adapter ent4 Default virtual adapter to use for non-VLAN-tagged packets True
real_adapter ent0 Physical adapter associated with the SEA True
virt_adapters ent4 List of virtual adapters associated with the SEA (comma s...)True
$
iii. Unconfigure the SEA, so you can perform the changes (this will set the SEA from "Available" to "Defined" and it will trigger a SEA failover if ha_mode=auto is set).
**NOTE: You must be sure that you are logged on the console. (!)
iv. Unconfigure the Devices you want to work on:
If the IP Address is bound directly to the SEA adapter, unconfigure the IP Address from the adapter. Otherwise skip this task and continue with unconfiguring the SEA:
$ rmdev -dev en6 -ucfg
en6 Defined
$
v. Unconfigure the SEA:
$ rmdev -dev ent6 -ucfg
ent6 Defined
$
vi. Unconfigure the Physical Adapter:
$ rmdev -dev ent0 -ucfg
ent0 Defined
$
vii. Change The Devices:
- Change the SEA:
$ chdev -dev ent6 -attr jumbo_frames=yes largesend=1 large_receive=yes
ent6 changed
- Change the Physical Adapter:
$ chdev -dev ent0 -attr large_receive=yes large_send=yes jumbo_size=9014jumbo_frames=yes
ent0 changed
$
viii. Optional:
As further optimization you may want to re size buffers of the trunk adapter VEA (the "virt_adapter" from "lsdev -dev ent6 -attr", ent4 in our example ) to have a better fitting with jumbo_frames.
- Unconfigure and Change the VEA:
$ rmdev -dev ent4 -ucfg
ent4 Defined
$
$ chdev -dev ent4 -attr min_buf_tiny=2048 max_buf_tiny=4096 min_buf_small=2048 max_buf_small=4096 min_buf_medium=512 max_buf_medium=1024 min_buf_large=96 max_buf_large=256 min_buf_huge=96 max_buf_huge=128
ent4 changed
$
4) Rescan the configuration to make to devices available again without reboot and restore the routes for the Ethernet interface:
$ cfgdev
$ chdev -dev en6 -attr state=up -restoreroute
$
5) Verify Your Changes On The VIO
i. Check success of your changes with:
$ lstcpip
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en6 9000 link#2 0.2.c9.2c.f9.7e 606 0 1 0 0
en6 9000 10.17.192 ish316v2.wdf.sap 606 0 1 0 0
lo0 16896 link#1 2178938 0 2178940 0 0
lo0 16896 127 localhost 2178938 0 2178940 0 0
lo0 16896 ::1%1 2178938 0 2178940 0 0
$
ii. Check the Physical Adapter for its values:
$ lsdev -dev ent0 -attr jumbo_frames,jumbo_size,large_receive,large_send
Value
yes
9014
yes
yes
$
iii. Check the SEA for its values:
$ lsdev -dev ent6 -attr jumbo_frames,large_receive,largesend
Value
yes
yes
1
$
6) Redo the changes on the second VIO
-------------------------------------------------------------------------------------------------------------
C. SLES and RHEL Requirements
-------------------------------------------------------------------------------------------------------------
1) SLES11 SP4, SLES12 SP1 or RHEL7.3 with latest maintenance updates.
Linux sunbeam 3.0.101-71-ppc64 #1 SMP Thu Mar 3 12:56:15 UTC 2016 (7bdad2e) ppc64 ppc64 ppc64 GNU/Linux
(This applies to ALL SLES LPARs in the system.)
i. First check the version of your virtual Ethernet driver:
# modinfo ibmveth
filename: /lib/modules/3.0.101-71-ppc64/kernel/drivers/net/ibmveth.ko
version: 1.05 <--- You will need 1.05 or higher
license: GPL
description: IBM Power Virtual Ethernet Driver
......
......
......
parm: rx_flush:Flush receive buffers before use (uint)
parm: ---> old_large_send :Use old large send method on P8 firmware that supports the new method (bool)
#
2) IMPORTANT!!!
ibmveth old_large_send parameter MUST be enabled for future compatibility on LPARs that are or will be running on P8. LPARs confined purely to P7 platforms can ignore old_large_send.
i. Check "old_large_send". If old_large_send is disabled, a reboot will be needed before changes take effect.
# cat /sys/module/ibmveth/parameters/old_large_send
Y (Y: enabled N: disabled)
#
NOTE: old_large_send can also be enabled by unloading/reloading the ibmveth module, but that requires console access since all ibmveth adapters will be shutdown. TSO will also revert to default OFF state.
ii. To enable ibmveth old_large_send on reboot, create the following file in the /etc/modprobe.d directory:
# echo "options ibmveth old_large_send=1" > /etc/modprobe.d/50-ibmveth.conf
#
iii. Enable TSO and MTU 9000 ( Jumbo frames ) for the target ibmveth adapter(s)
# ethtool -K eth0 tso on <--- ## The device name may be different on your system
#
# ip link set dev eth0 mtu 9000 <--- ## The device name may be different on your system
#
iv. Make both changes persistent across reboots:
Change to network configuration directory and edit target adapter config file: ifcfg-eth# (on most systems: ifcfg-eth0)
- For SLES11 and SLES12 :
# cd /etc/sysconfig/network
# vi <config file>
BROADCAST=''
ETHTOOL_OPTIONS=''
BOOTPROTO='static'
IPADDR='xx.xx.xx.xx/xx'
MTU=''
NAME='Virtual Ethernet card 0'
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
ETHTOOL_OPTIONS='-K iface tso on ' <---iface internally replaced with devicename
MTU='9000' <---
- For RHEL:
# cd /etc/sysconfig/network-scripts
# vi <config file>
BROADCAST=''
ETHTOOL_OPTIONS=''
BOOTPROTO='static'
IPADDR='xx.xx.xx.xx/xx'
MTU=''
NAME='Virtual Ethernet card 0'
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
ETHTOOL_OPTS='-K eth0 tso on ' <---devicename may differ in your config file
MTU='9000' <---
NOTE: Using MTU 9000 is mandatory to get above 9Gb/s throughput as verified by the SAP HWCCT (Hardware Configuration Check Tool). The infrastructure prerequisite for using this setting is that all components from the sender to the receiver can handle MTU 9000 settings. However, if the infrastructure does not allow for this the business SLAs can likely be achieved with a lower bandwidth. In this case apply all settings described in this document except for the MTU setting.
**MTU 1500 hosts or networks may become unreachable after setting the MTU to 9000!
3) Edit /etc/sysctl.conf to set/add larger tcp send/recv spaces in the Kernel
# tuning for largesend
net.core.rmem_max = 56623104
net.core.wmem_max = 56623104
net.ipv4.tcp_rmem = 65536 262088 56623104
net.ipv4.tcp_wmem = 65536 262088 56623104
net.ipv4.tcp_mem = 56623104 56623104 56623104
4) Activate Your Changes for the Kernel
Push out changes:
# sysctl -p
#
5) Activate Your Changes for the Ethernet driver
## if OS == SLES12 || if OS == RHEL
# service network restart
## elif OS == SLES11
# /etc/init.d/network restart
.....
.....
.....
#
NOTE: If old_large_send was changed then a reboot will be needed to complete the change.
-------------------------------------------------------------------------------------------------------------
D. Verification and Test
-------------------------------------------------------------------------------------------------------------
NOTE: This does not replace the mandatory usage of the SAP Hardware Configuration Check Tool (HWCCT).
1) For the test you need an external system which is connected over 10 Gb, has largesend/largereceive enabled and jumbo frames set. We name it EXT_UNX here.
i. Check if jumbo size packages are configured:
# ip addr show eth0 <--- ## The device name may be different on your system
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether fa:ab:d0:6a:7c:20 brd ff:ff:ff:ff:ff:ff
inet 10.17.102.22/23 brd 10.17.102.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::f8ab:d0ff:fe6a:7c20/64 scope link
valid_lft forever preferred_lft forever
#
ii. Simple test to check if jumbo sized packages are transmitted:
# ping -c 10 -s 8900 <ipaddr_of_EXT_UNX_system>
PING <ipaddr_of_EXT_UNX_system> (12.21.123.21) 8900(8928) bytes of data.
8908 bytes from 12.21.123.21: icmp_seq=1 ttl=64 time=0.235 ms
8908 bytes from 12.21.123.21: icmp_seq=2 ttl=64 time=0.213 ms
.......
.......
.......
--- 12.21.123.21 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 8997ms
rtt min/avg/max/mdev = 0.213/0.241/0.256/0.019 ms
#
-------------------------------------------------------------------------------------------------------------
For further tests a streaming tcp load is needed to test largesend/largereceive such as ftp or iperf.
The examples below use iperf which is available most Unix and Linux systems.
NOTE: iperf -c (client mode) transmits packets.
iperf -s (server mode) receives packets.
-------------------------------------------------------------------------------------------------------------
2) Large_receive test EXT_UNX -> VIOS -> SLES or RHEL
This checks packets received by VIOS are aggregated before transfer to SLES LPAR.
i. Start iperf SERVER on the SLES or RHEL lpar using the default port 5001
# iperf -s
ii. Reset SEA adapter statistics on VIOS
$ entstat -reset ent6 ## The name of the device may be different on your system
....
....
....
$
iii. On the EXT_UNX system, start an iperf CLIENT with four parallel connections
# iperf -w 400m -c <ip_addr_SLES_RHEL_lpar> -P4
.....
.....
.....
.....
#
iv. Check VIOS SEA for aggregated receive packets
$ entstat -all ent6 | grep -i "Aggregat" <--- ## The device name may be different on your system
Receive TCP Segment Aggregation: Enabled
Receive TCP Segment Aggregation Large Packets Created: 283270 <---
Receive TCP Packets Aggregated into Large Packets: 583692 <---
Receive TCP Payload Bytes Aggregated into Large Packets: 490290060 <---
Receive TCP Segment Aggregation Average Packets Aggregated: 2 <---
Receive TCP Segment Aggregation Maximum Packets Aggregated: 16 <---
$
3) Large_send test SLES or RHEL -> EXT_UNX
This checks that largesend packets from SLES or RHEL are being resegmented for transmission by VIOS SEA/Real adapter.
i. Reset SEA adapter statistics on VIOS:
$ entstat -reset ent6 <--- ## The device name may be different on your system
....
....
....
$
ii. Start iperf server on EXT_UNX system on default port 5001
# iperf -s
iii. On SLES or RHEL system start iperf client with four parallel connections:
# iperf -w 400m -c <ipaddr_of_EXT_UNX_system> -P4
.....
.....
.....
.....
#
iv. Check VIOS SEA for TCP segmentation offloaded packets:
$ entstat -all ent6 | grep -i "Segmentation Offload" <--- ## The device name may be different on your system
Transmit TCP Segmentation Offload: Enabled
Transmit TCP Segmentation Offload Packets Transmitted: 202656 <---
Transmit TCP Segmentation Offload Maximum Packet Size: 62702 <---
$
v. Check SLES or RHEL lpar tx_large_packets count:
# ethtool -S eth0 | grep -i "large" <--- ## The device name may be different on your system
tx_large_packets: 316534 <---
rx_large_packets: 0 <--- Note: This counter is NOT used in traditinal largesend/receive
fw_enabled_large_send: 0
------------------------------------------------------------------------------------------------------------- REFERENCES:
N/A
CATEGORY:
WWNETK
SUPPORT:
If additional assistance is required after completing all of the instructions provided in this document, please follow the step-by-step instructions below to contact IBM to open a service request (PMR) for software under warranty or with an active and valid support contract. The technical support specialist assigned to your support call will confirm that you have completed these steps.
a. Document and/or take screen shots of all symptoms, errors, and/or messages that might have occurred
b. Capture any logs or data relevant to the situation
c. Contact IBM to open a support call (PMR):
- For electronic support, please visit the web page:
- For telephone support, please visit the web page:
- Please visit the IBM Support Portal web page for additional resources:
d. Provide a good description of your issue and reference this technote
e. Upload all of the details and data to your support call (PMR):
Please visit this web page for instructions: https://www.secure.ecurep.ibm.com/app/upload
FEEDBACK:
Quality documentation is important to IBM and its customers. If you have feedback specific to this article, please send an detailed message to the email address:
- This email address is monitored for feedback purposes only.- No support for any IBM products or services will be provided through this email.
- To receive support, please follow the step-by-step instructions in the above "SUPPORT" section.
Was this topic helpful?
Document Information
Modified date:
19 February 2022
UID
isg3T1024094