Contents


Enabling largesend and jumbo_frames in IBM AIX to reduce processor usage

Using TCP segmentation offload and MTU greater than default value of 1500

Comments

IBM® AIX® allows you to transmit large packets and frames through a network. To send a large data chunk over network, TCP breaks it down into multiple segments. This requires multiple calls down the stack and results in higher processor utilization on the host processor. You can address the issue using the TCP large send offload (LSO) option, which allows the AIX TCP layer to build a TCP message that is up to 64 KB long.

For example, without the TCP large send offload option, to send 64 KB of data it takes 44 calls down the stack by using 1500 byte Ethernet frames. With the TCP large send offload option enabled, the TCP option can send up to 64 KB of data to the network interface card (NIC) in a single transmit-receive call. In a real-time scenario, the required number of processor cycles is controlled by the application and depends on the speed of the physical network. With faster networks, using LSO reduces the host processor utilization and increases throughput.

A jumbo frame is an Ethernet frame with a payload greater than the standard maximum transmission unit (MTU) of 1,500 bytes and can be as large as 9,000 bytes. It has the potential to reduce processor usage.

Using TCP large send offload and jumbo_frames in AIX is independent of each other. They can be used together or in isolation. In this article, we discuss the end-to-end enablement of both these attributes. We also discuss about the large_receive attribute and how to enable it. At the end, we compare the performance statistics with and without LSO enabled.

How to enable TCP large send offload in AIX and VIOS

With TCP LSO enabled, a TCP message can have data up to 64 KB. Figure 1 shows the format of a TCP packet with LSO enabled.

Figure 1. TCP packet with LSO enabled

In this section, we will discuss how to enable the TCP large send offload option in AIX. An AIX client LPAR can either have a virtual Ethernet adapter which is mapped to Virtual I/O Server (VIOS) shared Ethernet adapter (SEA) or a direct network adapter port. We will discuss both these configurations in this section.

  1. For a virtual Ethernet adapter to make TCP large send offload work between two logical partitions (LPARs) on two separate servers, the communication has to occur through SEA. Therefore, the largesend attribute has to be enabled at the AIX client and VIOS SEA.
    1. On the sender LPAR client (assuming ent0 is the virtual Ethernet adapter), perform the following steps:
      1. Turn on the largesend attribute using the following command:
        				   #ifconfig en0 largesend

        Alternatively, you can also run the following command to get the largesend attribute configured at the ODM level. All ODM configurations will sustain system reboot.

        				   #chdev -l en0 -a mtu_bypass=on
      2. Turn on chksum_offload using the following command:
        				   #chdev -l ent0 -a chksum_offload=yes
    2. On VIOS (assume ent6 is an SEA, ent0 is a physical adapter, and ent5 is a virtual adapter), perform the following steps:
      1. Turn on the largesend SEA attribute using the following command:
        				   #chdev -l ent6 -a largesend=1
      2. Generally, all physical adapters under SEA have largesend capability turned on by default. Verify this by running the following command:
        				   # lsattr -El ent0 -a large_send

        Check whether the following output is displayed.

        				   large_send yes Enable hardware transmit TCP resegmentation True

        If the required output is not displayed, then turn it on manually by running the following command:

        				   #chdev -l ent0 -a large_send=yes
      3. Turn on chksum_offload on the physical adapter ent0 using the following command:
        				   # lsattr -El ent0 -a chksum_offload

        Check whether the following output is displayed.

        				   chksum_offload yes Enable Checksum Offload for IPv4 packets True

        If the required output is not displayed, then turn it on manually by running the following command:

        				   #chdev -l ent0 -a chksum_offload=yes
  2. To enable the TCP large send offload option when the sender LPAR has either a physical network adapter port connected or a virtual NIC configured on it, you need to perform the following steps:
    1. Turn on the largesend attribute for the physical adapter ent0 using the following command:
      				   #chdev -l ent0 -a large_send=yes
    2. Turn on chksum_offload for the physical adapter ent0 using the following command:
      				   # lsattr -El ent0 -a chksum_offload

      Check whether the following output is displayed.

      				   chksum_offload yes Enable Checksum Offload for IPv4 packets True

      If the required output is not displayed, then turn it on manually by running the following command:

      				   #chdev -l ent0 -a chksum_offload=yes
    3. Turn on the largesend attribute for the en0 interface.
      				   #ifconfig en0 largesend

      Alternatively, you can also run the following command to get it configured at the Object Data Manager(ODM) level. All ODM configurations will sustain system reboot.

      				   #chdev -l en0 -a mtu_bypass=on

How to enable jumbo_frames in AIX

With the jumbo_frames attribute enabled in AIX, an Ethernet payload can be as large as 9000 bytes. Figure 2 shows the format of an Ethernet jumbo frame.

Figure 2. Ethernet jumbo frame

If the jumbo_frames attribute needs to be used in a public network, the underlying network (including routers and switches) also need to be configured to support jumbo frames.

This section describes how to enable the jumbo_frames attribute in AIX. We will discuss both the configurations where a client AIX LPAR is having a virtual Ethernet adapter mapped to SEA or is having a direct network adapter port connected to it.

  1. Perform the following steps to enable the jumbo_frames attribute when an LPAR is connected to a physical network adapter port.
    1. If any IP is configured on the network interface, bring down the interface first by running the following commands:
      				#chdev -l en0 -a state=down
      				#chdev -l en0 -a state=detach
    2. Enable the jumbo_frames attribute by running the following commands:
      				#chdev -l ent0 -a jumbo_frames=yes
      				#chdev -l en0 -a mtu=9000
    3. Bring up the interface again by running the following command:
      				#chdev -l en0 -a state=up
    4. Refresh the routes on inet0 by running the following command:
      				#mkdev -l inet0
    5. Verify all configured attributes using the following commands:
      #lsattr -El ent0 | grep jumbo
      use_jumbo_frame yes         Enable Gigabit Jumbo Frames     True
      	
      # lsattr -El en0 | grep mtu
      mtu           9000          Maximum IP Packet Size for This Device        True
    6. Verify if the interface is actually running at MTU 9000 using the following command:
      # netstat -in 
      
      Name     Mtu      Network     Address             Ipkts     Ierrs    Opkts    Oerrs   Coll
      en0      9000     link#2      86.21.e4.55.cc.6    56549783    0      443579     0       0
      en0      9000     9.47.64     9.47.66.137         56549783    0      443579     0       0
      lo0      16896    link#1                          3438        0       3438      0       0
      lo0      16896    127         127.0.0.1           3438        0       3438      0       0
      lo0      16896    ::1%1                           3438        0       3438      0       0
  2. To enable the jumbo_frames attribute when the LPAR has a virtual Ethernet adapter mapped to SEA, the attribute needs to be enabled both at VIOS and the AIX client. You need to perform the following steps to enable it at VIOS (assuming that ent7 is an SEA, ent0 is a physical adapter, ent4 is a trunk adapter, and ent8 is a non-trunk adapter where VIOS IP is configured):
    1. If any IP is configured on en8, bring down the interface first by running the following commands:
      				#chdev -l en8 -a state=down
      				#chdev -l en8 -a state=detach
    2. Run the following command to enable the jumbo_frames attribute for the SEA physical adapter.
      				#chdev -l ent0 -a jumbo_frames=yes
    3. Run the following command to enable jumbo_frames for SEA:
      				#chdev -l ent7 -a jumbo_frames=yes
    4. Perform the following steps at VIOS to use the jumbo_frames attribute for it's own non-trunk adapter:
      1. Set the value of MTU to 9000.
        				#chdev -l en8 -a mtu=9000
      2. Bring up the interface again by running the following command:
        				#chdev -l en8 -a state=up
      3. Refresh routes on inet0 using the following command:
        				#mkdev -l inet0
      4. Verify if the interface is actually running at MTU 9000 using the following command:
        				# netstat -in

        In the output, Mtu value should be displayed as 9000.

  3. To enable jumbo_frames attribute for Virtual Ethernet adapter at AIX LPAR client, execute following steps.
    1. Enable the jumbo_frames attribute at VIOS SEA first following above step 2.
    2. If any IP is configured, bring down the interface first by running the following commands:
      			#chdev -l en0 -a state=down
      			#chdev -l en0 -a state=detach
    3. Enable mtu=9000 for the interface by running following command:
      			#chdev -l en0 -a mtu=9000
    4. Bring up the interface again by running following command:
      			#chdev -l en0 -a state=up
    5. Refresh routes on inet0 by running following command:
      			#mkdev -l inet0
    6. To verify interface is actually running at 9000 MTU, run following command:
      			# netstat -in

      In the output, Mtu value should be displayed as 9000.

Using large receive

The TCP large receive feature has the potential to improve throughput by reducing processor usage. VIOS SEA can be configured to enable large_receive attribute. The feature works at receiver side of the Network, by aggregating incoming packets into fewer large sized packets before passing on to higher network stack. This results in reduction of number of incoming packets to be processed and reduces processor usage.

To enable the large_receive attribute at VIOS SEA, perform following steps.

At the VIOS (assume ent7 is SEA adapter, ent0 is physical adapter, ent4 is Trunk adapter) instance:

  1. Run the following command to enable the large_receive attribute for the SEA physical adapter.
    				#chdev -l ent0 -a large_receive=yes
  2. Run the following command to enable the large_receive attribute for SEA.
    				#chdev -l ent7 -a large_receive=yes

Performance statistics

The example in this section provides improved performance statistics of using largesend attribute in AIX. To verify this, we sent a large file from one LPAR to another using FTP. The adapter type used is 10GbE SFP+( 4 Port adapter)

Network ports of source and destination LPARs are connected through a physical switch. The jumbo_frames attribute is not being used here. All frames are standard MTU 1500.

Case 1: Verify the number of processor cycles required without enabling LSO.

At the sender side, run the following commands:

# lsattr -El en0 | grep mtu_bypass | grep -v grep
mtu_bypass    off  Enable/Disable largesend for virtual  True

# lsattr -El ent0 | grep large_send | grep -v grep
large_send      no                Request Tx TCP segment offload                   True

# tprof -z -Askex ftp
Starting Command ftp
ftp> open 192.168.20.11
Connected to 192.168.20.11.
220 p8tld-shsr.aus.stglabs.ibm.com FTP server (Version 4.2 Fri Feb 17 15:41:34 CST 2017) ready.
Name (192.168.20.11:root): root
331 Password required for root.
Password:
230-Last unsuccessful login: Wed Jul 26 06:49:18 CDT 2017 on /dev/vty0
230-Last login: Wed Jul 26 06:55:47 CDT 2017 on /dev/pts/0 from 9.109.248.33
230 User root logged in.
ftp> put "| dd if=/dev/zero bs=3350000 count=10000" /dev/null
200 PORT command successful.
150 Opening data connection for /dev/null.
10000+0 records in.
10000+0 records out.
226 Transfer complete.
33500000000 bytes sent in 783.3 seconds (4.177e+04 Kbytes/s)
local: | dd if=/dev/zero bs=3350000 count=10000 remote: /dev/null
ftp> bye
221 Goodbye.
stopping trace collection.
Wed Jul 26 06:58:20 2017
System: AIX 7.2 Node: p8tld-mshns Machine: 00F912204C00
Generating ftp.trc
Generating ftp.syms
Generating ftp.prof

#vi ftp.prof
Configuration information
=========================
System: AIX 7.2 Node: p8tld-mshns Machine: 00F912204C00
Tprof command was:
tprof -z -Askex ftp
Trace command was:
/usr/bin/trace -ad -M -L 799465881 -T 500000 -j 00A,001,002,003,38F,005,006,134,210,139,5A2,5A5,465,234,5D8, 
-o ftp.trc
Total Samples = 107419
Traced Time = 830.43s (out of a total execution time of 830.43s)
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Process                                FREQ  Total Kernel   User Shared  Other
=======                                ====  ===== ======   ==== ======  =====
wait                                      4  93593  93593      0      0      0
/usr/bin/ftp                              1  11641   2410   9216     15      0


# netstat  -p tcp | grep largesend
0 bytes sent using largesend
0 bytes is the biggest largesend

Case 2: Verify the number of processor cycles required after enabling LSO.

At the sender side, run the following commands:

# lsattr -El ent0 | grep large_send | grep -v grep
large_send      yes               Request Tx TCP segment offload                   True

# lsattr -El en0 | grep mtu_bypass | grep -v grep
mtu_bypass    on   Enable/Disable largesend for virtual

# tprof -z -Askex ftp
Starting Command ftp
ftp> open 192.168.20.11
Connected to 192.168.20.11.
220 p8tld-shsr.aus.stglabs.ibm.com FTP server (Version 4.2 Fri Feb 17 15:41:34 CST 2017) ready.
Name (192.168.20.11:root): root
331 Password required for root.
Password:
230-Last unsuccessful login: Wed Jul 26 06:49:18 CDT 2017 on /dev/vty0
230-Last login: Wed Jul 26 06:58:38 CDT 2017 on ftp from ::ffff:192.168.20.10
230 User root logged in.
ftp>  put "| dd if=/dev/zero bs=3350000 count=10000" /dev/null
200 PORT command successful.
150 Opening data connection for /dev/null.
10000+0 records in.
10000+0 records out.
226 Transfer complete.
33500000000 bytes sent in 790.8 seconds (4.137e+04 Kbytes/s)
local: | dd if=/dev/zero bs=3350000 count=10000 remote: /dev/null
ftp> bye
221 Goodbye.
stopping trace collection.
Wed Jul 26 07:24:54 2017
System: AIX 7.2 Node: p8tld-mshns Machine: 00F912204C00
Generating ftp.trc
Generating ftp.syms
Generating ftp.prof


# vi ftp.prof
Configuration information
=========================
System: AIX 7.2 Node: p8tld-mshns Machine: 00F912204C00
Tprof command was:
tprof -z -Askex ftp
Trace command was:
/usr/bin/trace -ad -M -L 181625241 -T 500000 -j 00A,001,002,003,38F,005,006,134,210,139,5A2,5A5,465,234,5D8, 
-o ftp.trc
Total Samples = 105488
Traced Time = 839.04s (out of a total execution time of 839.04s)
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Process                                FREQ  Total Kernel   User Shared  Other
=======                                ====  ===== ======   ==== ======  =====
wait                                      4  96641  96641      0      0      0
/usr/bin/ftp                              1   7688   1202   6472     14      0


# netstat -p tcp | grep largesend
33495951175 bytes sent using largesend
65160 bytes is the biggest largesend

In this setup, we obtained 34% reduction in the number of processor cycles with LSO enabled. This reduction count might vary based on the actual network infrastructure. With a faster physical network, performance should improve. Performance also depends on the speed with which the application is feeding data.

Resources


Downloadable resources


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=1048798
ArticleTitle=Enabling largesend and jumbo_frames in IBM AIX to reduce processor usage
publish-date=08172017