Optimizing AIX 5L performance: Tuning network performance, Part 3

Monitoring your network packets and tuning the network

This three-part series on AIX® networking focuses on the challenges of optimizing network performance. Part 1 provided a networking overview and also discussed the tools you need to monitor your hardware, including netstat, netpmon, entstat, and nmon. Part 2 discussed monitoring and tuning NFS subsystems. This final part, Part 3, shows you how to monitor network packets The series also offers best practices for network I/O performance tuning.

Ken Milberg, Future Tech UNIX Consultant, Technology Writer, and Site Expert, Future Tech

Ken Milberg is a Technology Writer and Site Expert for techtarget.com and provides Linux technical information and support at searchopensource.com. He is also a writer and technical editor for IBM Systems Magazine, Open Edition. Ken holds a bachelor's degree in computer and information science and a master's degree in technology management from the University of Maryland. He is the founder and group leader of the NY Metro POWER-AIX/Linux Users Group. Through the years, he has worked for both large and small organizations and has held diverse positions from CIO to Senior AIX Engineer. Today, he works for Future Tech, a Long Island-based IBM business partner. Ken is a PMI certified Project Management Professional (PMP), an IBM Certified Advanced Technical Expert (CATE, IBM System p5 2006), and a Solaris Certified Network Administrator (SCNA). You can contact him at kmilberg@gmail.com.

29 January 2008

Also available in Chinese


While running commands such as netstat can provide useful information, sometimes you need to drill down more to the packet level. This is where tracing tools come in handy. The article describes these utilities, including tools such as iptrace, ipreport, and tcpdump. This article also shows how you can tune a network using tools such as no. The no command is similar to vmo and ioo, but no is the network flavor. This article focuses on tcp workload tuning, udp workload tuning, and some other noteworthy parameters with the no utility. The article also addresses ARP cache tuning and how to monitor and tune ARP statistics. You will also look at name resolution and how you can easily increase performance by making small adjustments in how to resolve hostnames.

Monitoring network packets

In this section, you'll see an overview of the tools available that help you monitor your network packets. These tools allow you to quickly troubleshoot a performance problem and to capture data for historical trending and analysis.

Part 1 addressed some of the very basic flags, such as -in, that you typically use with netstat. Using netstat, you can also monitor more detailed information about the packets themselves. For example, the -D option shows the overall number of packets received, transmitted, and dropped in your communication subsystem. The results are sorted by device, driver, and protocol (see Listing 1).

Listing 1. netstat with the -D option
root@lpar37p682e[/home] > netstat -D

Source                         Ipkts                Opkts     Idrops     Odrops
ent_dev0                   238122150                 1805          0          0
ent_dev1                    17583646               301547          0          0
Devices Total              255705796               303352          0          0
ent_dd0                    238122150                 1805          0          0
ent_dd1                     17583646               301547          0          0
Drivers Total              255705796               303352          0          0
ent_dmx0                   238011223                  N/A     110927        N/A
ent_dmx1                    17466977                  N/A     116669        N/A
Demuxer Total              255478200                  N/A     227596        N/A
IP                         238073400               301739       2392       1691
IPv6                               0                    0          0          0
TCP                             7373               296758         93          0
UDP                        238063379                 4677  238055978          0
Protocols Total            476144152               603174  238058463       1691
en_if1                      17466977               301547          0          0
en_if0                     238011223                 1805          0          0
lo_if0                           609                  619         10          0
Net IF Total               255478809               303971         10          0

Another useful flag is the -s, which shows detailed statistics for all protocols used, including packets sent, received and dropped. If you only want to view tcp, you can also use the -p flag (see Listing 2).

Listing 2. netstat with the -p option
root@lpar37p682e[/home] > netstat -p tcp

        297240 packets sent
                284116 data packets (18296667 bytes)
                0 data packets (0 bytes) retransmitted
                316 ack-only packets (119 delayed)
                0 URG only packets
                0 window probe packets
                38 window update packets
                12770 control packets
                0 large sends
                0 bytes sent using largesend
                0 bytes is the biggest largesend
        7675 packets received
                6277 acks (for 18288752 bytes)
                133 duplicate acks
                0 acks for unsent data
                1864 packets (141133 bytes) received in-sequence
                92 completely duplicate packets (91 bytes)
                0 old duplicate packets

There are actually so many different ways of using netstat that the best place to start is by looking at the man page and go from there. Don't be afraid to run these commands, because they won't eat up disk space or affect performance. The tracing tools that are provided within AIX are used to record detailed information about the packets. Use them with more caution. These tools are extremely helpful when trying to determine the root cause of network performance problems.

First, look at iptrace and ipreport. The iptrace command records all the packets received from the network interfaces. The ipreport command formats the data that is generated from iptrace into a readable trace report. Further, you can also use ipfilter to sort the output file created from ipreport. Try starting the trace and keep it going for one minute (see Listing 3).

Listing 3. Starting the trace
root@lpar37p682e[/etc] > /usr/sbin/iptrace -a -i en0 iptrace.out &
[1]     737520
root@lpar37p682e[/etc] > [774252]

[1] +  Done                    /usr/sbin/iptrace -a -i en0 iptrace.out &
root@lpar37p682e[/etc] > ps -ef | grep iptrace
    root 205030 749602   0 10:57:32  pts/0  0:00 grep iptrace
    root 774252      1   2 10:57:25      -  0:00 /usr/sbin/iptrace -a -i en0 iptrace.out

When you are done with the trace, you need to kill the process:

root@lpar37p682e[/etc] > kill -1 774252
root@lpar37p682e[/etc] > iptrace: unload success!
root@lpar37p682e[/etc] > ipreport -r -s iptrace.out >/ipreport.network

Now, examine the output. 

root@lpar37p682e[/] > more ipreport.network
IPTRACE version: 2.0

ETH: ====( 114 bytes transmitted on interface en0 )==== 10:57:25.698790226
ETH:    [ da:bb:b8:b5:26:14 -> 6e:87:76:59:6e:cd ]  type 800  (IP)
IP:     < SRC = >  (lpar37p682e)
IP:     < DST = >
IP:     ip_v=4, ip_hl=20, ip_tos=16, ip_len=100, ip_id=18349, ip_off=0 DF
IP:     ip_ttl=60, ip_sum=945f, ip_p = 6 (TCP)
TCP:    <source port=22(ssh), destination port=53643 >
TCP:    th_seq=337783617, th_ack=1783353394
TCP:    th_off=8, flags<PUSH | ACK>
TCP:    th_win=65522, th_sum=0, th_urp=0
TCP:            nop
TCP:            nop
TCP:            timestamps TSVal: 0x47414604  TSEcho: 0x47826117
TCP: 00000000     520bea13 dfaefa7b e1c517d6 ce86f960     |R......{.......'|
TCP: 00000010     fdb24d69 947c8d48 fa7b6379 235d1a63     |..Mi.|.H.{cy#].c|
TCP: 00000020     840adfc2 e1b4b916 e1002983 f96fc1fb     |..........)..o..|

Listing 3 shows the captured information about each packet, including packet size and IP address information. As you can imagine, the trace file can get very large fairly quickly. The example file grew to 40 MB in less than one minute! Be very careful when running these traces, because you will run out of disk space really fast if you don't have the disk bandwidth for these files.

You can also start the trace using the System Resource Controller (SRC). See Listing 4.

Listing 4. Starting the trace using SRC
# startsrc -s iptrace -a "-i en1 /home/testing/iptrace/iptracelog"

Stopping it is easy;

# stopsrc -s iptrace

What about tcpdump? tcpdump prints out headers of the packets, which are captured for each NIC. One important difference with tcpdump is that, unlike iptrace, it can look at only one network interface at a time. And, because iptrace examines the entire packet from the kernel space, the results can offer lots of dropped packets. With tcpdump, you can also limit the amount of data to be traced. Also, you do not need to use an ipreport type of command to format binary data, because tcpdump does the trace and the output. See Listing 5 for an example.

Listing 5. Using tcpdump
root@lpar37p682e[/] > 
root@lpar37p682e[/] > tcpdump -w tcp.out
tcpdump: listening on en0, link-type 1, capture size 96 bytes

tcpdump continues to capture packets until you hit Ctrl+C. If any packets were dropped due to a lack of buffer space, it reports that also.

Listing 6 shows what you see when you end the example trace.

Listing 6. End of the trace
14755 packets received by filter
0 packets dropped by kernel

13:40:28.001711 IP lpar37p682e.ssh > P 374368029:374368077(48) 
  ack 3207376412 win 65522 <nop,nop,timestamp 1195479662 1199746434>
13:40:28.001765 IP lpar37p682e.ssh > P 48:96(48) 
  ack 1 win 65522 <nop,nop,timestamp 1195479662 1199746434>
13:40:28.001872 IP lpar37p682e.ssh > P 96:144(48) 
ack 1 win 65522 <nop,nop,timestamp 1195479662 1199746434>
13:40:28.001925 IP lpar37p682e.ssh > P 144:192(48) 
ack 1 win 65522 <nop,nop,timestamp 1195479662 1199746434>
P 400:448(48) ack 1 win 65522 <nop,nop,timestamp 1195479662 1199746434>
13:40:28.066856 IP > lpar37p682e.ssh: . 
ack 448 win 32761 <nop,nop,timestamp 1199746434 1195479662>
13:40:28.149698 IP > 
     isakmp: phase 2/others ? #64[EC]
13:40:28.150470 IP > 
     isakmp: phase 2/others ? #64[EC]
13:40:28.151257 IP > 
     isakmp: phase 2/others ? #64[EC]
13:40:28.151954 IP > 
     isakmp: phase 2/others ? #64[EC]
13:40:28.152756 IP > 
     isakmp: phase 2/others ? #64[EC]
13:40:28.153449 IP > 
     isakmp: phase 2/others ? #64[EC]
13:40:28.154251 IP > 
     isakmp: phase 2/others ? #64[EC]
13:40:28.154950 IP > 
     isakmp: phase 2/others ? #64[EC]
13:40:28.155754 IP > 
     isakmp: phase 2/others ? #64[EC]

The output shows that the kernel dropped no packets, which is a good thing.

Tuning network performance

In this section, you will learn how to use the no command to tune your network subsystem. You'll also look at other areas that can impact network performance, and you'll learn about recommended tuning methodologies where appropriate.

The most important command for tuning network parameters is the no command. First, take a look at all the parameters, using the -a flag (see Listing 7).

Listing 7. Viewing the parameters
root@lpar37p682e[/] > no -a
                 arpqsize = 12
               arpt_killc = 20
              arptab_bsiz = 7
                arptab_nb = 149
                bcastping = 0
      clean_partial_conns = 0
                 delayack = 0
            delayackports = {}
         dgd_packets_lost = 3
            dgd_ping_time = 5
           dgd_retry_time = 5
       directed_broadcast = 0
         extendednetstats = 0
                 fasttimo = 200
        icmp6_errmsg_rate = 10
          icmpaddressmask = 0
ie5_old_multicast_mapping = 0
                   ifsize = 256
          inet_stack_size = 16
               ip6_defttl = 64
                ip6_prune = 1
            ip6forwarding = 0
       ip6srcrouteforward = 1
       ip_ifdelete_notify = 0
                 ip_nfrag = 200
             ipforwarding = 0
                ipfragttl = 2
        ipignoreredirects = 0
                ipqmaxlen = 100
          ipsendredirects = 1
        ipsrcrouteforward = 1
           ipsrcrouterecv = 0
           ipsrcroutesend = 1
          llsleep_timeout = 3
                  lo_perf = 1
                lowthresh = 90
                 main_if6 = 0
               main_site6 = 0
                 maxnip6q = 20
                   maxttl = 255
                medthresh = 95
               mpr_policy = 1
              multi_homed = 1
                nbc_limit = 262144
            nbc_max_cache = 131072
            nbc_min_cache = 1
         nbc_ofile_hashsz = 12841
                 nbc_pseg = 0
           nbc_pseg_limit = 524288
           ndd_event_name = {all}
        ndd_event_tracing = 0
            ndp_mmaxtries = 3
            ndp_umaxtries = 3
                 ndpqsize = 50
                ndpt_down = 3
                ndpt_keep = 120
               ndpt_probe = 5
           ndpt_reachable = 30
             ndpt_retrans = 1
             net_buf_size = {all}
             net_buf_type = {all}
        net_malloc_police = 0
           nonlocsrcroute = 0
                 nstrpush = 8
              passive_dgd = 0
         pmtu_default_age = 10
              pmtu_expire = 10
 pmtu_rediscover_interval = 30
              psebufcalls = 20
                 psecache = 1
             pseintrstack = 24576
                psetimers = 20
           rfc1122addrchk = 0
                  rfc1323 = 0
                  rfc2414 = 1
             route_expire = 1
          routerevalidate = 0
                 rto_high = 64
               rto_length = 13
                rto_limit = 7
                  rto_low = 1
                     sack = 0
                   sb_max = 1048576
       send_file_duration = 300
              site6_index = 0
               sockthresh = 85
                  sodebug = 0
              sodebug_env = 0
                somaxconn = 1024
                 strctlsz = 1024
                 strmsgsz = 0
                strthresh = 85
               strturncnt = 15
          subnetsarelocal = 1
       tcp_bad_port_limit = 0
                  tcp_ecn = 0
       tcp_ephemeral_high = 65535
        tcp_ephemeral_low = 32768
             tcp_finwait2 = 1200
           tcp_icmpsecure = 0
          tcp_init_window = 0
    tcp_inpcb_hashtab_siz = 24499
              tcp_keepcnt = 8
             tcp_keepidle = 14400
             tcp_keepinit = 150
            tcp_keepintvl = 150
     tcp_limited_transmit = 1
              tcp_low_rto = 0
             tcp_maxburst = 0
              tcp_mssdflt = 1460
          tcp_nagle_limit = 65535
        tcp_nagleoverride = 0
               tcp_ndebug = 100
              tcp_newreno = 1
           tcp_nodelayack = 0
        tcp_pmtu_discover = 1
            tcp_recvspace = 16384
            tcp_sendspace = 16384
            tcp_tcpsecure = 0
             tcp_timewait = 1
                  tcp_ttl = 60
           tcprexmtthresh = 3
                  thewall = 1048576
         timer_wheel_tick = 0
       udp_bad_port_limit = 0
       udp_ephemeral_high = 65535
        udp_ephemeral_low = 32768
    udp_inpcb_hashtab_siz = 24499
        udp_pmtu_discover = 1
            udp_recvspace = 42080
            udp_sendspace = 9216
                  udp_ttl = 30
                 udpcksum = 1
                 use_isno = 1
           use_sndbufpool = 1

Alternatively, you can also use the -L flag, which provides much more detailed information. Listing 8 shows you only the first few lines.

Listing 8. Using the -L flag
root@lpar37p682e[/] > no -L | more
General Network Parameters
NAME                      CUR    DEF    BOOT   MIN    MAX    UNIT           TYPE
extendednetstats          0      0      0      0      1      boolean           R
fasttimo                  200    200    200    50     200    millisecond       D
inet_stack_size           16     16     16     1      32K-1  kbyte             R

There are many parameters here. Look at the most noteworthy. In older versions of AIX, thewall was an important tunable for which you needed to change the defaults. This parameter defines the upper limit for network kernel buffers. Today, its size is defined at installation time, depending on the amount of RAM and the kernel type. For example, if you are running AIX 5.3 on a 64-bit kernel, the parameter is set at half the size of real memory.

Listing 9. Setting the size of the thewall parameter
root@lpar37p682e[/] > vmstat 1
System configuration: lcpu=2 mem=2048MB ent=0.20

# no -a | grep thewall
                  thewall = 1048576

Part 1 discussed mbufs, but it's worth another mention here, because it relates to thewall. Remember that mbufs are used to store data in the kernel for both incoming and outgoing traffic. This is why determining the right amount of mbufs is extremely important. The value of the maxmbuf tunable limits the amount of memory that the communication systems use. If the value is 0, thewall tunable is used, and it cannot be modified from its default. Changing this tunable is a way to lower the thewall limit. As the default, if maxmbuf is 0, this value is used regardless of what thewall uses. netstat -m is used to detect shortages of failures of network memory requests (see Listing 10).

Listing 10. netstat with the -m option
root@lpar37p682e[/etc/tunables] >             netstat -m

Kernel malloc statistics:

******* CPU 0 *******
By size           inuse     calls failed   delayed    free   hiwat   freed
32                  117       217      0         0      11    5240       0
64                  109      6523      0         1      83    5240       0
128                 975     15951      0        29     785    2620       0
256                 520     67637      0        30    1016    5240       0

Streams mblk statistic failures:
0 high priority mblk failures
0 medium priority mblk failures
0 low priority mblk failures

In the example, there are no shortages (failures).

Although there are many parameters you can change with the no utility, most of them are better left alone. The most important parameters are ones that refer to TCP streaming workload tuning.

  • tcp_sendspace—This controls how much buffer space in the kernel is used to buffer application data. You really want to bump this up from the default, because if its limit is reached, the sending application suspends data transfer until TCP sends the data to the buffer.
  • tcp_receivespace—In addition to controlling the amount of buffer space to be consumed by receive buffers, AIX also uses this value to determine the size to make its transmit window.
  • udp_sendspace—With UDP, you can set this to no more than 65536, because IP has an upper limit of 65536 bytes per packet.
  • udp_resvspace—This value should be greater than udp_sendpsace, because it needs to handle as many simultaneous UDP packets per socket as it can. This parameter can easily be set to 10 times the value of udp_sendspace.

Now, make some changes. First, increase the size of udp_sendspace (see Listing 11).

Listing 11. Increasing the size of udp_sendspace
root@lpar37p682e[/] > no -p -o udp_sendspace=65536
Setting udp_sendspace to 65536
Setting udp_sendspace to 65536 in nextboot file

Next, change udp_recsvspace to the recommended configuration of 10 times udp_sendspace). See Listing 12.

Listing 12. Changing udp_recsvspace
root@lpar37p682e[/] > no -p -o udp_recvspace=655360
Setting udp_recvspace to 655360
Setting udp_recvspace to 655360 in nextboot file
Change to tunable udp_recvspace, will only be effective for future connections
root@lpar37p682e[/] >

Note that the -p flag keeps the entries, even after a reboot. It appends the /etc/tunables/nextboot stanza file, as shown in Listing 13.

Listing 13. Looking at the /etc/tunables/nextbook file
# tail /etc/tunables/nextboot

        udp_recvspace = "655360"
        udp_sendspace = "65536"
root@lpar37p682e[/etc/tunables] >

Regarding the tcp parameters for higher speed adapters, there is no problem setting tcp_sendspace to twice the value of tcp_recvspace. For example, you can use the settings in Listing 14.

Listing 14. Examples settings for tcp_sendspace
tcp_receivespace = 262144
tcp_sendspace= 524288

Other important workload parameters include rfc1323 and sb_max.

The rfc1323 tunable enables the TCP window scaling option, which allows TCP to use a larger window size. Turning it on enables the best TCP performance. The sb_max tunable sets an upper limit on the number of socket buffers queued to an individual socket, which controls the amount of buffer space consumed by buffers (queued to either a sender or received socket). This amount should usually be less than the wall and approximately 4 times the size of the largest value of the tcp or udp send and receive settings. For example, if your udp_recvspace is 655360, you can't go wrong by doubling this to 1310720.

Now look at tcp_nodelayack. This tunable prompts TCP to send an immediate acknowledgement, rather than a delayed acknowledgement. While this can add more overhead in some environments, it can greatly improve network performance in others. If you change this parameter but it does not improve performance, you can quickly change it back.

Next look at ipqmalen. This tunable controls the length of the IP input queue. If you see an overflow counter (through the use of netstat -s), setting a maximum length of this queue can help fix the overflow.

What about ARP? When many clients are connected to the system, you might want to tune the ARP cache. You can look at the statistics using netstat (see Listing 15).

Listing 15. Using netstat with -p arp
root@lpar37p682e[/etc/tunables] > netstat -p arp
        10 packets sent
        0 packets purged
root@lpar37p682e[/etc/tunables] >

If you see a high purge count, increase the size of the ARP table. For the example table, this isn't needed.

Here are the no parameters that relate to ARP (see Listing 16).

Listing 16. Using the no parameters
root@lpar37p682e[/etc/tunables] > no -a | grep arp
                 arpqsize = 12
               arpt_killc = 20
              arptab_bsiz = 7
                arptab_nb = 149

You can tune these buffers either system-wide or by specific interfaces. To tune by interface, set the no tunable isno option to 1, which is actually enabled by default in AIX 5.3.

Listing 17. Setting the no tunable isno option to 1
root@lpar37p682e[/etc/tunables] > no -a | grep use
                 use_isno = 1

Setting the disable option to 0 is used as a diagnostic tool of sorts to set the values across the board to help isolate performance problems. When these values are set for the specific interfaces, they actually override the default value in the no view, which can sometimes confuse system administrators. You can view the specific interface settings using either ifconfig or lsattr. In the example, look at the settings using ifconfig (look at the last line, which references some of the tunables mentioned earlier).

Listing 18. Viewing specific interface settings using ifconfig
root@lpar37p682e[/etc/tunables] > ifconfig en0
        inet netmask 0xffffc000 broadcast
         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

You can change these options (by interface) by using SMIT, chdev, or ifconfig. Note that ifconfig will not update the Object Data Manager (ODM), so on a reboot, it will revert to the previous values. Because of that, you should use SMIT: the fastpath of smit tcpip>further configuration>Network interfaces>Change/Show characteristics of an interface (see Figure 1).

Figure 1. Using SMIT to change interface settings
Using SMIT to change interface settings

You might wonder why the no parameters don't apply to some interfaces. Name resolution is another area that can impact performance. If you know how you want to resolve (using DNS or the hosts file), make sure name resolution is set up correctly in the /etc/netsvc.conf file. Look at a piece of the file in Listing 19.

Listing 19. Piece of /et/netsvc.conf file
# Example:
# aliases = nis, files
root@lpar37p682e[/etc/tunables] >

If you're using DNS, take out the local if you are not using a hosts file at all, or you can leave it in if you are using it as a backup to DNS (but make it the second entry). Alternatively, take out the bind if you're not using DNS at all, because it will only slow down your performance by first attempting (if it is the first entry in the record) to resolve using a Name Server that doesn't exist.


This article (Part 3 of a 3-part series on tuning network performance) discussed how to monitor network packets on the network. You used netstat and drilled down to the packet level using tracing tools, such as iptrace and tcpdump. Further, you learned how to tune your network using the no utility. Using this utility, you explored tcp and udp workload tuning while also learning some other noteworthy parameters. You made tuning changes and read about how you might want to tune certain settings. You also examined ARP cache tuning and saw how you could monitor and tune ARP statistics. You looked at ISNO and learned how you could tune specific no tunables by interface. You also looked at name resolution and how you could easily increase performance by making small adjustments in how to resolve hostnames.

This series discussed network tuning best practices and how you can benefit from effective monitoring of your I/O subsystems. There are many other open source and commercial network utilities—hardware and software—that can help monitor and troubleshoot performance problems. While the scope of this series was on tools available on AIX, don't be shy about researching other tools that might also be of value to you.



Get products and technologies



developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks

Zone=AIX and UNIX
ArticleTitle=Optimizing AIX 5L performance: Tuning network performance, Part 3