Optimizing AIX 6.1 performance tuning

Tuning methodology, general enhancements, monitoring, and tuning

AIX® 6.1 has many important innovations and improvements, including:

  • Virtualization—Features such as workload partitioning and Live Application Mobility
  • Security—Features such as encrypted file systems and trusted AIX and role-based access control (RBAC)
  • Availability—Features such as AIX concurrent updates and dynamic tracing
  • Manageability—Features such as the new Systems Director Console for AIX and Workload Partition Manager

It also has support for POWER6 performance innovations, such as advanced simultaneous multithreading (SMT), shared dedicated processors, and variable page size. It's important to fully understand which innovations and enhancements are more the reflection of the POWER6, AIX 6.1, or a combination of both. For example, purely from an operating system perspective, AIX improves on the older defaults tunables for:

  • vmo
  • ioo
  • aio
  • no
  • nfso
  • schedo

While there are some real performance enhancements in AIX 6.1, such as improvements in I/O pacing and AIX's implementation of AIO servers, there is nothing breathtakingly different. In fact, there were more performance changes made from AIX 5.1 to AIX 5.2 and 5.2 to 5.3 (including new monitoring tools, tuning tools, new tunables that changed how you set Virtual Memory Manager (VMM) settings, and concurrent I/O improvements) than you will see moving from AIX 5.3 to AIX 6.1. In AIX 6.1, all the tuning commands remain the same, except for the ones that have been taken away (I.E. and aioo), and there are no new monitoring tools either. Other changes reflect updates that were made to the utilities that reflect support for other workload partitioning innovations, including:

  • curt
  • filemon
  • iostat
  • netpmon
  • pprof
  • procmon
  • proctree
  • svmon
  • topas
  • tprof
  • vmstat

Workload partitions (WPARs) allow for the usage of separate virtual partitions within one AIX image. It is more of a compliment to logical partitions (LPARs) rather than a replacement to them. WPARs actually run inside of LPARs and are very similar in concept to Solaris containers. I've built WPARs in less than 15 minutes; in fact, you'll do some of your analysis inside WPARs so that you can actually view some of the updated tools that now support WPARs. Note that WPARs are only required in AIX 6.1 and a POWER6 is not necessary. Some commands also run differently or not run at all within WPARs—a few of them will be discussed where applicable. Other AIX 6.1 performance improvements include unique tunable documentation and restricted tunables.

Tuning methodology

This section summarizes a tuning methodology that you should use to tune your systems. You don't necessarily have to use this one; there are many others, but you should be working with some kind of tuning methodology or structured program.

The five-step methodology consists of:

  1. Establishing a baseline
  2. Stress testing and monitoring
  3. Identification of bottleneck
  4. Tuning bottleneck
  5. Repeat (starting with the second step)

Let's look at the steps in more depth:

  1. Establishing a baseline—Before you tune or even start monitoring, you must establish a baseline. The baseline is a snapshot of what the system looks when it is performing well. This baseline should not only capture performance-type statistics, but it should also document the actual configuration of your system (amount of memory, CPU, and disk). This is important, because you need to know what a system that performs well looks like prior to troubleshooting performance that users might have complained about. Doing so helps you translate the data into a service level agreement (SLA) in which the customer signs off on.
  2. Stress testing and monitoring—Here is where you monitor and stress your systems at peak workloads. You should use several monitoring tools here to help validate your findings. The monitoring section is critical, as you cannot effectively tune anything without having an accurate historical record of what has been going on in your system, particularly during periods of stress.
  3. Identification of bottleneck—The objective of stressing and monitoring your system is to determine the bottleneck. You cannot provide the correct medicine without the proper diagnosis. If the system is in fact CPU bound, then you can run additional tools, such as trace, curt, splat, tprof, and ps to further identify the actual processes that are causing the bottleneck.
  4. Tuning bottleneck—After you've finally identified the bottleneck, it is time to tune it. The tuning that you do is dependent on the bottleneck, for example, CPU, virtual memory, or I/O. Some subsystems lend themselves more to active tuning, such as virtual memory and using vmo, while the "cure" for CPU-type bottlenecks is usually managing your workload more efficiently or assigning (using dynamic logical partitioning (DLPAR), uncapped partitions, or Partition Load Manager) more resources to your system.
  5. Repeat—Time to go through this process again, starting from the second step of the process. Only by repeating your tests and consistently monitoring your systems can you determine if your tuning has really made an impact.

AIX 6.1—memory

This section discusses the changes made in the virtual memory subsystem.

Through the years, many have complained about the default parameters of AIX. The complaints have not just reflected the reality of what most users run on top of AIX, for example, mission-critical database applications like Oracle. Because of this, systems administrators have had to change settings on many subsystems—most notably with virtual memory (in other words, minperm and maxperm). IBM® has listened and changed these parameters to reflect that reality. It should also be noted that you should not rely on these settings exclusively, and you should always check with your independent software vendor (ISV) to validate what their recommended settings are for AIX 6.1 and make the changes, accordingly.

Let's look at a side-by-side view of vmo parameters for AIX 5.3 and AIX 6.1 in Table 1. You will see that some parameters are now in restricted mode. The most important changes were made to address paging issues, where database servers frequently page out computational pages, even though the system has enough free memory. In prior articles on tuning (see Related topics), it was recommended to change those parameters to defaults fairly close to what was indicated on the table. The changes are indicated in the AIX 5.3 tuning recommendation column. Many tunables are now also classified differently as restricted in an attempt to discourage changing certain parameters. The net is that only 29 vmo tunables can now be changed without a firm warning; 30 are now deemed restricted tunables, which IBM officially states should not be modified unless instructed by "IBM support professionals." IBM has made this change in an attempt to discourage junior administrators from changing certain parameters that were deemed critical enough to be classified as restricted.

Table 1. List of vmo parameters and tuning recommendations
TypeAIX 5.3AIX 5.3 tuning change recommendationsAIX 6.1Restricted
cpu_scale_memp = 888Yes
data_stagger_interval = 161161161Yes
defps = 11
force_relalias_lite = 000
framesets = 222no longer availableYes
htabscale = n/an/an/aYes
kernel_heap_psize = 409640964096
kernel_psize = 4096409665536Yes
large_page_heap_size = 000Yes
lgpg_regions = 000
lgpg_size = 000
low_ps_handling = 110
lru_file_repage = 1100Yes
lru_poll_interval = 101010Yes
lrubucket = 131072131072131072Yes
maxclient% = 80809090Yes
maxfree = 108810881088
maxperm = 598781598781442683
maxperm% = 80809090Yes
maxpin = 635249635249423471
maxpin% = 808080
mbuf_heap_psize = 655366553664436Yes
memory_affinity = 111Yes
memory_frames = 786432786432524288
memplace_data = 222
memplace_mapped_file = 222
memplace_shm_anonymous = 222
memplace_shm_named = 222
memplace_stack = 222
memplace_text = 222
memplace_unmapped_file = 222
mempools = 55
minfree = 960960960
minperm = 14969414969414756
minperm% = 202053
nokilluid = 000
npskill = 307230723072
npsrpgmax = 24576245768192Yes
npsrpgmin = 18432184326144Yes
npsscrubmax = 24576245768192Yes
npsscrubmin = 18432184326144Yes
npswarn = 122881228812288
num_spec_dataseg = 000Yes
numpsblks = 393216393216393216
page_steal_method = 001Yes
pagecoloring = n/an/a
pinnable_frames = 675516675516349814
pta_balance_threshold = n/an/a
relalias_percentage = 000
rpgclean = 000Yes
rpgcontrol = 222Yes
scrub = 000
scrubclean = 000Yes
soft_min_lgpgs_vmpool = 00
spec_dataseg_int = 512512512Yes
strict_maxclient = 111Yes
strict_maxperm = 000Yes
v_pinshm = 000
vm_modlist_threshold = -1-1-1Yes
vmm_fork_policy = 111Yes
vmm_mpsize_support = 112Yes

A new vmo flag, -F, provides all parameters, including restricted ones. Most of the content here is stripped out—only a small sampling is provided that includes the restricted section (see Listing 1).

Listing 1. Using the -F flag with vmo to view all the tunable values
lpar9ml162f_pub[/] > vmo -F -a 
   force_relalias_lite = 0
      vmm_default_pspa = -1
##Restricted tunables
              maxperm% = 90

Even restricted tunables can be changed. When you do make a change, you just receive a stern warning (see Listing 2).

Listing 2. Adjusting tunables warning
lpar9ml162f_pub[/] > vmo -o maxperm%=99
Setting maxperm% to 99
Warning: a restricted tunable has been modified

When changing the parameter after a reboot, you'll get a further rebuke and be asked to actually confirm whether or not you want do to this. You'll have to physically type in "yes" (see Listing 3).

Listing 3. Restricted tunables—answering yes to change
lpar9ml162f_pub[/] > vmo -p -o  maxperm%=99
Modification to restricted tunable maxperm%, confirmation required yes/no yes
Setting maxperm% to 99 in nextboot file
Setting maxperm% to 99
Warning: a restricted tunable has been modified
lpar9ml162f_pub[/] >

The most important out-of-the-box performance values include new values for:

  • minperm
  • maxperm
  • maxclient
  • strict_maxclient

This is based on changes that first appeared during AIX 5.3, when you no longer had to turn off strict_maxclient, increase minfrree and maxfree, or reduce minperm, maxperm, and maxclient. The new recommendation (now incorporated as the default value in AIX 6.1) is to turn off the repage ratio check, lru_file_repage, which ensures that working storage is not paged and to consider only file paging. In AIX 6.1, the VMM replacement default is changed to utilize up to 90 percent of its real memory for file caching, favoring computational pages over file pages. Unless the amount of active virtual memory exceeds 97 percent of the size of real memory, minperm is reduced to three percent (see Listing 4) to ensure that computational pages will not be stolen.

Listing 4. minperm reduced to three percent
# vmo -o minperm%=97
Value of the tunable minperm% cannot be changed in a WPAR

Some changes will not work in WPARs. WPARs are a subset of an LPAR, but they are still part of the single operating system image.

Another important change includes VMM dynamic variable page size support (VPSS). Pages are defined as fixed length data blocks and held in virtual memory. In AIX 6.1 (on POWER6 processors only), VMM can now dynamically use the larger page size based on application memory usage, which should substantially improve performance. This feature is completely transparent to applications. AIX only uses the larger pages sized if it doesn't result in increasing process memory usage. The use of larger pages increases performance because fewer hardware address translations would need to be made. It only supports doing this on working storage memory, not persistent storage. The new parameters are vmm_default_pspa (it also works in conjunction with the vmm_mpsize_support tunable).

Listing 5. lsattr output illustrating processor architecture
lpar9ml162f_pub[/usr/samples/tcpip] > lsattr -El proc0
Frequency   1654344000     Processor Speed       False
smt_enabled true           Processor SMT enabled False
smt_threads 2              Processor SMT threads False
state       enable         Processor state       False
type        PowerPC_POWER5 Processor type        False
lpar9ml162f_pub[/usr/samples/tcpip] >

Let's view the setting for tunable settings for VPSS.

Listing 6. VPSS automatically disabled on a POWER5 system
lpar9ml162f_pub[/usr/samples/tcpip] > vmo -a | grep pspa
      vmm_default_pspa = -1


This section highlights changes made in the CPU subsystem.

While schedo can also be used to tune certain memory facilities, such as paging load control, it is mostly there for CPU scheduler tuning. In AIX 6.1, only 27 of the 42 tunables are restricted, leaving 15 tunables that can be modified without explicit warnings. Though some defaults were changed, there were no substantial changes that were made with respect to CPU monitoring and tuning on AIX 6.1 (see Listing 7).

Listing 7. Restricted tunables
lpar9ml162f_pub[/] > schedo  -F -a
##Restricted tunables
              %usDelta = 100
       allowMCMmigrate = 0
            fast_locks = n/a
       hotlocks_enable = 0
idle_migration_barrier = 4
    krlock_confer2self = 1
  krlock_conferb4alloc = 1
         krlock_enable = 1
    krlock_spinb4alloc = 1
   krlock_spinb4confer = 1024
    n_idle_loop_vlopri = 100
 search_globalrq_mload = 256
  search_smtrunq_mload = 256
  setnewrq_sidle_mload = 384
   shed_primrunq_mload = 64
    sidle_S1runq_mload = 64
    sidle_S2runq_mload = 134
    sidle_S3runq_mload = 134
    sidle_S4runq_mload = 4294967040
    slock_spinb4confer = 1024
      smt_snooze_delay = 0
     smtrunq_load_diff = 2
         v_exempt_secs = 2
         v_min_process = 2
           v_repage_hi = 0
         v_repage_proc = 4
            v_sec_wait = 1
lpar9ml162f_pub[/] >

AIX 6.1—Disk I/O

This section gives an overview of the changes that were made in the disk I/O subsystem.

In AIX 6.1, of the 48 tunables you can control with ioo, 27 of them are now restricted, leaving 21 tunables that can be modified without explicit warnings. The most important changes impact I/O pacing and AIO dynamic tunables (see Listing 8).

Listing 8. Restricted ioo tunables
# ioo  -F -a
##Restricted tunables
                  aio_fastpath = 1
                aio_fsfastpath = 1
                 aio_kprocprio = 39
              aio_multitidsusp = 1
               aio_sample_rate = 5
         aio_samples_per_cycle = 6
       j2_maxUsableMaxTransfer = 512
      j2_nBufferPerPagerDevice = 512
      j2_nonFatalCrashesSystem= 0
         j2_syncModifiedMapped = 1
       j2_syncdLogSyncInterval = 1
            jfs_clread_enabled = 0
             jfs_use_read_lock = 1
                    maxpgahead = 8
                    maxrandwrt = 0
                 memory_frames = 524288
                    minpgahead = 2
                      numclust = 1
                     numfsbufs = 196
            pgahd_scale_thresh = 0
            posix_aio_fastpath = 1
          posix_aio_fsfastpath = 1
           posix_aio_kprocprio = 39
         posix_aio_sample_rate = 5
   posix_aio_samples_per_cycle = 6
                   pv_min_pbuf = 512
            sync_release_ilock = 0


Changes were made to the Enhanced Journaling File System (JFS2) that now allow you to mount a JFS2 f/s without logging. While this can substantially improve performance, it is not recommended you do this. If you need to recover your data, you have to use the dreaded fsck, which has been pretty much banished from memory since journaling file systems. Some useful circumstances can include restoring data from backups and saving time during an activity where you might have a very small window and availability is not a concern.


The target software driver can now be used over a gigabyte Ethernet adapter, which should clearly improve performance in this type of environment. The target driver exports local disks or logical volumes to iSCSI initiators that connect to AIX using the iSCSI protocol. The proliferation of iSCSI is a viable alternative over Fiber-based storage, making this an important enhancement.

I/O pacing

Disk I/O pacing is a mechanism that allows you to limit the number of pending I/O requests to a file. What this does is prevents disk I/O-intensive processes (usually in the form of large sequential writes) from exhausting the CPU. AIX 6.1 enables I/O pacing by default. In AIX 5.3, you needed to explicitly enable this feature. The way it does this is by setting the sys0 settings of minpout and maxput parameters to 4096 and 8193, respectively:

lpar9ml162f_pub[/] > lpar9ml162f_pub[/] > lsattr -El sys0
Figure 1. lsattr output illustrating new I/O pacing settings
lsattr output
lsattr output

Asynchronous I/O (AIO)

AIO is an AIX software subsystem that allows processes to issue I/O operations without waiting for I/O to finish. Because both I/O operations and applications processing run concurrently, they essentially run in the background and improve performance. This is particularly important in a database environment. There are two types of AIX subsystems: Legacy AIO and POSIX AIO. The differences between them involve different parameter passing at the application layer. In other words, the developers pick the implementation that the application uses. Regardless of which subsystem is chosen, both run concurrently on AIX. In AIX 5L™, if applications used AIO, the subsystem would need to be explicitly activated in the autoconfig parameter. The system also requires a reboot because the kernel extensions need to be loaded. In fact, any release prior to AIX 5.3 TL5 required reboots if there were any changes to the either of the following tunables:

  • maxreqs
  • maxservers
  • minservers

In AIX 5.3, you also had the aioo command, which allowed you to make these changes dynamically without a reboot (decreasing required reboots). It should be noted that the command did not change the Object Data Manager (ODM) attributes, meaning they would not be persistent across a reboot.

In AIX 6.1, the tunables fastpath and fsfastpath are now restricted tunables and set to 1 by default. It has the following effect on the tunables (see Listing 9):

  • fastpath: AIO requests that raw logical volumes get passed directly to the disk layer.
  • fsfastpath: AIO requests that files opened with concurrent I/O on JFS2 get passed directly to LVM or disk.
Listing 9. Restricted tunables
##Restricted tunables
                  aio_fastpath = 1
                aio_fsfastpath = 1

Further, AIO subsystems are now loaded by default and not activated. They are automatically started at the time when the application initiates the AIO I/O requests. There is no more aioo command (what a short life span) and these tunables are now used only with ioo (see Listing 10).

Listing 10. AIX old method (AIX 5.3)
lpar9ml24f_pub[/] > # aioo -a

            minservers = 1
            maxservers = 1
             maxreqs= 4096
             fsfastpath = 0

Listing 11 provides the new method for AIX 6.1.

Listing 11. AIO new method (AIX 6.1)
lpar9ml162f_pub[/] > ioo -a | grep active
                    aio_active = 0
              posix_aio_active =

It's also worth noting that there are no more AIO devices in the ODM.

As you can see, two new parameters have also been added to ioo: aio_active and posix_aix_active. These can only be changed by AIX, and they are set to 1 only when AIO kernel extensors are used and pinned. If you like grep, you won't find any more AIO servers. You'll now see aioLpools and aioPpools; these are the kernel processes that manage the AIO subsystems for Legacy and POSIX. As a result of this change, there is less pinned memory and fewer processes that are running on the system; both have positive affects on overall systems performance (see Listing 12).

Listing 12. New AIO kernel processes
lpar9ml162f_pub[/] > pstat -a | grep aio
 39 a   2704e      1  2704e     0     0     1  aioLpool
 40 a   28050      1  28050     0     0     1  aioPpool
lpar9ml162f_pub[/] >

The minserver and maxserver parameters, as they relate to AIO servers, are now tuned per each CPU tunable. Changing these values will not result in changes to the number of available servers on the system; the amount is dependent on the number of concurrent I/O requests (see Listing 13).

Listing 13. Changes in default parameters for minservers and maxservers
lpar9ml162f_pub[/] > ioo -a | grep minservers
                aio_minservers = 3
          posix_aio_minservers = 3
lpar9ml162f_pub[/] > ioo -a | grep maxservers
                aio_maxservers = 30
          posix_aio_maxservers = 30
lpar9ml162f_pub[/] >

It's also worth noting that there are no more AIO devices in the ODM.

AIX 6.1—Network and Network File System (NFS) I/O

This section outlines changes that were made in the Network I/O subsystem, including NFS.

Of the 133 no tunables, only five have been classified as restricted (see Listing 14).

Listing 14. Restricted tunables
#no  -F -a 
##Restricted tunables
         extendednetstats = 0
          inet_stack_size = 16
        net_malloc_police = 16384
             pseintrstack = 24576
                 use_isno = 1
lpar9ml162f_pub[/] >

A new network caching daemon has also been introduced to improve performance when resolving using DNS.

This daemon can be started from the System Resource Controller (SRC).

It's main configuration file is in /etc/netcd.conf, and you can copy the one in /usr/samples/tcpip to /etc and use that as a template. The command used to manage the daemon is netcdctrl. With this command, you can dump the cache contents to a file, display statistics of cache usage, flush the cache table, and change the logging level of the daemon. Listing 15 displays the stats.

Listing 15. Displaying caching stats with netcdctrl
lpar9ml162f_pub[/usr/samples/tcpip] > netcdctrl -t dns -e hosts -s /tmp/netcd.stats
lpar9ml162f_pub[/usr/samples/tcpip] > more /tmp/netcd.stats
CACHE dns, hosts, name
Hash index : 0, Max number of entries : 0, Current number of entries : 0
Hash index : 1, Max number of entries : 0, Current number of entries : 0
Hash index : 2, Max number of entries : 0, Current number of entries : 0
Hash index : 3, Max number of entries : 0, Current number of entries : 0

Regarding the /etc/netsvc.conf file, nothing has changed; this file is still necessary in determining the order of resolving.


Of the 24 NFS tunables, 21 have been classified as restricted (see Listing 16).

Listing 16. Restricted tunables
#nfso -a 

##Restricted tunables
           lockd_debug_level = 0
       nfs_allow_all_signals = 0
        nfs_auto_rbr_trigger = 0
         nfs_dynamic_retrans = 1
        nfs_gather_threshold = 4096
            nfs_iopace_pages = 0
             nfs_max_threads = 3891
         nfs_repeat_messages = 0
              nfs_socketsize = 600000
nfs_tcp_duplicate_cache_size = 5000
          nfs_tcp_socketsize = 600000
nfs_udp_duplicate_cache_size = 5000
                 nfs_v2_pdts = 1
                 nfs_v3_pdts = 1
                 nfs_v4_pdts = 1
              nfs_v2_vm_bufs = 10000
              nfs_v3_vm_bufs = 10000
              nfs_v4_vm_bufs = 10000
           statd_debug_level = 0
           statd_max_threads = 50
                 udpchecksum = 1
lpar9ml162f_pub[/] >

The only noteworthy change here is that RFC1323 (on the TC/IP stack) is now enabled by default, which allows TCP connections to use the TCP scaling window for any NFS connections. Further, the default number of biod daemons has now increased to 32 for each NFS V3 mount point.


AIX 6.1 provides many innovations, such as the use of WPARs and live application availability. It also provides for many changes with respect to performance tuning. Some of the more important changes include improving default parameters to more accurately reflect real-world data processing, unique tunable documentation, the incorporation of restricted tunables, and enhancements to I/O pacing and AIO servers. AIX 6.1 is simply wonderful, but also remember that in the world of IBM, AIX 6.1 is the new 6.0. If you recall, AIX 5.2 was a lot better and more stable than AIX 5.1. Consider everything before doing an operating system upgrade, but certainly don't be afraid of moving over if your application fully supports it and you feel the innovations are important enough in your environment to make the move.

Downloadable resources

Related topics

Zone=AIX and UNIX
ArticleTitle=Optimizing AIX 6.1 performance tuning