About this series
This three-part series focuses on the various aspects of memory management and tuning on IBM System p™ servers running AIX®. Part 1 provides an overview of memory on AIX, including a discussion of virtual memory and the Virtual Memory Manager (VMM). It also drills down into the tuning parameters and outlines recent improvements in AIX Version 5.3 with respect to memory management. Part 2 focuses on the detail of actual memory subsystem monitoring and discusses how to analyze the results. Part 3 focuses specifically on swap space and how best to tune your VMM settings to provide for optimum swap space configuration and performance.
Just what is swap (paging) space? It all starts with the VMM. VMM uses swap space (paging) as a holding bin for a process that is not using active RAM. Because of its purpose, it is a critical component of overall system performance. As an administrator, you need to know how to monitor and tune your paging parameters. The paging space itself is a special logical volume that stores the information that is currently not accessed. You must make sure that your system has adequate paging space. If the paging space is too low, entire processes can be lost and the system can crash when your space fills up. Though it is important to reiterate that paging is a normal part of VMM, it is even more important you really understand how the kernel brings the process into RAM—too much paging definitely hinders performance. AIX, through tight integration of the kernel and VMM, makes use of a methodology called demand paging. In fact, most of the kernel itself resides in virtual memory, which helps free up segments for other processes. I'll dig deeper into how this works and discuss some of the tools you need to use to manage and tune your paging space.
You will find that the tuning you do is based on what type of system you have. For example, systems that are using an Oracle Online Transaction Processing (OLTP) type of database usually have specific recommendations on how much swap space to configure and how to tune the paging parameters. As discussed in previous installments of the series (see Resources), you cannot really tune your paging settings unless you really know what is going on in the host system. You need to understand the tools to use, how best to analyze the data that you will be capturing, and familiarize yourself with best practices for implementing your paging space. It has been my experience that the number one cause for a system crash is running out of paging space. If you read this article carefully and follow its recommendations, this should never happen to you. Obviously, you never want your system to crash but, if it does, you want it to be due to a hardware failure and nothing that you did or forgot to do as the systems administrator.
In this section, I provide an overview of how AIX handles paging, define swapping and paging, and drill down into the different modes of paging space allocation. These concepts help you understand subsequent sections on monitoring, configuring, and tuning.
Most administrators think of paging as something that is onerous. Paging is actually a very normal part of what AIX does, due to the tight integration of its kernel with the VMM and its implementation of demand paging. The way demand paging works is that the kernel only loads a few pages at a time into real memory. When the CPU is ready for another page, it looks at the RAM. If it cannot find it there, a page fault occurs, and this signals the kernel to bring more pages into RAM from disk. One advantage of demand paging is that the paging space does not have to be particularly large, because data is constantly being shuffled between paging space and RAM. On older UNIX® systems, paging was preallocated to disks, whether they were used or not. This caused a condition where disk space would be allocated that was never used. Demand paging, in essence, avoids the condition where this disk space is allocated for no purpose. Swapping of processes is kept to a minimum, because many more jobs can be stored in RAM. This is true, because only parts of processes (pages) are stored in RAM.
What about swapping? Though often used interchangeably, there is a subtle difference between paging and swapping. As discussed, only parts of the process are moved back and forth between disk and RAM with paging. When swapping occurs, you are moving entire processes back and forth. For this to happen, AIX suspends the entire process prior to moving it to paging space. It could then only continue to process when it is swapped back into RAM at a later event. This is not good and you should do everything you can to prevent swapping from occurring, which can cause another condition called thrashing (I'll get into this more later).
As a UNIX administrator, you are probably already aware of some of the concepts of paging and swapping. AIX provides three different modes of paging space allocation: deferred page space allocation, late page space allocation, and early page space allocation. The default policy of AIX is deferred page space allocation. This works by making sure that the allocation of paging space is delayed until the time that it is necessary to page out the page, which ensures that there is no wasted paging space. In fact, when you have a large amount of RAM, you might actually never even use any of your paging space (see Listing 1).
Listing 1. Ensuring that there is no wasted paging space
# lsps -a Page Space Physical Volume Volume Group Size %Used Active Auto Type hd6 hdisk0 rootvg 4096MB 1 yes yes lv
Only one percent of paging space is used in Listing 1.
Let's view how AIX is currently handling paging space allocation (see Listing 2).
Listing 2. Checking how AIX is handling paging space allocation
# vmo -a | grep def defps = 1
Listing 2 illustrates that the default method, deferred page
space allocation, is being used. To disable this policy, you need to set the
parameter to 0. This activates the system to use the late paging space allocation
policy. Late paging space allocation causes paging disk blocks not to be allocated
until its corresponding pages in RAM are touched. This method is usually intended
for environments where optimum performance is more important than reliability. In
the scenario presented here, a program can fail due to the lack of memory. What
about early page space allocation? This policy is usually used if you want to make
certain that processes will not be killed because of low paging conditions. Early
page space allocation preallocates paging space. This is the opposite end of the
spectrum from late paging space allocation. It is used in environments where
reliability rules. The way to turn this on would be to set the PSALLOC
environment variable to early (
You should also be aware of the garbage collection feature first introduced in AIX Version 5.3. This allows you to free up paging-space disk blocks, which allows you to configure less paging space than you would ordinarily need. This feature is only available for the default deferred page space allocation policy.
Monitoring and configuring paging space
In this section, I'll show you how to monitor the paging space on your system. I'll also discuss the various commands used for configuring paging space and other tools that help you work with paging space as a systems administrator.
The simplest way of determining the amount of paging space used on your system is
by running the
lsps command (see
Listing 3. Running the lsps command
# lsps -s Total Paging Space Percent Used 4096MB 1%
You looked earlier at the
-a flag. I prefer using the
-s flag, because the
shows only paging space that is being used while the
command gives you a summary of all paging space allocated, including space
allocated using early page space allocation. Of course, this only applies if the
default method of paging allocation was turned off.
Next on the plate is
2 of this series discussed
vmstat in great detail, which is one of my favorite VMM
monitoring tools. I find that it is the quickest way to determine what is going on
in your system. If there is a lot of paging and thrashing going on, you will find
Let's look at some output shown in Listing 4.
Listing 4. Using vmstat
# vmstat 1 5 System Configuration: lcpu=2 mem=4096MB kthr memory page faults cpu ----- ----------- ------------------------ ------------ ----------- r b avm fre re pi po fr sr cy in sy cs us sy id wa 1 0 166512 627 0 0 1 0 92 0 277 3260 278 3 1 96 0 1 0 166512 623 0 0 1 0 40 0 253 2260 108 2 1 96 1 1 0 166512 627 0 0 0 0 0 0 248 3343 91 0 1 96 2 1 0 166512 627 0 0 0 0 2 0 247 3164 84 0 1 99 0 1 0 166512 627 0 1 0 0 0 0 277 3260 83 2 1 97 0
The columns most meaningful for your purposes here are:
- avm—This column represents the amount of active virtual memory (in 4k pages) you are using, not including file pages.
- fre—This column represents the size of your memory free
list. In most cases, I don't worry when this is small, as AIX loves using every
last drop of memory and does not return it as fast as you might like. This
setting is determined by the minfree parameter of the
vmocommand. At the end of the day, the paging information is more important.
- pi—This column represents the pages paged in from the paging space.
- po—This column represents the pages paged out to the paging space.
As you can see in Listing 4, there is essentially no paging going on in the system.
Listing 5 shows an example of a system that is probably thrashing.
Listing 5. Possible thrashing system
# vmstat 2 3 System Configuration: lcpu=4 mem=4096MB kthr memory page faults cpu ----- ----------- ------------------------ ------------ ----------- r b avm fre re pi po fr sr cy in sy cs us sy id wa 1 2 166512 7 0 57 127 0 929 0 2779 3260 1278 3 30 50 0 20 1 5 166512 12 0 39 129 0 409 0 2538 2260 1108 2 10 30 10 50 1 6 166512 110 0 8 212 0 480 0 2487 3343 991 0 27 33 20 30
How can you tell this? First of all, look at the po column. This signifies
that pages are consistently being moved back and forth between disk and RAM. You
should also see a bottleneck on your system, as the blocked processes and wait
times are abnormally high. The freelist is also lower than it should be. In
looking at the freelist with the
vmo command, you
determined that the number was 120. This means that this number should not be
falling below the 120 mark. Ordinarily, I would say it is not a problem when your
freelist is low but, in this case, it is below where it should be. When this
occurs, it usually signifies that thrashing is going on in your system. A classic
sign of thrashing is when the operating system attempts to release resources by
first warning processes to release paging space and then killing entire processes.
vmo parameters, you can help set the
thresholds when thrashing starts. You can also look at memory usage with either
topas or nmon. Both of these utilities graphically display the paging in a more
user-friendly format (see Listing 6).
Listing 6. Paging displayed graphically using topas
Topas Monitor for host: testbox EVENTS/QUEUES FILE/TTY Sun May 20 11:48:42 2007 Interval: 2 Cswitch 86 Readch 90043 Syscall 1173 Writech 1336 Kernel 0.5 |# | Reads 103 Rawin 1 User 0.0 | | Writes 91 Ttyout 157 Wait 0.0 | | Forks 0 Igets 0 Idle 99.5 |############################| Execs 0 Namei 147 Runqueue 0.0 Dirblk 0 Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue 0.0 en1 1.6 4.0 4.0 0.2 1.4 en2 0.0 0.0 0.0 0.0 0.0 PAGING MEMORY lo0 0.0 0.0 0.0 0.0 0.0 Faults 0 Real,MB 4095 Steals 0 % Comp 16.6 Disk Busy% KBPS TPS KB-Read KB-Writ PgspIn 0 % Noncomp 84.3 hdisk0 0.0 0.0 0.0 0.0 0.0 PgspOut 0 % Client 0.5 hdisk1 0.0 0.0 0.0 0.0 0.0 PageIn 0 hdisk3 0.0 0.0 0.0 0.0 0.0 PageOut 0 PAGING SPACE Sios 0 Size,MB 4096 Name PID CPU% PgSp Owner % Used 0.5 topas 156220 0.2 2.5 root NFS (calls/sec) % Free 99.4 sldf 96772 0.2 0.2 rds ServerV2 0 syncd 12458 0.0 0.6 root ClientV2 0 Press: lrud 9030 0.0 0.0 root ServerV3 0 "h" for help gil 10320 0.0 0.1 root ClientV3 0 "q" to quit
The PAGING column (shown in bold in Listing 6) shows that there is no paging going on at all.
What about maintaining the size of your paging space? You do this with the
swap command (see Listing 7) in
Listing 7. Using the swap command
# swap -l device maj,min total free /dev/hd6 10, 2 4096MB 4093MB
This tells you that you have one swap partition defined. You'll also notice that only 3MB are actually being used. Listing 8 shows what happens if your paging space utilization is too high.
Listing 8. Running out of paging space
# lsps -a Page Space Physical Volume Volume Group Size %Used Active Auto Type hd6 hdisk0 rootvg 4096MB 78 yes yes lv
In this case, your paging space is starting to get dangerously low. It is possible that your system has been up for a very long time. If you are running a database such as Oracle, virtual memory does not get released until you recycle your database. Let's see how long your system has been up (see Listing 9).
Listing 9. Using the uptime command
# uptime 11:58AM up 9 days, 15:50, 23 users, load average: 0.00, 0.03, 0.04
As shown in Listing 9, the system has been up for only nine days. If the paging space utilization has increased to 78 percent in such a short amount of time, you should consider adding more paging space. If you have plenty of space on your system, I would add another partition.
One best practice to keep in mind is to keep your paging spaces at the same size.
In this case, I would add another 4GB of paging space to your rootvg volume. You
can do this with the System Management Interface Tool (SMIT) and use either the
smit mkps and
commands to activate the paging space. Alternatively, you can use the
commands from the command line. If you can, use disks that are least used for
paging areas. Also try not to allocate more than one paging logical volume for
each physical disk. Though some administrators don't mind putting paging space on
external storage, I personally don't like that practice. If you do this and the
external storage is not available on a reboot, your system might crash (depending
upon the amount of space allocated to paging). If you can, spread them across
multiple platters and, of course, make sure they are online by using the
How much paging space do you need on your system? What is the rule of thumb?
First, start with the folks that own your application. The DB2® or Oracle
teams should be able to tell you how much paging space needs to be allocated on
your system from a database perspective. If you are a small shop, you'll have to
do the research on your own. Be careful, though. Database administrators usually
like to request the highest number of everything and might instruct you to double
the amount of paging space as your RAM (the old rule of thumb). Generally
speaking, if my system has greater than 4GB of RAM, I usually like to
create a one-to-one ratio of paging space versus RAM. Monitor your system
frequently after going live. If you see that you are never really approaching 50
percent of paging space utilization, don't add the space. A quick look at the
recent Oracle for AIX documentation (see Resources)
confirms this principle. It states that the recommended initial setting for paging
space be half the size of RAM plus 4GB with an upper limit of 32GB. It recommends monitoring space with the
lsps -a command and not to worry unless the
utilization is over 25 percent on the system. Adding additional space that you
won't use gives you absolutely nothing extra.
I'm often asked how can you tell if a process is using paging space? Take a look
svmon, as shown in Listing 10.
Listing 10. Using svmon
# svmon -P | grep -p 17602 ------------------------------------------------------------------------------- Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage 17602 sendmail 11877 3211 0 11691 N N N
After identifying the PID number, using
drill down to this level. This can help you determine whether or not tuning needs
to be done to your application to either help stop the paging or to tune your
operating system. Do a man on
svmon, as there are many
other purposes to this AIX memory-specific utility.
Tuning with vmo
In this section, I use
vmo to tune paging parameters
that can significantly reduce the amount of paging on your systems. I also discuss
thresholds to change and parameters that can influence your overall scanning
So what can you tune on VMM to cut down on paging? In the first installment of
the series (see Resources) , I discussed the
parameters in great detail, and I'll summarize some of the most important concepts
vmo settings allows you to favor
either working or persistent storage. You want it to favor working storage. The
way to prevent AIX from paging working storage and to utilize the caching from
your database would be to set
maxperm to a high value
(greater than 80) and to make sure the
parameter indicates whether or not the VMM re-page counts should be considered and
what type of memory it should steal. The default setting is 1, so you need to
change it to 0. This is done using the
When you set the parameter to 0, it tells the VMM that you prefer that it steal
only file pages rather than computational pages. This is what you want to do. You
also need to set the
parameters, as shown in Listing 11 below.
Listing 11. Setting the
vmo -p -o minperm%=5 vmo -p -o maxperm%=90 vmo -p -o maxclient%=90
In prior AIX versions, you would tune
strict_maxclient from their default numbers. With
AIX Version 5.3, changing the
is a far more effective way of tuning, as you would prefer AIX file caching not be
used at all. Now let's briefly summarize
maxfree. If the number of pages on your free list falls
minfree parameter, VMM starts to steal pages
until the free list has at least the amount of pages in the maxfree parameter. The
default settings in AIX Version 5.3 usually seem to work (see
Listing 12. Default settings for
# vmo -a | grep free maxfree = 1088 minfree = 960
Let's discuss tuning page space thresholds. As stated earlier, when your
paging space starts becoming very low, it starts to warn offending processes and
then kills them. What thresholds can you change here to influence this activity?
They would be
Npswarn is the threshold that is used to signal the
processes when space is getting low.
Npskill is the
threshold where AIX starts killing processes. If your policy is early page space
allocation, it will not kill the process. If you recall, I discussed earlier that
this was the most reliable method of paging.
is an important threshold because, if this is set to 1, it makes certain that
processes owned by root will not be killed, even when the
npskill threshold is reached.
Further, when a process cannot be forked because of a paging space issue, the
scheduler retries to fork it again up to five times, delaying 10 clock ticks
before each retry. You can change the
to increase or decrease the amount of tries. The parameter used for this is the
pacefork value. Another important parameter you can look at is
lrubucket. Tuning this can reduce the scanning
overhead. Because the page replacement algorithm is always looking for free frames
while it is doing its scanning on systems with a lot of memory, the number of
frames to scan can be significant. Increasing the value decreases the amount of
buckets that need to be scanned. This can help performance.
13 uses the
-a option to display the values for
Listing 13. Displaying the value for
# vmo -a | grep lru lru_file_repage = 1 lru_poll_interval = 0 lrubucket = 131072 (this is in 4 KB frames)
To increase the default value from 512MB to 1GB, use
# vmo -o lrubucket=262144.
And that's how you can significantly reduce paging on your AIX system using
Part 3 of this series looked at some of the tools that are available to you in capturing data for swap analysis. You used some system administration commands to display and configure swap on your system, and learned about paging and swapping and the various methods of paging that are available on AIX. You also reviewed some best practices when configuring paging space on your systems. Finally, you studied specific methods of tuning your VMM specific to handle paging and swapping. Parts 1 and 2 of this series went over the VMM in great detail and covered troubleshooting memory bottlenecks. You used various tools to help you monitor your systems for both short-term analysis and long-term trending. You also learned all about the general tuning methodology and the importance of monitoring systems prior to bottlenecks occurring. This enables you to establish a baseline while your system is healthy so that you can practice some of the methods discussed in this series, which include tuning your memory subsystems. Just make sure you test them on your development or test environments prior to deploying any changes to production.
- Use RSS
feed to request notification for the upcoming articles in this series:
- Optimizing AIX 5L™ performance: Tuning disk performance
- Optimizing AIX 5L performance: Tuning your memory settings
- Optimizing AIX 5L performance: Monitoring your CPU
- Check out other parts in each series:
- "System Administration Toolkit: Swap space management and tricks" (Martin Brown, developerWorks, October 2006): Learn how to monitor your system to determine an effective swap space figure.
- Get information on Paging versus swapping from the UNIX for Advanced Users notes page published by Unix Workstation Support Group of Indiana University.
- Read the Paging space entry on the IBM AIX 5L Wiki.
- Administering Oracle Database on AIX: Get more information on memory and paging from the Administering Oracle Database on AIX section of the Oracle Database Administrator's Reference.
- AIX memory affinity support: Visit the IBM System p and AIX InfoCenter to learn more about AIX memory affinity support.
- Tuning IBM AIX5L for an Oracle Database: This document discusses performance analysis and tuning for different types of Oracle workloads and AIX 5L setup configurations.
- IBM Redbooks: Database Performance Tuning on AIX is designed to help system designers, system administrators, and database administrators design, size, implement, maintain, monitor, and tune a Relational Database Management System (RDMBS) for optimal performance on AIX.
- Power Architecture: High-Performance Architecture with a History: Read this white paper.
- "Power to the People; A history of chip making at IBM" (developerWorks, December 2005): This article covers the IBM power architecture.
- "Processor Affinity on AIX" (developerWorks, November 2006): Using process affinity settings to bind or unbind threads can help you find the root cause of troublesome hang or deadlock problems. Read this article to learn how to use processor affinity to restrict a process and run it only on a specified central processing unit (CPU).
- "CPU Monitoring and Tuning" (March, 2002): Learn how standard AIX tools can help you determine CPU bottlenecks.
- IBM Redbooks: AIX 5L Practical Performance Tools and Tuning Guide is a comprehensive guide about performance monitoring and tuning tools that are provided with AIX 5L Version 5.3.
- "AIX 5L Version 5.3: What's in it for you?" (Shiv Dutta, developerWorks, June 2005): Learn what features you can benefit from in AIX 5L Version 5.3.
- Operating System and Device Management: This document from IBM provides users and system administrators with complete information that can affect your selection of options when performing such tasks as backing up and restoring the system, managing physical and logical storage, and sizing appropriate paging space.
- IBM Redbooks: The AIX 5L Differences Guide Version 5.3 Edition focuses on the differences introduced in AIX 5L Version 5.3 when compared to AIX 5L Version 5.2.
- Check out other articles and tutorials written by Ken Milberg:
- Popular content: See what AIX and UNIX content your peers find interesting.
- AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
- New to AIX and UNIX?: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.
- AIX 5L Wiki: Discover a collaborative environment for technical information related to AIX.
- Search the AIX and UNIX library by topic:
- Safari bookstore: Visit this e-reference library to find specific technical resources.
- developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
- Podcasts: Tune in and catch up with IBM technical experts.
- Future Tech: Visit Future Tech's site to learn more about their latest offerings.
Get products and technologies
- IBM trial software: Build your next development project with software for download directly from developerWorks.
- Participate in the developerWorks blogs and get involved in the developerWorks community.
- Participate in the AIX and UNIX forums:
Dig deeper into AIX and Unix on developerWorks
Experiment with new directions in software development.
Read and subscribe for the best and latest technical info to help you deal with your development challenges.
Software development in the cloud. Register today and get free private projects through 2014.
Evaluate IBM software and solutions, and transform challenges into opportunities.