Introduction

As a systems administrator, you should already be familiar with the basics of memory, such as the differences between physical and virtual memory. What you might not fully understand is how the Virtual Memory Manager (VMM) works in AIX® and how it relates to performance tuning. Further, some of the tuning commands and parameters have changed in recent years and, if you have been away from AIX for a while, you might find that some of the commands you're accustomed to using do not even work anymore. This article discusses the AIX VMM in detail and the tuning commands that you need to use to tune the VMM. I introduce some of the monitoring tools that you can use to put you in position to tune your systems and outline some of the more important AIX Version 5.3 memory management enhancements. Implementation of these enhancements, as they apply to your systems environment, can optimize memory performance on your box.

While you might find tuning your memory to be more difficult than other subsystems, the reward is often greater. Depending on the type of system you are running, there might also be specific tuning recommendations that should be set on your systems. To help validate the findings, I use a specific example and discuss some best practices for setting these parameters. Tuning one or two parameters on the fly, in some cases, can make a significant difference in the overall performance of your system.

One area that does not change, regardless of which subsystem you are looking to tune, is tuning systems—you should always think of it as an ongoing process. The best time to start monitoring your systems is when you have first put your system in production and it is running well, rather than when your users are screaming about slow performance. You can never really be sure if they are having a problem without a real baseline of what the system looked like when it was behaving normally. Further, only one change should be made at a time, and data should be captured and analyzed as quickly as possible after that change to determine what difference, if any, the change really made.

Memory overview

This section gives an overview of memory as it relates to AIX. I discuss how AIX uses virtual memory to address more memory than is physically on your system. I also explain how the VMM actually works and how it services requests.

Any discussion of memory and AIX must start with a description of the VMM. AIX newbies are sometimes surprised to hear that the VMM services all memory requests from the system, not just virtual memory. When RAM is accessed, the VMM needs to allocate space, even when there is plenty of physical memory left on the system. It implements a process of early allocation of paging space. Using this method, the VMM plays a vital role in helping manage real memory, not only virtual memory. Here is how it works. In AIX, all virtual memory segments are partitioned into pages. In AIX, the default size per page is 4KB. Allocated pages can be either RAM or paging space (virtual memory stored on disk). VMM also maintains what is referred to as a free list, which is defined as unallocated page frames. These are used to satisfy page faults. There are usually a very small amount of unallocated pages (which you configure) that the VMM uses to free up space and reassign the page frames to. The virtual memory pages whose page frames are to be reassigned are selected using the VMM's page replacement algorithm. This paging algorithm determines which virtual memory pages currently in RAM ultimately have their page frames brought back to the free list. AIX uses all available memory, except that which is configured to be unallocated and known as the free list.

To reiterate, the purpose of VMM is to manage the allocation of both RAM and virtual pages. From here, you can determine that its objectives are to help minimize both the response time of page faults and to decrease the use of virtual memory where it can. Obviously, given the choice between RAM and paging space, most people would prefer to use physical memory, if the RAM is available. What VMM also does is classify virtual memory segments into two distinct categories. The categories are working segments using computational memory and persistent segments using file memory. It is extremely important to understand the distinction between the two, as this helps you tune the systems to their optimum capabilities.

Computational memory

Computational memory is used while your processes are actually working on computing information. These working segments are temporary (transitory) and only exist up until the time a process terminates or the page is stolen. They have no real permanent disk storage location. When a process terminates, both the physical and paging spaces are released in many cases. When there is a large spike in available pages, you can actually see this happening while monitoring your system. In the VMM world when free physical memory starts getting low, programs that have not been used recently are moved from RAM to paging space to help release physical memory for more real work.

File memory

File memory (unlike computational memory) uses persistent segments and has a permanent storage location on the disk. Data files or executable programs are mapped to persistent segments rather than working segments. The data files can relate to filesystems, such as JFS, JFS2, or NFS. They remain in memory until the time that the file is unmounted, a page is stolen, or a file is unlinked. After the data file is copied into RAM, VMM controls when these pages are overwritten or used to store other data. Given the alternative, most people would much rather have file memory paged to disk rather than computational memory.

When a process references a page which is on disk, it must be paged, which could cause other pages to page out again. VMM is constantly lurking and working in the background trying to steal frames that have not been recently referenced, using the page replacement algorithm I discussed earlier. It also helps detect thrashing, which can occur when memory is extremely low and pages are constantly being paged in and out to support processing. VMM actually has a memory load control algorithm, which can detect if the system is thrashing and actually tries to remedy the situation. Unabashed thrashing can literally cause a system to come to a standstill, as the kernel becomes too concerned with making room for pages than to actually do anything productive.

Tuning

Let's examine the tools that can allow you to tune the VMM to optimize performance for your system. I give an example of an environment where you want to tune parameters using a certain type of methodology. I also go over some of the key parameters you need to be aware of.

Prior to AIX 5L™, you would have used vmtune to tune your VMM system. While the vmo command came around in AIX Version 5.2, vmtune actually hung around until AIX Version 5.3. With AIX Version 5.3, you have to make a clean break. Most of the actual parameters are the same, though there are some fundamental changes that you need to be made aware of.

Let's discuss one important change in AIX 5L with respect to page frames. Starting with the POWER4 processor, AIX supported up to 16MB pages sizes. The POWER5 chip actually supports four virtual memory page sizes: 4KB, 64KB, 16MB, and 16GB. With a simple vmo change here, you can actually tune the system to provide for large page usage, which can improve system performance substantially in very memory-intensive applications. The performance improvement is due to the reduction of Translation Lookaside Buffer (TLB) misses, which occurs because the TLB can now map to a much larger virtual memory range. For example, an Oracle database, either Online Transaction Processing (OLTP) or a Data Warehouse application, can benefit when using large pages. This is because Oracle uses a lot of virtual memory, particularly with respect to its System Global Area (SGA). The example I use here is an Oracle database server running on a p550 LPAR running AIX Version 5.3. The system is used for both OLTP and a data warehouse.

The command in Listing 1 allocates 16777216 bytes to provide large pages with 128 actual large pages.

                
# vmo -r -o lgpg_size=16777216 lgpg_regions=128

The most important vmo settings are minperm and maxperm. Setting these parameters determine the appropriate value for your system to ensure that it is tuned to either favor computational memory or file memory. In most cases, you do not want to page working segments, as doing so causes your system to page unnecessarily and decrease performance. The way it worked in the past was actually quite simple: If your file pages (numperm%) were greater than the actual maxperm%, then the page replacement would only steal file pages. When it falls below minperm, it could steal both file and computational pages. If it was between both, then it would only steal file pages unless the number of file repages was greater then the amount of computational pages. Another way of looking at this is if your numperm is greater than the maxperm, then you would start to steal from persistent storage. Based on this methodology, the old approach to tuning your minperm and maxperm parameters was to bring maxperm to a low amount (for example, <20) and minperm to <=10. This is how you would have normally tuned your database server.

That has all changed. The new approach sets maxperm to a high value (for example, >80) and makes sure the lru_file_repage parameter is set to 0. lru_file_repage was first introduced in AIX Version 5.2 with ML4 and on ML1 of AIX Version 5.3. This parameter indicates whether or not the VMM re-page counts should be considered and what type of memory it should steal. The default setting is 1, so you need to change it. When you set the parameter to 0, it tells VMM that you prefer that it steal only file pages rather than computational pages. This can change if your numperm is less than the minperm or greater than the maxperm, which is why you would now want maxperm to be high and minperm to be low. Let's not lose sight of the fact that the primary reason you need this value tuned is because you want to protect the computational memory. Getting back to the example, Oracle uses its own cache, and using AIX file caching for this purpose only causes confusion, so you want to stop it. If you were to reduce the maxperm in this scenario, then you would now make the mistake of stopping the application caching programs that are running.

Listing 2 sets these critical tuning parameters.

                
vmo -p -o minperm%=5
vmo -p -o maxperm%=90
vmo -p -o maxclient%=90

Although you used to have to change these parameters, you now leave strict_maxperm and strict_maxclient at their default numbers. If strict_maxperm were changed to 1, it would place a hard limit on the amount of memory that could be used for persistent file cache. This is done by making the maxperm value the upper limit for the cache. These days it is unnecessary, because changing the lru_file_repage parameter is a far more effective way of tuning, as you would prefer not to use AIX file caching.

Two other important parameters worth noting here are minfree and maxfree. If the number of pages on your free list falls below the minfree parameter, VMM starts to steal pages (just to add to the free list), which is not good. It continues to do this until the free list has at least the number of pages in the maxfree parameter.

In older versions of AIX when the default minfree was set at 120, you would commonly see your free list at 120 or lower, which led to more paging than was necessary, and worse, threads needing free frames were actually getting blocked because the value would be so low. To address this issue, the default values of minfree and maxfree were bumped up in AIX Version 5.3 to 960 and 1088, respectively. If you are running AIX Version 5.2 or lower, I would recommend these settings, which you can manually change using the commands in Listing 3.

                
vmo -p -o minfree=960
vmo -p -o maxfree=1088

AIX Version 5.3 memory changes and improvements

Let's examine some of the recent changes made in AIX Version 5.3 that relate to memory. Appropriate usage of the enhanced functionality of memory management in AIX Version 5.3 can certainly impact your ability to effectively tune your systems. Among other areas, let's discuss page space scrubbing, dynamic xmalloc, memory affinity, and Watson malloc.

• Page space scrubbing: Under certain conditions, this feature allows the system to free up a paging-space disk block so that you do not have to configure as much paging space for a given workload. This feature is available only when using the deferred page space allocation policy.
• Dynamic xmalloc debug (xmdbg): This feature improves system reliability by improving the overall diagnostics capabilities of the allocation of memory. It permits clients to vary the amount of diagnostic memory collected without a reboot.
• Watson malloc: This new malloc subysystem handles small requests more quickly over the default implementation of memory fragmentation. It does so, with very little wasted memory in comparison to the default traditional Yorktown method. Some new features were also added to the debugging facility, to help in your ability to repair memory allocation problems.
• Improved multiple page size support: As discussed earlier in the Tuning section, there are now four different page sizes available. These sizes come in the following flavors:
  • 4KB
  • 64KB
  • 16MB
  • 16GB
• VMM monitoring tools: Monitoring tools, such as vmstat and svmon, have been tuned to allow for the virtualization capabilities of the POWER5 architecture. They also now support some of the improvements shown in this section. In Part 2 of this series, I'll discuss these changes in detail as you dive into the memory monitoring tools of AIX to determine bottlenecks and analyze historical trends and data.
• Memory affinity: With AIX Version 5.3, you can no longer disable memory affinity, as was the case with AIX Version 5.2. What this does is provides the capability to allocate memory for a process from the memory module which contains the processor that caused the page fault. This is done so that a processor accesses the memory attached to its own module before it looks at other modules to increase performance. To disable memory affinity support on AIX Version 5.2, you can use the following vmo command: vmo -o memory_affinity=0

You should also note that you don't have to save any of your tunables in rc.tune anymore. The new way is now using /etc/tunables, which is definitely an improvement. While this feature was introduced in AIX Version 5.2, not Version 5.3, it is worth noting here.

Summary

As discussed, before you tune or even start monitoring AIX, you must establish a baseline. After you tune, you must capture data and analyze the results of your changes. Without this type of information, you never really understand the true impact of tuning. In Part 1 of this series, you learned all about the VMM of AIX and how it works. You also tuned an Oracle system to optimize utilization of the memory subsystem. You examined some important kernel parameters, what they do, and how to tune them. While doing so, you studied some of the fundamental changes made in recent years regarding the approach to certain parameters and improvements made in AIX Version 5.3 as they relate to memory.

Part 2 focuses much more on the detail of systems monitoring for the purposes of determining memory bottlenecks, along with analyzing trends and results. Part 3 focuses primarily on swap space and other methods to tune your VMM to maximize performance.

Resources

Learn

• Use RSS feed to request notification for the upcoming articles in this series:
  • Optimizing AIX 5L™ performance: Tuning disk performance
  • Optimizing AIX 5L performance: Tuning your memory settings
  • Optimizing AIX 5L performance: Monitoring your CPU
  (Find out more about RSS feeds of developerWorks content.)
  
• Check out other parts in each series:
  • Optimizing AIX 5L performance: Tuning disk performance
  • Optimizing AIX 5L performance: Tuning your memory settings
  • Optimizing AIX 5L performance: Monitoring your CPU
  
  
• Virtual Memory Management -- Tuning Parameter lru_file_repage: This presentation provides details on using the lru_file_repage parameter.
  
  
• AIX memory affinity support: Learn about AIX memory from the IBM System p™ and AIX InfoCenter.
  
  
• Tuning IBM AIX 5.3 and AIX 6.1 for Oracle Database : Find out about performance analysis and tuning for Oracle workloads on IBM AIX 5.3 and IBM AIX 6.1 configurations on IBM POWER systems when running an Oracle database (including simultaneous multithreading and IBM Micro-Partitioning technology).
  
  
• IBM Redbooks: See how Database Performance Tuning on AIX is designed to help system designers, system administrators, and database administrators design, size, implement, maintain, monitor, and tune a Relational Database Management System (RDMBS) for optimal performance on AIX.
  
  
• Power Architecture: High-Performance Architecture with a History: Read this white paper.
  
  
• "Power to the People" (developerWorks, May 2004): Read this article for a history of chip making at IBM.
  
  
• "Processor Affinity on AIX" (developerWorks, November 2006): Using process affinity settings to bind or unbind threads can help you find the root cause of troublesome hang or deadlock problems. Read this article to learn how to use processor affinity to restrict a process and run it only on a specified CPU.
  
  
• "CPU Monitoring and Tuning" (March, 2002): Read this article to learn how standard AIX tools can help you determine CPU bottlenecks.
  
  
• "AIX 5L Version 5.3: What's in it for you?" (developerWorks, June 2005): Learn what features you can benefit from in AIX 5L Version 5.3.
  
  
• Operating System and Device Management: This document from IBM provides users and system administrators with complete information that can affect your selection of options when performing such tasks as backing up and restoring the system, managing physical and logical storage, and sizing appropriate paging space.
  
  
• "nmon performance: A free tool to analyze AIX and Linux® performance" (developerWorks, February 2006): This free tool gives you a huge amount of information all on one screen.
  
  
• "nmon analyser—A free tool to produce AIX performance reports" (developerWorks, April 2006): Read this article to learn how to produce a wealth of report-ready graphs from nmon output.
  
  
• IBM Redbooks: For a comprehensive guide about the performance monitoring and tuning tools that are provided with AIX 5L Version 5.3, read the AIX 5L Practical Performance Tools and Tuning Guide.
  
  
• The AIX 5L Differences Guide Version 5.3 Edition (developerWorks, December 2004): Read about the differences introduced in AIX 5L Version 5.3 when compared to AIX 5L Version 5.2.
  
  
• Check out other articles and tutorials written by Ken Milberg:
  • Across developerWorks and IBM
  
  
• Popular content: See what AIX and UNIX content your peers find interesting.
  
  
• AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
  
  
• New to AIX and UNIX?: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.
  
  
• AIX 5L Wiki: Discover a collaborative environment for technical information related to AIX.
  
  
• Search the AIX and UNIX library by topic:
  • System administration
  • Application development
  • Performance
  • Porting
  • Security
  • Tips
  • Tools and utilities
  • Java™ technology
  • Linux
  • Open source
  
  
• Safari bookstore: Visit this e-reference library to find specific technical resources.
  
  
• developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
  
  
• Podcasts: Tune in and catch up with IBM technical experts.
  
  
• Future Tech: Visit Future Tech's site to learn more about their latest offerings.
  
  

Get products and technologies

• IBM trial software: Build your next development project with software for download directly from developerWorks.
  
  

Discuss

• Participate in the developerWorks blogs and get involved in the developerWorks community.
  
  
• Participate in the AIX and UNIX forums:
  • AIX 5L—technical forum
  • AIX for Developers Forum
  • Cluster Systems Management
  • IBM Support Assistant
  • Performance Tools—technical
  • Virtualization—technical
  • More AIX and UNIX forums
  
  

About the author

Rate this article

Comments

Back to top

Table of contents

• Introduction
• Memory overview
• Tuning
• AIX Version 5.3 memory changes and improvements
• Summary
• Resources
• About the author
• Comments

Dig deeper into AIX and UNIX on developerWorks

IBM SmartCloud trial. No charge.

Unleash the power of hybrid cloud computing today!

---

Public and private solutions in one trial.

Special offers