Optimizing AIX 5L performance: Tuning your memory settings, Part 1

Memory overview and tuning memory parameters in AIX Version 5.3

This three-part series on memory tuning dives right into tuning parameters, focusing on the many challenges and the various best practices of optimizing memory performance, and it also discusses some improvements in AIX® Version 5.3. While memory tuning might be more difficult to implement than Central Processing Unit (CPU) tuning, it certainly is no less important. You can do more to tune memory on an AIX server than any other subsystem. Changing some memory parameters on your system can increase performance dramatically, particularly when these parameters are not optimized for the environment which you are running. Part 1 of this series provides an overview of memory on AIX, including a discussion of virtual memory and the Virtual Memory Manager (VMM).

Ken Milberg, Future Tech UNIX Consultant, Technology Writer, and Site Expert, Future Tech

Ken Milberg is a Technology Writer and Site Expert for techtarget.com and provides Linux technical information and support at searchopensource.com. He is also a writer and technical editor for IBM Systems Magazine, Open Edition. Ken holds a bachelor's degree in computer and information science and a master's degree in technology management from the University of Maryland. He is the founder and group leader of the NY Metro POWER-AIX/Linux Users Group. Through the years, he has worked for both large and small organizations and has held diverse positions from CIO to Senior AIX Engineer. Today, he works for Future Tech, a Long Island-based IBM business partner. Ken is a PMI certified Project Management Professional (PMP), an IBM Certified Advanced Technical Expert (CATE, IBM System p5 2006), and a Solaris Certified Network Administrator (SCNA). You can contact him at kmilberg@gmail.com.



15 May 2007

Also available in Chinese Russian

Introduction

As a systems administrator, you should already be familiar with the basics of memory, such as the differences between physical and virtual memory. What you might not fully understand is how the Virtual Memory Manager (VMM) works in AIX® and how it relates to performance tuning. Further, some of the tuning commands and parameters have changed in recent years and, if you have been away from AIX for a while, you might find that some of the commands you're accustomed to using do not even work anymore. This article discusses the AIX VMM in detail and the tuning commands that you need to use to tune the VMM. I introduce some of the monitoring tools that you can use to put you in position to tune your systems and outline some of the more important AIX Version 5.3 memory management enhancements. Implementation of these enhancements, as they apply to your systems environment, can optimize memory performance on your box.

While you might find tuning your memory to be more difficult than other subsystems, the reward is often greater. Depending on the type of system you are running, there might also be specific tuning recommendations that should be set on your systems. To help validate the findings, I use a specific example and discuss some best practices for setting these parameters. Tuning one or two parameters on the fly, in some cases, can make a significant difference in the overall performance of your system.

One area that does not change, regardless of which subsystem you are looking to tune, is tuning systems—you should always think of it as an ongoing process. The best time to start monitoring your systems is when you have first put your system in production and it is running well, rather than when your users are screaming about slow performance. You can never really be sure if they are having a problem without a real baseline of what the system looked like when it was behaving normally. Further, only one change should be made at a time, and data should be captured and analyzed as quickly as possible after that change to determine what difference, if any, the change really made.

Memory overview

This section gives an overview of memory as it relates to AIX. I discuss how AIX uses virtual memory to address more memory than is physically on your system. I also explain how the VMM actually works and how it services requests.

Any discussion of memory and AIX must start with a description of the VMM. AIX newbies are sometimes surprised to hear that the VMM services all memory requests from the system, not just virtual memory. When RAM is accessed, the VMM needs to allocate space, even when there is plenty of physical memory left on the system. It implements a process of early allocation of paging space. Using this method, the VMM plays a vital role in helping manage real memory, not only virtual memory. Here is how it works. In AIX, all virtual memory segments are partitioned into pages. In AIX, the default size per page is 4KB. Allocated pages can be either RAM or paging space (virtual memory stored on disk). VMM also maintains what is referred to as a free list, which is defined as unallocated page frames. These are used to satisfy page faults. There are usually a very small amount of unallocated pages (which you configure) that the VMM uses to free up space and reassign the page frames to. The virtual memory pages whose page frames are to be reassigned are selected using the VMM's page replacement algorithm. This paging algorithm determines which virtual memory pages currently in RAM ultimately have their page frames brought back to the free list. AIX uses all available memory, except that which is configured to be unallocated and known as the free list.

To reiterate, the purpose of VMM is to manage the allocation of both RAM and virtual pages. From here, you can determine that its objectives are to help minimize both the response time of page faults and to decrease the use of virtual memory where it can. Obviously, given the choice between RAM and paging space, most people would prefer to use physical memory, if the RAM is available. What VMM also does is classify virtual memory segments into two distinct categories. The categories are working segments using computational memory and persistent segments using file memory. It is extremely important to understand the distinction between the two, as this helps you tune the systems to their optimum capabilities.

Computational memory

Computational memory is used while your processes are actually working on computing information. These working segments are temporary (transitory) and only exist up until the time a process terminates or the page is stolen. They have no real permanent disk storage location. When a process terminates, both the physical and paging spaces are released in many cases. When there is a large spike in available pages, you can actually see this happening while monitoring your system. In the VMM world when free physical memory starts getting low, programs that have not been used recently are moved from RAM to paging space to help release physical memory for more real work.

File memory

File memory (unlike computational memory) uses persistent segments and has a permanent storage location on the disk. Data files or executable programs are mapped to persistent segments rather than working segments. The data files can relate to filesystems, such as JFS, JFS2, or NFS. They remain in memory until the time that the file is unmounted, a page is stolen, or a file is unlinked. After the data file is copied into RAM, VMM controls when these pages are overwritten or used to store other data. Given the alternative, most people would much rather have file memory paged to disk rather than computational memory.

When a process references a page which is on disk, it must be paged, which could cause other pages to page out again. VMM is constantly lurking and working in the background trying to steal frames that have not been recently referenced, using the page replacement algorithm I discussed earlier. It also helps detect thrashing, which can occur when memory is extremely low and pages are constantly being paged in and out to support processing. VMM actually has a memory load control algorithm, which can detect if the system is thrashing and actually tries to remedy the situation. Unabashed thrashing can literally cause a system to come to a standstill, as the kernel becomes too concerned with making room for pages than to actually do anything productive.

Tuning

Let's examine the tools that can allow you to tune the VMM to optimize performance for your system. I give an example of an environment where you want to tune parameters using a certain type of methodology. I also go over some of the key parameters you need to be aware of.

Prior to AIX 5L™, you would have used vmtune to tune your VMM system. While the vmo command came around in AIX Version 5.2, vmtune actually hung around until AIX Version 5.3. With AIX Version 5.3, you have to make a clean break. Most of the actual parameters are the same, though there are some fundamental changes that you need to be made aware of.

Let's discuss one important change in AIX 5L with respect to page frames. Starting with the POWER4 processor, AIX supported up to 16MB pages sizes. The POWER5 chip actually supports four virtual memory page sizes: 4KB, 64KB, 16MB, and 16GB. With a simple vmo change here, you can actually tune the system to provide for large page usage, which can improve system performance substantially in very memory-intensive applications. The performance improvement is due to the reduction of Translation Lookaside Buffer (TLB) misses, which occurs because the TLB can now map to a much larger virtual memory range. For example, an Oracle database, either Online Transaction Processing (OLTP) or a Data Warehouse application, can benefit when using large pages. This is because Oracle uses a lot of virtual memory, particularly with respect to its System Global Area (SGA). The example I use here is an Oracle database server running on a p550 LPAR running AIX Version 5.3. The system is used for both OLTP and a data warehouse.

The command in Listing 1 allocates 16777216 bytes to provide large pages with 128 actual large pages.

Listing 1. Allocating bytes
# vmo -r -o lgpg_size=16777216 lgpg_regions=128

The most important vmo settings are minperm and maxperm. Setting these parameters determine the appropriate value for your system to ensure that it is tuned to either favor computational memory or file memory. In most cases, you do not want to page working segments, as doing so causes your system to page unnecessarily and decrease performance. The way it worked in the past was actually quite simple: If your file pages (numperm%) were greater than the actual maxperm%, then the page replacement would only steal file pages. When it falls below minperm, it could steal both file and computational pages. If it was between both, then it would only steal file pages unless the number of file repages was greater then the amount of computational pages. Another way of looking at this is if your numperm is greater than the maxperm, then you would start to steal from persistent storage. Based on this methodology, the old approach to tuning your minperm and maxperm parameters was to bring maxperm to a low amount (for example, <20) and minperm to <=10. This is how you would have normally tuned your database server.

That has all changed. The new approach sets maxperm to a high value (for example, >80) and makes sure the lru_file_repage parameter is set to 0. lru_file_repage was first introduced in AIX Version 5.2 with ML4 and on ML1 of AIX Version 5.3. This parameter indicates whether or not the VMM re-page counts should be considered and what type of memory it should steal. The default setting is 1, so you need to change it. When you set the parameter to 0, it tells VMM that you prefer that it steal only file pages rather than computational pages. This can change if your numperm is less than the minperm or greater than the maxperm, which is why you would now want maxperm to be high and minperm to be low. Let's not lose sight of the fact that the primary reason you need this value tuned is because you want to protect the computational memory. Getting back to the example, Oracle uses its own cache, and using AIX file caching for this purpose only causes confusion, so you want to stop it. If you were to reduce the maxperm in this scenario, then you would now make the mistake of stopping the application caching programs that are running.

Listing 2 sets these critical tuning parameters.

Listing 2. Setting tuning parameters
vmo -p -o minperm%=5
vmo -p -o maxperm%=90
vmo -p -o maxclient%=90

Although you used to have to change these parameters, you now leave strict_maxperm and strict_maxclient at their default numbers. If strict_maxperm were changed to 1, it would place a hard limit on the amount of memory that could be used for persistent file cache. This is done by making the maxperm value the upper limit for the cache. These days it is unnecessary, because changing the lru_file_repage parameter is a far more effective way of tuning, as you would prefer not to use AIX file caching.

Two other important parameters worth noting here are minfree and maxfree. If the number of pages on your free list falls below the minfree parameter, VMM starts to steal pages (just to add to the free list), which is not good. It continues to do this until the free list has at least the number of pages in the maxfree parameter.

In older versions of AIX when the default minfree was set at 120, you would commonly see your free list at 120 or lower, which led to more paging than was necessary, and worse, threads needing free frames were actually getting blocked because the value would be so low. To address this issue, the default values of minfree and maxfree were bumped up in AIX Version 5.3 to 960 and 1088, respectively. If you are running AIX Version 5.2 or lower, I would recommend these settings, which you can manually change using the commands in Listing 3.

Listing 3. Setting the minfree and maxfree parameters manually
vmo -p -o minfree=960
vmo -p -o maxfree=1088

AIX Version 5.3 memory changes and improvements

Let's examine some of the recent changes made in AIX Version 5.3 that relate to memory. Appropriate usage of the enhanced functionality of memory management in AIX Version 5.3 can certainly impact your ability to effectively tune your systems. Among other areas, let's discuss page space scrubbing, dynamic xmalloc, memory affinity, and Watson malloc.

  • Page space scrubbing: Under certain conditions, this feature allows the system to free up a paging-space disk block so that you do not have to configure as much paging space for a given workload. This feature is available only when using the deferred page space allocation policy.
  • Dynamic xmalloc debug (xmdbg): This feature improves system reliability by improving the overall diagnostics capabilities of the allocation of memory. It permits clients to vary the amount of diagnostic memory collected without a reboot.
  • Watson malloc: This new malloc subysystem handles small requests more quickly over the default implementation of memory fragmentation. It does so, with very little wasted memory in comparison to the default traditional Yorktown method. Some new features were also added to the debugging facility, to help in your ability to repair memory allocation problems.
  • Improved multiple page size support: As discussed earlier in the Tuning section, there are now four different page sizes available. These sizes come in the following flavors:
    • 4KB
    • 64KB
    • 16MB
    • 16GB
  • VMM monitoring tools: Monitoring tools, such as vmstat and svmon, have been tuned to allow for the virtualization capabilities of the POWER5 architecture. They also now support some of the improvements shown in this section. In Part 2 of this series, I'll discuss these changes in detail as you dive into the memory monitoring tools of AIX to determine bottlenecks and analyze historical trends and data.
  • Memory affinity: With AIX Version 5.3, you can no longer disable memory affinity, as was the case with AIX Version 5.2. What this does is provides the capability to allocate memory for a process from the memory module which contains the processor that caused the page fault. This is done so that a processor accesses the memory attached to its own module before it looks at other modules to increase performance. To disable memory affinity support on AIX Version 5.2, you can use the following vmo command: vmo -o memory_affinity=0

You should also note that you don't have to save any of your tunables in rc.tune anymore. The new way is now using /etc/tunables, which is definitely an improvement. While this feature was introduced in AIX Version 5.2, not Version 5.3, it is worth noting here.

Summary

As discussed, before you tune or even start monitoring AIX, you must establish a baseline. After you tune, you must capture data and analyze the results of your changes. Without this type of information, you never really understand the true impact of tuning. In Part 1 of this series, you learned all about the VMM of AIX and how it works. You also tuned an Oracle system to optimize utilization of the memory subsystem. You examined some important kernel parameters, what they do, and how to tune them. While doing so, you studied some of the fundamental changes made in recent years regarding the approach to certain parameters and improvements made in AIX Version 5.3 as they relate to memory.

Part 2 focuses much more on the detail of systems monitoring for the purposes of determining memory bottlenecks, along with analyzing trends and results. Part 3 focuses primarily on swap space and other methods to tune your VMM to maximize performance.

Resources

Learn

Get products and technologies

  • IBM trial software: Build your next development project with software for download directly from developerWorks.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=219985
ArticleTitle=Optimizing AIX 5L performance: Tuning your memory settings, Part 1
publish-date=05152007