Active Memory Expansion is an innovative AIX and POWER technology that allows the effective maximum memory capacity to be much larger than the true physical memory for AIX partitions.

AME relies on compression of in-memory data to increase the amount of data held in memory, at near memory access times, and thus expand the effective memory capacity of a server. The in-memory data compression is managed by the AIX operating system, and this compression is transparent to applications, and user processes.  You do not need to check whether your application is certified for AME - it is hidden within the AIX kernel.

Active Memory Expansion is configurable on a per-logical partition (LPAR) basis, so it can be activated on one, a few or all AIX LPARs of a server.  AME is a licensed program product (LPP) and comes at a price to have the capability added to a specific POWER server.

But what about the cost?

When first released AME seemed a little expensive. In 2010 on POWER7 processor-based servers a tricky calculation had to be made: purchase more memory or the purchase AME.  AME was a good deal, especially on larger POWER7 processor-based servers like the Power 770 and 780.  In 2020, the price of AME is the same but the amount of RAM purchased in every server has risen up enormously.  I would estimate by a factor of 10 in 10 years. The extra memory is used to hold more data to feed the faster processors.  The cost per extra effective memory makes AME good value as it makes a huge difference to effective server memory at a low cost.

To give you an idea of the AME cost per server, the List Price 1900 Euro Dollar Pounds (deliberately vague currency). Once enabled for the server that is available for all AIX LPARs on that server.  The cost of 512 GB is something like 24,000 Euro Dollar Pounds.  So with an expansion factor of 2.0 that 1900 for AME just saved you 22,000 units of currency and you effectively have 1024 GB of RAM.  Even with a conservative expansion factor of 1.5 then you are saving 10,000 units of currency.

Contact your local IBM representative for a real price.

The principles (assuming you already understand a RAM disk - memory being used as a fast disk): 

Now use the RAM disks for paging out memory and paging it back in memory on demand

Next, add compression of the pages as they are get paged out and expanded when pages back to regular memory.

The result is higher numbers of pages in memory.

The barrier between regular pages and compressed pages can be moved depending on the compression ratio and Expansion Factor target.

Compressing anduncompressing was originally performance using the main POWER processor but with POWER8 and POWER9, a co-processor accelerator is used to performance this far faster.  Running the AME algorithm takes some compute time so it's a balance of CPU time against more effective memory.

When Active Memory Expansion is enabled for an LPAR, the operating system compresses a portion of the LPAR's memory and leaves the remaining portion of memory decompressed and used to running processes access to code and data. AME results in memory effectively being broken up into two pools - a compressed pool and a decompressed pool. The AIX operating system dynamically varies the amount of memory that is compressed based on the current workload and the configuration of the LPAR.

When a process needs access to a memory page that is in the compressed pool, it causes a page interrupt and the AIX kernel decompresses the page and adds to the decompressed pool. Following this action, the process is allowed to continue to access the decompressed memory page as normal.  Mean while, the AIX kernel is compressing memory pages that remain unused for a time.  This process balances the memory use of compressed and decompressed memory pools.

Because Active Memory Expansion relies on memory compression, some additional CPU cycles are consumed when Active Memory Expansion is in-use. The additional CPU cycles needed for Active Memory Expansion is based on the workload and the required memory AME expansion factor. The factor is set on the HMC for each individual LPAR and can be dynamically changed.  Typical AME expansion factors are 1.5 to 2.5 but it varies by workload. Fortunately, you can find a recommended expansion factor running the AIX command amepat before you even purchase AME, although many clients purchase it with their server.

System Requirements
Active Memory Expansion is supported from Power Systems from POWER7 processor-based servers onwards. In order to use Active Memory Expansion, the following minimum levels of software from 2010 are required:

• HMC: V7R7.1.0.0
• System firmware: 7.1
• AIX: 6.1 TL4 SP2

Places for more information:

• AIX Performance Management Article on AME with excellent details and useful commands for monitoring AME - highly recommended

• AIX 7.2 amepat command manual page the Active Memory Expansion Planning and Advisory Tool
• IBM Support: Active Memory Expansion (AME) Question & Answer
• AIX Active Memory Expansion 64KB blocks article AIX Support Article including 64 KB support 
• AME performance statistics can be found in the following performance monitoring commands:
  • topas
    • The main screen:
      • "TMEM" = true memory,
      • "CMEM" = compressed memory,
      • "co" = compress out and
      • "ci" = compresses in,
      • "EFT and EPA" = Expansion Factor Target and Actual
  • vmstat -c
    • See the columns:
      • "co" = compress out and
      • "ci" = compresses in columns
  • lparstat -c
    • See the column:
      • "%xcpu" = expansion CPU use

YouTube Video from Nigel Griffiths - highly recommended

• Active Memory Expansion on YouTube 
  • Including a demo of starting AME and watching its effects
  • 18 minutes
• POWER7 processor-based servers introduced Active Memory Expansion (AME), which makes the memory of an AIX 6 or 7 Logical Partition (LPAR) seem larger than it is by compression memory pages "on the fly".  This movie covers the basics of how AME works, how to switch it on at the LPAR level, running the amepat tool to predetermine good AME settings and then how to monitor AME while it is running a workload.

Power Systems Virtual User Group webinar

• The top Virtual User Group page https://ibm.biz/powersystemsvug with 100's of webinar presentation and replays.
• "29 April 2010: Active Memory Expansion presented by Nigel Griffiths"
  • Presentation slides in PDF
  • Webinar Replay

What AME Expansion Factor to use?

• Fortunately, you do not have to purchase or even switch on AME to find your Expansion Factor and this passive mode works on any POWER processor running the minimum AIX level.
• Provided you are running AIX 6.1 TL4 SP02 or higher, then the amepat command is installed.
  • Check with: whence amepat
  • The answer is: /usr/bin/amepat
• So decide your peak-busy hour of the week and run amepat command as follows - as the root user
  1. Start monitoring with

     amepat -R /tmp/amepat.raw 60

  2. The amepat command takes 60 minutes and saves the stats in a binary format
  3. Then, try various reports as the root user
  4. Shrink memory report - how much memory could be removed:

     • amepat -P /tmp/amepat.raw

  5. Expand memory - can we make the memory look bigger:

     • amepat -P /tmp/amepat.raw -t 4096
     • The 4096 is the target memory size in MB - you have to decide what makes sense to your machine

If you have "spare" CPU capacity that you are willing to trade for expanded memory:

• Look for the CPU use, under, for example, 10% of the virtual machine (LPAR) Shared CPU Entitlement or Dedicated CPUs number and the Expansion Factor recommended in the report.

If you don't have much "spare":

• Look, for example, 0.1 CPU use and the Expansion Factor recommended in the report.

  Expansion    Modeled True      Modeled              CPU Usage  Factor       Memory Size       Memory Gain          Estimate  ---------    -------------     ------------------   -----------       1.00          1.50 GB         0.00 KB [  0%]   0.00 [  0%]       1.09          1.38 GB       128.00 MB [  9%]   0.00 [  0%]       1.20          1.25 GB       256.00 MB [ 20%]   0.00 [  0%]       1.33          1.12 GB       384.00 MB [ 33%]   0.13 [  3%]       1.50          1.00 GB       512.00 MB [ 50%]   0.28 [  7%]

In my small example, if you are OK with using about 0.25 CPU then the Expansion Factor of 1.5 is near enough.
Either, we can:

• Remove 512 MB (and AME makes the real 1 GB of memory look like 1.5 GB) or
• We can make the real 1.5 GB look like 2.25 GB

And give the application more room to play with for better caching and reduced disk I/O.

From client sharing their experience, expansion factors in the field range from 1.3 to 2.6

Don't forget that if you are NOT stressing memory then it is on the free list. The amepat report highlights the free or little used memory. Memory that you can remove from your LPAR and use elsewhere!

Active Memory Expansion is an award winner

Active Memory Expansion (AME) was rated number four in my Top Ten Smarter Technologies from 2010 to be business as usual in the next few years.  Hundreds of votes came from client and IBMer across the world.  Number 1 was Solid-State Drives and we all known that that became true with the whole sale move to SSD technology for high-performance disk I/O.  

• AME allows memory to operate as if it is larger than it is.
• AME allows more LPARs to be running on a server or giving headroom to boost performance.
• The usefulness of AME is all down to the Expansions Factor (EF) - which is based on the compression ratio achieved for memory pages. If the expansion factor is set very low, then you can consider just buying more real memory.  I suspect the expansion factor can be set higher than most people realize. 
• It is a lot higher than I expected.  On my own Systems Director server, for example, I ran expansion factor = 1.5.  So 6 GB looks like 9 GB and it runs well.  But the amepat command that investigates and recommends an expansion factor setting against CPU consumption suggests 2.25.  That means 4 GB of memory looking like 9 GB.  I also suspect that memory containing Java data compresses well and we seem to be running many Java workloads these days. 
• A customer that I work with regularly, reported even higher numbers and the record was about 2.8.
• You can even run amepat passively on unsupported unlicensed hardware or even switching on AME.

---

AME is still popular but IBM could do better informing clients of the low-cost performance boost.

---

Update 64 KB Page Size Enablement of Active Memory Expansion

1. Overview
Active Memory Expansion (AME) is an existing technology. Memory expansion is achieved by compressing in-memory data. Until now, an AME environment only supported 4 KB and 16 MB page sizes. This document details all the enhancements to existing AME technology. 

This feature, 64K page size enablement of AME, requires:

• AIX 7.2 TL1 or later
• POWER8 or later
• Firmware FW860 or later 

Enhanced POWER8 or POWER9 memory bandwidth has enabled its compression accelerators to efficiently compress or decompress 64 KB pages. The firmware leverages this capability by providing 64 KB compression services to guest operating systems.

All AIX supported page sizes are enabled in an AME environment, except for 16 MB Mixed
Page Size Segment (MPSS).  The AME environment supports the following page sizes:

• 4 KB
• 64 KB
• MPSS 4 KB 64 KB
• 16 MB
• 16 GB

POWER8 or POWER9 now has support for 64 KB pageable page size in an AME environment. With an expansion factor of 1.0, applications there is no performance degradation, which previously was the case with AME enabled, due to the lack of 64 KB page size support. 

Now, that the 64 KB pageable page size is supported in an AME environment, administrators can set AME ON as the default policy, with an expansion factor of 1.0, for all the deployed LPARs (on a POWER8). With the expansion factor of 1.0, administrators don't have to worry about tracking individual LPARs and workloads to manage when AME can be used and when it cannot. AME modeling with amepat can be done later for workloads and an expansion factor adjusted.

In an AME enabled environment, both 16 MB and 16 GB pages remain in a decompressed form. The expansion of memory is achieved by compressing 4 KB and 64 KB pages. For the rest of the document, references to “all page sizes” means “all AIX supported page sizes except 16 MB MPSS”.

2. Prerequisites to enable all page sizes in an AME environment
Until now, AME environments only had 4 KB and 16 MB pages enabled. 4 KB pages would use an accelerator for compression and decompression, if available.  It was advisable to have an accelerator but not necessary.  64 KB page size requires an accelerator for compression and decompression in an AME environment.  POWER8 or POWER9 systems with firmware have accelerators that support 64 KB page compression and decompression as a single unit. AIX 7.2 TL01 leverages this POWER8 or POWER9 accelerator support of 64 KB page size. 

3. Enabling all AIX supported page sizes in an AME environment
An upgrade of an existing POWER8 AME environment to AIX 7.2 TL01 does not automatically get all page sizes enabled.  A new install of AIX 7.2 TL01 on a POWER8 server does not automatically get all page sizes enabled in an AME environment.  In both these scenarios, by default, legacy behavior is preserved in an AME environment.  Only 4 KB and 16 MB page sizes are enabled.  There is no change in the environment.  To enable all page sizes in an AME environment, the user would need to explicitly specify the intent to do so. The user uses a new vmo tunable to enable all page sizes in an AME environment.

A new, unrestricted vmo tunable is added to facilitate enabling of all page sizes in an AME environment. To activate this setting it requires you run the bosboot command followed by a reboot of the LPAR. 

Steps to enable all page sizes in an AME environment:

1. On the HMC profile for the LPAR, enable AME with an expansion factor of 1.0 or with an expansion factor as recommended by the amepat command.
2. On the LPAR, enable all page sizes in the AME environment by setting the new vmo tunable

   • vmo -ro ame_mpsize_support=1

3. bosboot and reboot the LPAR

4. Modeling for expansion factor in mixed page size Workloads
The amepat command is used to model for expansion factors for an AME environment. 

In AIX releases before AIX 7.2 TL01 AME worked on 4 KB pages only and the amepat command model assumes 4 KB data compression.  When it encounters a 64 KB page, it is broken down into 16 4KB separate compresses. 

In AIX 7.2 TL01, the enhanced amepat command works well in a multiple page size environment with both 4 KB and 64 KB page workloads. A single unit of 64 KB page data compresses better than 16 4KB units. 

The amepat command has a new -M option. This flag indicates to amepat to compress a 64 KB page as a single unit when determining compressibility of the workload. This flag works on POWER8 or POWER9 servers.

The amepat command gives more accurate expansion factor recommendations when specified

with the -M flag in 64 KB page workload environments. The default behavior for amepat remains unchanged unless the new flag is specified.
Now, with both 4 KB and 64 KB pageable page sizes being supported in an AME environment, it is advised to model for expansion factors in an AME enabled environment with an expansion factor set to 1.0.  With the expansion factor of 1.0, AME is enabled but not used (since there is no memory expansion achieved at an expansion factor of 1.0). When modeled in an AME enabled environment with an expansion factor of 1.0, the amepat command gives more accurate expansion factor recommendations.

5. Live Partition Mobility (LPM) feature or Suspend and Resume feature
The 64 KB page size accelerator support is required to enable the 64 KB page size in an AME environment. POWER8 or POWER9 enables accelerators to compress and decompress 64 KB pages.

When performing LPM or Suspend and Resume of a '64 KB page enabled AME' LPAR, the destination LPAR requires to have 64 KB page accelerator support as well.  LPM fails in the initial stage, if it determines that the destination LPAR doesn't have 64 KB accelerator capabilities and the source LPAR has AME with 64 KB page size enabled.  If using suspend and resume, the suspend succeeds but resume fail, if resume is initiated on an LPAR that doesn't have 64 KB accelerator capabilities.

6. Deactivating accelerator in a 64 KB page AME environment
A hardware accelerator can be deactivated on an LPAR via the HMC. If 64 KB page size is enabled in an AME environment and the accelerator is deactivated, an error log entry is created by the system log facility. After the accelerator is deactivated, both 4 KB and 64 KB pages are compressed and decompressed that uses software methods. Memory already compressed by the accelerator, before deactivating of the accelerator, is decompressed by software methods.

Once the accelerator is activated again for the LPAR, memory is compressed and decompressed that uses the accelerator.  Memory already compressed using software methods, while the accelerator was deactivated, is decompressed that uses software methods. Note, that deactivating of the accelerator in a 64 KB page enabled AME environment is not supported.  If the accelerator is deactivated, AME falls back to software compression and decompression methods to keep the system alive.  It is advised not to deactivate the accelerator if 64 KB page size is enabled in an AME environment.

7. Best practice

1. Use the amepat command to model for expansion factor recommendations once the workload has stabilized.
2. Workloads that are predominantly file-cache do not benefit from AME technology.
3. When modeling in an environment with a 64 KB page workload, specify '-M' flag to the amepat command.
4. To get more accurate expansion factor recommendations from the amepat command, model in an AME enabled environment with the expansion factor of 1.0.

8. Performance of a 64 KB page enabled AME environment
  Now that the 64 KB page size can be enabled in an AME environment, applications do not see any performance degradation with a 1.0 expansion factor. Until now, when AME was activated on an LPAR, 64 KB page workloads were converted to 4 KB workloads, it resulted in performance degradation.

  Here is a comparison of a predominantly 64 KB page workload in an AME enabled environment.  The comparison is made with AME enabled in the legacy mode, with only 4 KB page support, and also with the enhanced AME, with both 4 KB and 64 KB page size support.

  The workload is a Java-based application, which simulates warehouse transaction processing. The Java environment version was 1.8.  First, the workload was run with AME disabled. The LPAR has 60 GB of memory assigned and 42 dedicated cores running in SMT-4. Once the benchmark reaches a steady state, the workload was using 98% of the physical memory at 80% of CPU utilization.

  Throughput of the workload was recorded after a steady state of 1 hour.

  Then, with AME enabled both in legacy mode and enhanced mode, the same workload was run with different memory configurations and throughput recorded. There is no compression and decompression activity when AME is enabled with an expansion factor of 1.0.

Note from Nigel:

• The tests are showing the effect of removing up to 50% of the memory from a busy LPAR heavily use of memory.  Then, trying to carry on by using AME to make up the missing memory.  Without AME, this application would probably fail completely!  As you would expect, it would be hard to over come the lack of memory problem and the results show the negative effect on performance.
• I would prefer to keep the 60 GB of memory and use AME with an expansion factor of 1.5 or 2.0 giving effectively 90 GB or 120 GB of memory. Then we can see how much faster the application can run.  The improved performance would give results that feel much more positive.

  The results are shown in the following table 1:

Memory configuration (in GB)	60	48	40	34.25	30
Memory Expansion	0.00%	25%	50%	75%	100%
Expansion Factor	1.0	1.25	1.5	1.75	2.0
Relative Performance legacy mode = no 64 KB support	0.72	0.12	0.06	0.06	0.07
Relative Performance enhanced mode with 64 KB page support	0.99	0.78	0.67	0.67	0.65
AME CPU utilization % legacy mode = no 64 KB support	0	29	30	45	50
AME CPU utilization % enhanced mode with 64 KB page support	0	7	12	12	14

The Memory Expansion metric in table 1, is calculated by:

• ((baseline physical memory – new physical memory) / (new physical memory)) * 100

The graph, in Fig 1, shows the relative performance of the workload in an AME enabled environment, at different expansion factors, compared to AME disabled environment. With AME enabled in legacy mode, with only 4 KB page support, the performance of the workload went down significantly even at an expansion factor of 1.0. Due to there being no 64 KB page size support in legacy mode. In enhanced AME mode, there is no loss of performance at an expansion factor of 1.0.

  The graph, in Fig 2, shows the percentage of the CPU time used by AME at different expansion factors. The graph demonstrates that AME uses considerably less CPU when enabled in enhanced mode.

  Both graphs in Fig 1 and Fig 2, show that a 64 KB page-focused workload benefited immensely from enhanced AME.

  Generally, a 64 KB chunk of data achieves a higher compression ratio than 16 individual 4 KB chunks. For predominantly 64 KB page-focused workloads, AME enhanced mode, a more aggressive expansion factor can be set while still consuming considerably less CPU as compared to legacy AME.