Active Memory Expansion (AME)

Active Memory Expansion (AME) is a new technology for expanding a system's effective memory capacity. AME employs memory compression technology to transparently compress in-memory data, allowing more data to be placed into memory and thus expanding the memory capacity of configured systems.

Overview

Active Memory Expansion (AME) relies on compression of in-memory data to increase the amount of data that can be placed into memory and thus expand the effective memory capacity of a IBM® Power Systems processor-based server. The in-memory data compression is managed by the operating system, and this compression is transparent to applications and users. AME is configurable on a per-logical partition (LPAR) basis. Thus, AME can be selectively enabled for one or more LPARs on a system.

When Active Memory Expansion is enabled for an LPAR, the operating system compresses a portion of the LPAR's memory and leaves the remaining portion of memory uncompressed. This results in the memory being effectively broken up into two pools. They are:

  • Compressed pool
  • Uncompressed pool

The operating system varies dynamically the amount of memory that is compressed based on the workload and the configuration of the LPAR.

The operating system moves data between the compressed and uncompressed memory pools based on the memory access patterns of applications. When an application needs to access data that is compressed, the operating system automatically extracts the data and moves it from the compressed pool to the uncompressed pool, making it available to the application. When the uncompressed pool is full, the operating system compresses the data and moves it from the uncompressed pool to the compressed pool.

This compression and decompression activity is transparent to the application. As AME relies on memory compression, some additional CPU utilization are consumed when the AME is in use. The amount of additional CPU utilization needed for AME varies based on the workload and the level of memory expansion being used.

Memory Expansion Factor
Note: When AME is enabled, the AIX operating system uses 4 KB pages.

Memory expansion factor and expanded memory size

When configuring Active Memory Expansion, there is a single configuration option that must be set for the LPAR that is the memory expansion factor. An LPAR's memory expansion factor specifies the target effective memory capacity for the LPAR. This target memory capacity provides an indication to the operating system of how much memory is made available with memory compression. The target memory capacity specified is referred to as the expanded memory size. The memory expansion factor is specified as a multiplier of an LPAR's true memory size.
LPAR_expanded_mem_size = LPAR_true_mem_size * LPAR_mem_exp_factor 
For example, using a memory expansion factor of 2.0 for an LPAR indicates that memory compression must be used to double the LPAR's memory capacity. If an LPAR is configured with a memory expansion factor of 2.0 and a memory size of 20 GB, then the expanded memory size for the LPAR is 40 GB.
40 GB = 20 GB * 2.0 

The operating system compresses enough in-memory data to fit 40 GB of data into 20 GB of memory. The memory expansion factor and the expanded memory size can be dynamically changed at runtime by using the Hardware Management Console (HMC) through dynamic LPAR operations. The expanded memory size is always rounded down to the nearest logical memory block (LMB) multiple.

Memory Deficit

When configuring the memory expansion factor for an LPAR, it is possible that a memory expansion factor might be chosen that is too large and cannot be achieved based on the compressibility of the workload. When the memory expansion factor for an LPAR is too large, then a memory expansion deficit forms, indicating that the LPAR cannot achieve its memory expansion factor target. For example, if an LPAR is configured with a memory size of 20 GB and a memory expansion factor of 1.5, which results in a total target expanded memory size of 30 GB. However, the workload running in the LPAR does not compress well, and the workload's data only compresses by a ratio of 1.4 to 1. In this case, it is impossible for the workload to achieve the targeted memory expansion factor of 1.5. The operating system limits the amount of physical memory that can be used in a compressed pool to a maximum of 95%. In this example, if 19 GB was reserved for the compressed, then the maximum achievable expanded memory size would be 27.6 GB. The result is a 2.4 GB shortfall. This shortfall is referred to as the memory deficit.

The effect of a memory deficit is the same as the effect of configuring an LPAR with too little memory. When a memory deficit occurs, the operating system cannot achieve the expanded memory target configured for the LPAR, and the operating system might have to resort to paging out virtual memory pages to paging space. Thus, in the above mentioned example, if the workload used more than 27.6 GB of memory, the operating system would start paging out virtual memory pages to paging space. To get an indication of whether a workload can achieve its expanded memory size, the operating system reports a memory deficit metric. This is a “hole” in the expanded memory size that cannot be achieved. If this deficit is zero, the target memory expansion factor can be achieved, and the LPAR's memory expansion factor is configured correctly. If the expanded memory deficit metric is non-zero, then the workload falls short of achieving its expanded memory size by the size of the deficit. To eliminate a memory deficit, the LPAR's memory expansion factor should be reduced. However, reducing the memory expansion factor reduces the LPAR's expanded memory size. Thus to keep the LPAR's expanded memory size the same, the memory expansion factor must be reduced and more memory must be added to the LPAR. Both the LPAR's memory size and memory expansion factor can be changed dynamically.

Memory Expansion Factor

Planning Considerations

Before deploying a workload in the Active Memory Expansion (AME) environment, some initial planning is required to ensure that a workload gets the maximum benefit from AME. The benefit of AME to a workload varies based on the workload's characteristics. Some workloads can get a higher level of memory expansion than other workloads. The Active Memory Expansion Planning and Advisory Tool amepat assists in planning the deployment of a workload in the Active Memory Expansion environment and provides guidance on the level of memory expansion a workload can achieve.

AME Planning Tool

The AME Planning Tool (located in /usr/bin/amepat) serves two primary purposes. They are

  • To plan an initial Active Memory Expansion configuration
  • To monitor and fine-tune an active AME configuration

AME Planning Tool can run on LPARs with and without AME enabled. In an LPAR where AME has not been enabled, run amepat with a representative workload. It should be set to monitor that workload for a meaningful period of time. For example, the amepat tool is set to run for the duration of a workload's peak resource usage. Once completed, the tool displays a report with a variety of potential memory expansion factors and the expected CPU utilization attributable to an AME for each factor. The tool also provides a recommended memory expansion factor that seeks to maximize memory savings while minimizing additional CPU utilizationThe report and recommendation can be a useful initial configuration for an AME deployment. In an LPAR where AME is enabled, amepat serves a similar purpose. When run at peak-time for a representative workload, the tool provides a report with the actual CPU utilization attributable to AME at the current memory expansion factor. It also displays memory deficit information, if one is present. Because the AME is enabled, the tool can also provide a more accurate representation of what CPU utilization levels can be expected at different memory expansion factors. A new recommendation based on this information will be presented to the user.

An example of a report generated in an AME disabled partition and a sample workload follows:
# amepat 5 2

Command Invoked                : amepat 2 5

Date/Time of invocation        : Wed Dec  2 11:29:29 PAKST 2009
Total Monitored time           : 10 mins 58 secs
Total Samples Collected        : 5

System Configuration:
---------------------
Partition Name                 : aixfvt19
Processor Implementation Mode  : POWER5
Number Of Logical CPUs         : 8
Processor Entitled Capacity    : 4.00
Processor Max. Capacity        : 4.00
True Memory                    : 4.25 GB
SMT Threads                    : 2
Shared Processor Mode          : Disabled
Active Memory Sharing          : Disabled
Active Memory Expansion        : Disabled

System Resource Statistics:    Average       Min         Max
---------------------------    -----------   ----------  -----------
CPU Util (Phys. Processors)    2.00 [ 50%]   1.00 [ 25%] 3.00 [ 75%]
Virtual Memory Size (MB)       1366 [ 31%]   1113 [ 26%] 2377 [ 55%]
True Memory In-Use (MB)        1758 [ 40%]   1234 [ 28%] 3834 [ 88%]
Pinned Memory (MB)             673 [ 15%]     673 [ 15%]  675 [ 16%]
File Cache Size (MB)           391 [  9%]     124 [  3%] 1437 [ 33%]
Available Memory (MB)          841 [ 65%]    1812 [ 42%] 3099 [ 71%]

Active Memory Expansion Modeled Statistics
-------------------------------------------
Modeled Expanded Memory Size   :   4.25 GB
Average Compression Ratio      :   5.29

Expansion    Modeled True      Modeled              CPU Usage
Factor       Memory Size       Memory Gain          Estimate
---------    -------------     ------------------   -----------
     1.00          4.25 GB         0.00 KB [  0%]   0.00 [  0%]
     1.31          3.25 GB         1.00 GB [ 31%]   0.34 [  8%]
     1.55          2.75 GB         1.50 GB [ 55%]   0.39 [ 10%]
     1.89          2.25 GB         2.00 GB [ 89%]   0.45 [ 11%]
     2.12          2.00 GB         2.25 GB [112%]   0.50 [ 12%]
     2.43          1.75 GB         2.50 GB [143%]   0.65 [ 16%]
     2.83          1.50 GB         2.75 GB [183%]   0.70 [ 18%]

Active Memory Expansion Recommendation:
---------------------------------------
The recommended AME configuration for this workload is to configure the LPAR
with a memory size of 1.50 GB and to configure a memory expansion factor
of 2.83.  This will result in a memory gain of 183%. With this
configuration, the estimated CPU usage due to AME is approximately 0.50
physical processors, and the estimated overall peak CPU resource required for
the LPAR is 3.50 physical processors.

NOTE: amepat's recommendations are based on the workload's utilization level
during the monitored period. If there is a change in the workload's utilization
level or a change in workload itself, amepat should be run again.

The modeled Active Memory Expansion CPU usage reported by amepat is just an
estimate.  The actual CPU usage used for Active Memory Expansion may be lower
or higher depending on the workload.

The report comprises six sections that are explained in order.

Command Information

This section provides details about the arguments passed to amepat, time of invocation, the total time the system is monitored and the number of samples collected. In the report, the amepat

is invoked for 10 minutes with an interval of 2 minutes and 5 samples.
Note: It can be observed that the Total Monitored time displayed is 10 minutes and 58 seconds. The extra 58 seconds is used for gathering system statistics required for the Active Memory Expansion Modeling.

System Configuration

This section provides configuration information such as host name, processor architecture, CPUs, entitlement, true memory size, SMT state, processor, and memory mode. In the report above, the disabled status of Active Memory Expansion indicates that amepat is invoked in an AME-disabled partition.
Note: The amepat tool can also be invoked in an AME-enabled partition to monitor and fine-tune an active AME configuration. In an AME-enabled partition, the System Configuration Section also displays the target expanded memory size and the target memory expansion factor.

System Resource Statistics

This section displays the Utilization of system resources over the monitoring period. It shows the average, minimum, and maximum values along with the corresponding percentages for system resource utilization.

In the report mentioned, the workload on average utilizes 2.00 physical processors (as shown in the CPU Util row) which is 50% of maximum physical capacity (4.00 as shown in the Processor Max Capacity row) of the LPAR.

Note: The CPU utilization report includes the processing units consumed for AME Modeling. In an AME-enabled partition this also includes CPU utilization that results from compression or decompression activities.

In the report mentioned above, the memory utilization of the workload can also be observed. All the memory percentages are relative to the total true memory size of LPAR

Note: High percentages of Pinned Memory and File Cache indicate that the workload might not get significant benefit from AME.

Active Memory Expansion Statistics

This section is displayed only when the amepat tool is invoked in an AME enabled LPAR.

Find an example of the sample output:

AME Statistics:                  Average        Min       Max
---------------                 -----------    --------   ---------
AME CPU Usage (Phy. Proc Units) 0.25 [  6%]   0.01 [  0%] 0.50 [ 13%]
Compressed Memory (MB)          264 [ 13%]    264 [ 13%]  264 [ 13%]
Compression Ratio               2.15          2.15        2.16
Deficit Memory Size (MB)        562 [ 55%]    562 [ 55%]   562 [ 55%]

This section of the report shows the AME CPU Usage, Compressed Memory,
Compression Ratio & Deficit Memory. 
The Deficit Memory will be reported only if there is a memory deficit
in achieving the expanded memory size. 
Otherwise the tool will not report this information. 
For example in the above report, it can be observed that there is
an average memory deficit of 562 MB which is 55% of the Target 
Expanded Memory Size of 2 GB (which is reported in the 
System Configuration Section when AME is enabled). 
The report also shows on an average 264 MB out of 2GB of expanded memory is in
compressed form as reported against Compressed Memory.

Active Memory Expansion Modeled Statistics

This section displays the Modeled Expanded Memory size, compression ratio, and a table with a number of possible AME configurations. In the mentioned report, the Modeled Expanded Memory Size is

reported as 4.25 GB, which is the LPAR's True Memory Size. By default amepat uses the partition's true memory as the Modeled Expanded Memory. This can be altered by using the

-t or -a options. The report shows an average compression ratio of 5.29 which indicates that the workload compresses well. Compression ratios close to 1 might indicate that little

memory expansion is possible. The configurations displayed in the modeling table are based on the Modeled Expanded Memory Size as the target memory.

The table shows Modeled True Memory Size, Modeled Memory Gain, and additional CPU usage attributed to AME for various expansion factors.

For example, see the following row in the table

1.55          2.75 GB         1.50 GB [ 55%]   0.39 [ 10%]

Here the original true memory size of 4.25 GB can be achieved with 2.75 GB physical memory size and an expansion factor of 1.55. This configuration may result in the CPU usage increasing by

0.39 physical processors (10% of maximum capacity).

Active Memory Expansion Recommendation

This section displays an Active Memory Expansion Configuration recommendation that is made for the workload. The optimal configuration will never have an AME CPU usage of over 15%. This section also provides information on the AME CPU Usage and Memory Gain for the recommended configuration. In the report mentioned, it is observed that an expansion factor of 2.12 is recommended. The AME CPU usage target can be altered using the -c or -C options.

Note: All recommendations made in this section are based on the running workload over the monitoring period. AME CPU usage reported in this section is an estimate and may vary. The amepat's modeling output does not estimate changes to a workload's throughput or response time at different levels of memory expansion. Before deploying a workload into production with Active Memory Expansion, the configuration used should be closely monitored to ensure that the workload's performance goals are met. When amepat is run in an AME-enabled environment, a warning is produced when there is a deficit in expanded memory size.
Note: The output appears as follows:
WARNING: This LPAR currently has a memory deficit of 562 MB.
A memory deficit is caused by a memory expansion factor that is too
high for the current workload.  It is recommended that you reconfigure
the LPAR to eliminate this memory deficit.  Reconfiguring the LPAR
with one of the recommended configurations in the above table should
eliminate this memory deficit.

The recommended AME configuration for this workload is to configure
the LPAR with a memory size of ...

For more information on these and other uses of the AME Planning tool, please refer to the amepat man page.

Performance Monitoring

Several AIX performance tools can be used to monitor Active Memory Expansion statistics and gather information about Active Memory Expansion.

The following table summarizes the different tools and options that can be used to monitor Active Memory Expansion statistics:
Tool Option Description
amepat -N Provides general CPU and memory utilization statistics. Also provides an indication of the CPU utilization for AME compression and decompression activity, as well as compression, decompression, and memory deficit statistics.
vmstat -c Provides compression, decompression, and deficit statistics.
lparstat -c Provides an indication of the CPU utilization for AME compression and decompression activity. Also provides memory deficit information.
svmon -O summary=ame Provides a summary view of memory usage broken down into compressed and uncompressed pages.
topas   The default topas screen displays the memory compression statistics when it is run in the AME environment.

vmstat command

The vmstat command can be used with it's –c option to display AME statistics.

# vmstat –c 2 1

System configuration: lcpu=2 mem=1024MB tmem=512MB ent=0.40 mmode=dedicated-E

kthr                memory                      page          faults               
 r   b     avm   fre    csz   cfr    dxm   ci   co  pi  po  in   sy   cs 
 0   0  309635  2163  43332   943  26267  174  386   0   0  93  351  339  

cpu
us sy id wa    pc    ec
2  3 89  7  0.02   5.3



In the output above, the following memory compression statistics are provided:

  • Expanded memory size mem of the LPAR is 1024 MB.
  • True memory size tmem of the LPAR is 512 MB.
  • The memory mode mmode of the LPAR is Active Memory Sharing disabled and Active Memory Expansion enabled.
  • Compressed Pool size csz is 43332 4K- pages.
  • Amount of free memory cfr in the compressed pool is 943 4K- pages.
  • Size of expanded memory deficit dxm is 26267 4K- pages.
  • Number of compression operations or page-outs to the compressed pool per second co is 386.
  • Number of decompression operations or page-ins from the compressed pool per second ci is 174.

lparstat command

The lparstat command can be used with –c option to display AME statistics.

# lparstat -c 2 5

System configuration: type=Shared mode=Uncapped mmode=Ded-E smt=On 
lcpu=2 mem=1024MB tmem=512MB psize=14 ent=0.40

%user  %sys  %wait  %idle physc %entc  lbusy   app  vcsw phint  %xcpu    dxm
----- ----- ------ ------ ----- ----- ------   --- ----- ----- ------ ------
 45.6  51.3    0.2    2.8  0.95 236.5   62.6 11.82  7024     2    5.8    165
 46.1  50.9    0.1    2.8  0.98 243.8   64.5 11.80  7088     7    6.0    162
 46.8  50.9    0.3    2.1  0.96 241.1   69.6 11.30  5413     6   19.4    163
 49.1  50.7    0.0    0.3  0.99 247.3   60.8 10.82   636     4    8.6    152
 49.3  50.5    0.0    0.3  1.00 248.9   56.7 11.47   659     1    0.3    153

In the output, the following memory compression statistics are provided

  • Memory mode mmode of the LPAR is Active Memory Sharing disabled and AME enabled.
  • Expanded memory size mem of the LPAR is 1024 MB.
  • True memory size tmem of the LPAR is 512 MB.
  • Percentage of CPU utilized for Active Memory Expansion activity %xcpu.
  • Size of expanded memory deficit dxm in megabytes.

topas command

The topas main panel in an LPAR with Active Memory Expansion enabled

displays memory compression statistics automatically under the sub-section AME.

Topas Monitor for host:    proc7                EVENTS/QUEUES    FILE/TTY
Mon Dec 14 16:30:50 2009   Interval:  2         Cswitch    1240  Readch  43.2M
                                                Syscall  110.8K  Writech 102.5K
CPU  User%  Kern%  Wait%  Idle%  Physc   Entc   Reads     12594  Rawin     0
ALL   49.1   50.7    0.0    0.3   1.00  249.7   Writes      515  Ttyout   388
                                                Forks       218  Igets      0
Network  KBPS   I-Pack  O-Pack   KB-In  KB-Out  Execs       218  Namei    5898
Total     1.2      7.5     1.0     0.9     0.3  Runqueue    1.0  Dirblk      0
                                                Waitqueue   0.0
Disk    Busy%     KBPS     TPS KB-Read KB-Writ                   MEMORY
Total     0.0      0.0     0.0     0.0     0.0  PAGING           Real,MB   1024
                                                Faults    53184  % Comp     85
FileSystem        KBPS     TPS KB-Read KB-Writ  Steals        0  % Noncomp   0
Total             75.4K   21.1K  75.3K  95.4    PgspIn        0  % Client    0
                                                PgspOut       0
WLM-Class (Active)     CPU%    Mem%  Blk-I/O%   PageIn        0  PAGING SPACE
System                    0      61         0   PageOut       0  Size,MB    512
Default                   0       4         0   Sios          0  % Used      1
                                                                 % Free     99
Name            PID CPU% PgSp Class             AME
inetd        364682  3.5  0.5 wpar1             TMEM,MB     512  WPAR Activ   1
xmtopas      622740  0.4  0.7 wpar1             CMEM,MB     114  WPAR Total   1
topas        413712  0.1  1.5 System            EF[T/A] 2.0/1.5  Press: "h"-help
random       204934  0.1  0.1 System            CI:5.5  CO:0.0          "q"-quit

In the above output, the following memory compression statistics are provided.

  • True memory size TMEM,MB of the LPAR is 512 MB.
  • Compressed pool size CMEM,MB of the LPAR is 114 MB.
  • EF[T/A] – Target Expansion Factor is 2.0 and Achieved Expansion Factor is 1.5.
  • Rate of compressions co and decompressions ci per second are 0.0 and 5.5 pages respectively.

svmon command

The svmon tool can provide a detailed view of AME usage on an LPAR.

# svmon -G -O summary=ame,pgsz=on,unit=MB
Unit: MB
-------------------------------------------------------------------------------
               size       inuse        free   pin     virtual  available mmode
memory      1024.00      607.54      144.11   306.29  559.75    136.61   Ded-E
 ucomprsd         -      387.55           -
 comprsd          -      219.98           -
pg space     512.00        5.08

               work        pers        clnt       other
pin          213.34           0           0        28.9
in use       534.12           0        9.42
 ucomprsd    314.13
 comprsd     219.98

PageSize   PoolSize       inuse        pgsp         pin     virtual   ucomprsd
s    4 KB         -      543.54        5.02      242.27      560.59     323.55
L   16 MB         4           0           0        64.0           0          0
-------------------------------------------------------------------------------
True Memory: 512.00

              CurSz  %Cur       TgtSz        %Tgt       MaxSz   %Max   CRatio
ucomprsd     405.93  79.28      168.38       32.89      -          -    -
comprsd      106.07  20.72      343.62       67.11      159.59   31.17  2.51

                txf         cxf         dxf         dxm
AME            2.00        1.46        0.54      274.21

In the output above, the following memory compression statistics are provided:

  • Memory mode mmode of the LPAR is Active Memory Sharing disabled and AME enabled.
  • Out of a total of 607.54 MB in use memory_inuse, uncompressed pages ucomprsd_inuse constitute 387.55 MB and compressed pages comprsd_inuse constitute the remaining 219.98 MB
  • Out of a total of 534.12 MB working pages in use inuse_work, uncompressed pages ucomprsd_work constitute 314.13 MB and compressed pages comprsd_work constitute 219.98 MB.
  • Out of a total of 543.54 MB of in use pages 4KB_inuse in 4K-PageSize Pool, uncompressed pages 4KB_ucomprsd constitute 323.55 MB.
  • Expanded memory size memory_size of the LPAR is 1024 MB.
  • True memory size True Memory of the LPAR is 512 MB.
  • Current size of the uncompressed pool ucomprsd_CurSz is 405.93 MB (79.28% of the total true memory size of the LPAR).
  • Current size of the compressed pool comprsd_CurSz is 106.07 MB (20.72% of the total true memory size of the LPAR).
  • The target size of the compressed pool comprsd_TgtSz needed to achieve the target memory expansion factor txf of 2.00 is 343.62 MB (67.11% of the total true memory size of the LPAR).
  • The size of the uncompressed pool ucomprsd_TgtSz in that case becomes 168.38 MB (32.89% of the total true memory size of the LPAR).
  • The maximum size of the compressed pool comprsd_MaxSz is 159.59 MB (31.17% of the total true memory size of the LPAR).
  • The current compression ratio CRatio is 2.51 and the current expansion factor cxf achieved is 1.46
  • The amount of expanded memory deficit dxm is 274.21 MB and the deficit expansion factor dxf is 0.54.

The –O summary=longame option provides summary of memory compression details as follows:

# svmon -G -O summary=longame,unit=MB
Unit: MB
                                   Active Memory Expansion
--------------------------------------------------------------------
	Size     Inuse     Free   DXMSz  UCMInuse  CMInuse   TMSz    TMFr    
1024.00  607.91  142.82  274.96   388.56    219.35   512.00  17.4  
 
CPSz  CPFr  txf  cxf   CR
106.07 18.7 2.00 1.46   2.50

In the output, the following memory compression statistics are provided:

  • Out of the total expanded memory size Size of 1024.00 MB, 607.91 MB is in use Inuse and 142.82 MB is free Free. The deficit in expanded memory size DXMSz is 274.96 MB.
  • Out of the total in use memory Inuse of 607.91 MB, uncompressed pages UCMInuse constitute 388.56 MB and the compressed pages CMInuse constitute the remaining 219.35 MB.
  • Out of the true memory size TMSz of 512.00 MB, only 17.4 MB of True Free memory TMFr is available.
  • Out of the compressed pool size CPSz of 106.07 MB, only 18.7 MB of free memory CPFr is available in the compressed pool.
  • Whereas the target expansion factor txf is 2.00, the current expansion factor cxf achieved is 1.46.
  • The compression ratio (CR) is 2.50.