General considerations regarding cpuplugd rules for memory management

Edit online

This section describes cpuplugd basic memory management, rule priority and how to calculate Linux™ guest sizes, as well as CMM pool sizing.

Rule priority

The cpuplugd mechanism ensures that the plugging rule (adding resources) always overrules the unplugging rule (removing resources), for both CPU and memory allocation. This protects the system against unexpected effects when testing overly aggressive unplugging rules.

Memory management basics

To identify memory which could be removed, any free pages could be taken as a first approach. However, a system continuously doing disk I/O, such as a database, will sooner or later use all its unused memory for page cache, so that no free memory remains. Therefore the memory used for cache and buffers need to be taken into account as well. The critical points here are:
  • Buffers are used by the kernel, and a shortage here may lead to unpredictable effects
  • Page cache is counted as cache
  • Shared memory is also counted as cache

Considering shared memory as free memory is very critical for a system with a Java™ heap or database buffer pools, because they reside in shared memory. Therefore another approach is to calculate the page cache as the difference between cache and shared memory and consider this as free memory. The page cache itself always uses the oldest memory pages for new I/O requests or, in case of cache hits , the accessed page is marked as recently referenced.

Reducing the memory size leads to a reduction of the page cache at the cost of the oldest referenced pages. How much page cache is needed depends on the application type; web server and WebSphere® have relatively low requirements, because in the case under study they are doing only a small amount of disk I/O, as the database itself constitutes a very powerful caching system.

The page scan rate is calculated as the sum of the following parameters:
  • vmstat.pgscan_kswapd_dma
  • vmstat.pgscan_kswapd_normal
  • vmstat.pgscan_kswapd_movable

The direct page scan rate is calculated as the sum of the following parameters:

  • vmstat.pgscan_direct_dma
  • vmstat.pgscan_direct_normal
  • vmstat.pgscan_direct_movable

The available part of the cache (from here on referred to as 'page cache') is calculated as the following difference:

meminfo.Cached -meminfo.Shmem.

All runs were done with a minimum cmm pool size of 0 (CMM_MIN="0") and a maximum of the system size (5 GB) minus 256 KiB (CMM_MAX="1245184") to allow the cmm pool to grow to the maximal possible size. Reserving 256 KiB for the kernel was intended as a safety net; if our rules work well the size of the cmm pool should never approach that maximum value .

Monitoring the guest sizes

The following methods were used to calculate the guest size during the tests:
  • Linux view: the guest definition size minus the cmm pools size over time.
  • z/VM® view: sum of resident pages below 2 GB and pages above 2 GB from the UPAGE report (FCX113) over time.

These two views typically differ. One reason is that Linux provides a view on virtual memory allocation, while z/VM shows the physical memory allocation. Due to optimizations inside z/VM, not all virtual memory pages Linux allocates are backed up with physical memory.

CMM pool increments and decrements

Another important consideration is the size of the increments and decrements of the CMM pool. There are two important requirements:
  • Starting an application, middleware or workload leads to a relatively large memory requirement in a short time. If the system does not react fast enough this in turn may lead to an out of memory exception. Our results indicate that the middleware typically does not allocate the entire amount of configured memory in a single step. For example, on the WebSphere Application Server systems with a 1 GB Java heap the CMM pool is reduced in multiple (4) steps when the workload was started, where only the first step was at the maximum size of 500 MiB. This softens this requirement for a fast reaction time, because even a system with a much larger Java heap (for example, 6 GB) would probably not require a CMM_DEC of 6 GB.
  • A highly frequent oscillating CMM pool size should be avoided, because of the related overhead for operating system and z/VM.
These considerations suggest the following approach:
  • The parameter CMM_INC was defined as the percentage of memory that is free (for example, 10%). This causes the increment of the pool to become smaller and smaller the closer the system comes to the 'ideal' configuration.
  • The parameter CMM_DEC was defined as a percentage of the system size (for example, 10%, which would correspond to ~500 MiB). This results in a fix value independent of the current load situation and leads to a relatively fast decrement of the pool whenever a memory shortage is detected, depending on the applied rule.

The effect is an asymptotic increase of the CMM pool, up to the level where no more volatile memory is available for removal, while a request for new memory can be served in a small number of steps. With this setup both requirements where fulfilled in most cases.

CMM_MIN and CMM_MAX

The minimum size of the pool was specified as 0 pages, to allow full removal of the pool. As maximum value a very large value of 1,245,184 pages (4,864 MiB) was specified, which stops increasing the pool when less than 256 GB memory remains. The expectation was that the indicators for memory shortage would appear before the pool reaches that size, causing a reduction in the pool size, which in turn increases the available memory. This approach worked very well.