IBM Support

Large page best practices, considerations and monitoring

Question & Answer


Question

What are the best practices and key considerations for the AIX® large page support? how to monitor the large pages activity?

Answer

This document discusses the best practices needed when you use the AIX® large page support, also the tools used to monitor the large pages behavior.
A configuration is needed to use large pages!
  • The system is not configured for large page support by default, you need to specify the amount of memory needs to get allocated for the large pages.
  • lgpg_regions and lgpg_size tunables need to be configured to specify the large page memory amount, configuration is done by using the vmo command.
  • The application still needs to be configured to use the large page support by involving the ldedit command or LDR_CNTRL environment variable or both.
  • When you use large pages for shared memory, use # vmo -p -o v_pinshm=1, if v_pinshm set to 1, the system will allow pinning of shared memory segments, a value of 0 (default) indicates off.
  • The large page pool is dynamic, the amount of physical memory that you specify takes effect immediately and does not require a system reboot.
  • Run the bosboot command so that the vmo command that you previously ran takes effect following the next system boot.
 
4 KB page against 16 MB large page
  • AIX® maintains separate 4 KB and 16 MB physical memory pools.
  • After configuring the large page pool, the remaining physical memory backs the 4 KB virtual pages.
  • In out_of_memory conditions, the system converts the available (unused) large pages to 4 KB pages, check details below.
  • If large pages were not configured for the shared memory, the 4 KB pages will handle the shared memory segments job.
AIX® treats the large page as pinned memory!
  • The large pages are permanently pinned in memory and have a separate memory pool.
  • Be careful when you use large page support, make sure the system has enough memory when the large page is configured.
  • AIX® might not boot up when a high percentage of large page memory was assigned, as it might not find enough memory for the boot process.
  • With medium range memory partitions, allocating more than 50% of memory as large page memory is not recommended.
  • One case showed a system configured with large page support and had multiple dynamic memory increase operations, later after a hard reboot it didn't start up as it restored the old lower memory assigned from its profile while most of this memory was assigned for the large page pool.
  • The AIX® kernel is not using the large pages.
Assign enough memory for:
  • The AIX® operating system kernel.
  • The operating system applications.
  • The applications that don't use the large page.
When the system is at or near the pinned memory limit!
  • In general, don't let the system reaches such a condition, make sure your system still has enough memory for the non-large page applications.
  • In this case, you might try to tune maxpin% percentage or lgpg_regions parameter, better contact IBM AIX® Support.
  • The parameter maxpin% in vmo tunables can control the large page, as maxpin% specifies the maximum percentage of real memory that can be pinned.
  • You might decrease the large page pool size by lowering lgpg_regions tunable if there were large pages available (unused) so you can lower the total pinned memory on the system.
Out of memory condition
  • If the system has high usage in active virtual memory (computational memory) and becomes out of 4 KB pages, the system searches for the available (unused) large pages and converts them to 4 KB pages.
  • If there were no available (unused) large pages, the out_of_memory condition could hang the system and becomes unreachable.
  • The only solution in most of these hang cases is to have a forced reboot with a system dump option (contact AIX Support Team to check this dump for you).
  • When a large page is converted to a 4 KB page, you can see an informational error generated in the error report, the error label is LGPG_FREED (ONE OR MORE LARGE PAGES HAS BEEN CONVERTED INTO PAGEABLE PAGES).
  • A single converted large page will provide 4096 small (4 KB) pages to the system.
  • The converted pages from large page size to 4 KB size will remain as 4 KB until the next system reboot, this is working as designed.
  • So if the system didn't reboot after this conversion, the large page pool will remain with a size lower than the configured pool size, hence this could have an effect on the application that uses the large pages.
  • A reboot would return back the converted (large pages to 4 KB) pages back to large pages again, so the large page pool configured size restored.
  • While some applications might benefit from large page usage, a reduction of 4 KB pages available impacts the system performance, that's why you need to be sure you assigned enough memory for your system.
Check with the application vendor!
  • Consult the application vendor to confirm their application is certified to use large pages, and that it should have improved performance. Confirm whether the application has better performance when it uses large pages.
  • If your application works better with non-large page memory, you might experience performance problems if this application uses large pages.
  • It's not always a good choice to work with the large page support, some applications are not performing better with large page support.
  • Some applications can be severely degraded by the use of large pages.
  • One test showed a small shell script execution time when using large pages increased 5 times the time while using normal 4 KB pages.
LDR_CNTRL and non-large pages programs notes!
  • The non-large programs or the AIX® programs should not be using the large page support, they are working better with the default operating system pages 4 KB and 64 KB.
  • Using ldedit is better than LDR_CNTRL in some scenarios, this is to control specific applications to use large page support.
  • If the LDR_CNTRL variable used, configure it to pick up the needed application only to use large pages.
  • Setting LDR_CNTRL in the application startup scripts might be better than setting it in /etc/environment file or in the general application user environment.
  • A security access control mechanism prevents unauthorized applications from using large pages.
  • It also prevents unauthorized users from using large pages for their applications.
  • The inappropriate configuration will make the large page pool gets busy serving non-large page programs, while it should be ready for your large page application. 
  • Check Enabling, disabling, and testing the large page support document, look in the section "Testing a non-large page program" and how could using large pages affects the non-large page program execution time.
  • Execution of the non-large page command showed a higher execution time when using the large pages exceeding 5 times the time taken when using the default pages (the non-large-pages).
Notes of caution
  • The application's large page configuration is needed before its execution, a working application with 4 KB or 64 KB pages will remain on 4 KB and 64 KB until execution ends.
  • The application cannot switch from non-large pages to the large page modes after execution.
  • Application data backed by large pages remain in physical memory until the application completes.
  • AIX® does not provide paging support for large pages, so large pages are not paged in/out from/to the paging space, hence make sure to have enough large page pool size to cover your application workload.
  • The large pages are permanently pinned in memory, they have a separate memory pool hence you need to keep an eye over the pinned percentage when using large pages.
  • Using large pages incorrectly may drive your system out of memory causing a system hang.
  • Set the application to use large pages when possible and to use default pages if no large pages are available.
  • Be sure to monitor the number of pages allocated and actually in use with the svmon –G command.
  • Large page support main purpose is performance improvement but you need to configure it appropriately for the best results.
  • Large page support is not recommended for general use, specific applications only can use large pages (this part has been confirmed with cfgmgr test above).
  • Partitions use shared memory cannot use large page, the large pages are supported on dedicated memory partitions.
  • Active Memory® Expansion (AME) can work with large pages but it is not recommended, (Active Memory Expansion is a feature to expand the operating system working memory starting from AIX 6.1 TL04 SP02).
Monitoring large page
Checking used and available (free) large pages
  • Use the vmstat -l command.
    • If the -l flag is specified, an extra "large-page" section is displayed with 2 columns, alp, and flp.
    • alp: Indicates the number of large pages currently in use.
    • flp: Indicates the number of large pages on the large page free list.
      kthr    memory           page                   cpu                 large-page
      ------- ---------------- ---------------------- ------------------- ----------
        r   b      avm     fre re  pi  po  fr  sr  cy us sy id wa  pc  ec alp    flp
        5   1 16446108 5241821  0   0   0   2   4   0  2  1 97  0 0.3 5.4   0      0
  • Vmstat also has two options available to display memory statistics for the large page size.
    1. Use # vmstat -p 16M, check the line starts with 16M, and find avm (used) and fre (free) values.
    2. And # vmstat -P 16M, check the line starts with 16M and find siz (total), avm (used), and fre (free) values.
      # vmstat -p 16M
        psz        avm        fre    re    pi    po    fr     sr    cy     siz
        16M          0          0     0     0     0     0      0     0       0
 
      # vmstat -P 16M
        pgsz            memory                           page
        ----- -------------------------- ------------------------------------
                   siz      avm      fre    re    pi    po    fr     sr    cy
          16M        0        0        0     0     0     0     0      0     0
  • The svmon command in parallel with the -P flag
    • It could be used as well to display used large pages per process.
    • Check the Pagesize section, find the line starts with L (L for large page), Inuse is the current large pages' utilization.
    • Check the last column, 16MB as 'N' indicates no large pages configured.
      # svmon -P 26673710
        -------------------------------------------------------------------------------
             Pid Command          Inuse      Pin     Pgsp  Virtual 64-bit Mthrd  16MB
        26673710 ksh              66005    17731        0    65925      N     N     N
             PageSize             Inuse      Pin     Pgsp  Virtual
             s    4 KB            14005    13139        0    13925
             m   64 KB             3250      287        0     3250
             L   16 MB                0        0        0        0
             S   16 GB                0        0        0        0
  • Using KDB
    • With vmstat sub-command
    • It displays the large page pool size in GB.
    • Check the line "16M number of frames"
      (0)> vmstat | grep 16M
                16M number of frames                : 00000040   1.0GB
                16M frames pinned                   : 00000040   1.0GB
                16M system pinnable frames remaining: 00000000   0.0MB
           16M user pinnable frames remaining  : 00000000   0.0MB
  • Using the NMON tool
    • To monitor the large pages activity, run # nmon then press M
    • The output will be similar to the following:
      • memory frames in 16MB shows 64 (large page pool size)
      • free list pages in 16MB shows 48 (available large pages)
      • accessed virtual pages in 16MB shows 16 (the used large pages)
      Page Sizes= 4   Page Size ->           4KB   64KB  16MB  16GB
        numframes memory frames          1532128  18930    64     0
           numfrb free list pages         208387   2812    48     0
        numclient client frames           939415      0     0     0
      numcompress compressed segments          0      0     0     0
          numperm non-working segments    939415      0     0     0
        numvpages accessed virtual pages  371506  16118    16     0
          minfree min free list              960     60     0     0
          maxfree max free list             1088     68     0     0
          numpout page-outs                    0      0     0     0
        numremote remote page-outs             0      0     0     0
       numwseguse in use working seg      371506  16118    16     0
       numpseguse in use persistent            0      0     0     0
      numclseguse in use client           952235      0     0     0
       numwsegpin pinned working          358891   9677    16     0
       numpsegpin pinned persistent            0      0     0     0
      numclsegpin pinned client seg         2108      0     0     0
      numpgsp_pgs allocated PageSpace       6410      0     0     0
        numralloc remote allocations           0      0     0     0
       pfrsvdblks system reserv.blocks    144945   1893     0     0
          pfavail pin available          1171129   9253     0     0
       pfpinavail pinnable@apps level    1354234  16572     0     0
       system_pgs SCBs marked V_SYS       270755   8819     0     0
       nonsys_pgs SCBs not V_SYS          970250   7284    16     0
Reading svmon -G output
 
    # /usr/bin/svmon -G
                     size       inuse        free         pin     virtual   mmode
      memory     54591488    51039208     3552280    43984475    46040778     Ded
      pg space    8388608       37621
                     work        pers        clnt       other
      pin        42343300           0        3655     1637520
      in use     46040778           0     4998430
      PageSize   PoolSize       inuse        pgsp         pin     virtual
      s    4 KB         -     7625384       37621     2251051     2626954
      m   64 KB         -      419604           0      314579      419604
      L   16 MB      8960        8960           0        8960        8960
      S   16 GB         -           0           0           0           0
This partition has the following configuration:
         Tunable    Current    Default  Next Boot
       lgpg_size   16777216          0   16777216
    lgpg_regions       8960          0       8960
  • Large page support is used with a large page pool size of (16777216*8960 = 140.0 GB)
  • The total assigned memory is 54591488 pages (208.25 GB)
  • The pinned memory shows high percentage (pinned memory = pin / size = 42343300 / 54591488 = 77.6% pinned memory)
  • The large pages are responsible for the most amount of this pinned memory (16777216 * 8960/4096) / 54591488 = 67.2% of memory pinned).
  • This large page pinned memory is allocated and put aside at system boot. It does not get released unless you unconfigure the large pages.
  • The system is using (46040778 pages) which means ~ 176 GB in computational memory. Once we subtract the 140GB that is pinned for large pages, we find the system is using 36GB in-memory resources.
  • The working computational memory reached (46040778 / 54591488 = 84.3%) of total memory, hence the total assigned memory is fine enough at the moment.
  • Increasing the large page pool (means increasing the pinned memory) in this system is not recommended.
  • Any increase of the large page pinned memory in this system requires assigning more memory in parallel just to keep the total computational memory consumption not to exceed the recommended values ~ 85% (especially for database servers).

Cheers, Mahmoud M. Elshafey

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"ARM Category":[{"code":"a8m0z0000001fMuAAI","label":"AIX General Support"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
11 January 2021

UID

ibm16253907

Page Feedback