IBM Support

Large page definition, purpose, application and system configuration

Question & Answer


Question

What is the large page, its purpose, application, and system configuration needed to use it?

Answer

This document discusses the large page support highlighting the following:
What is the large page?
  • AIX® Virtual Memory Manager offers virtual memory segments partitioned into fixed-size units called memory pages.
  • AIX® operating system supports different virtual memory page sizes as 4 KB, 64 KB, 16 MB, and 16 GB.
  • The memory page types of 4 KB and 64 KB are the default working memory pages on AIX®.
  • Most of the applications and the operating system programs work with the default pages.
  • The large page is the virtual memory pages with a size of 16 MB that POWER5+® and later support.
  • Use the command # pagesize -af to print all the page size values supported on the system.
  • AIX® supports large page usage by both 32-bit and 64-bit applications.
  • The svmon command with -P flag lists a process with different pages sizes statistics.
Virtual memory pages size on AIX®
Page Size Default Svmon mark Description
4 KB YES s Small page size
64 KB YES m Medium page size
16 MB NO L Large page size
16 GB NO S Huge (supreme) page size
What is the large page purpose?
  • The main large page purpose is to improve application performance, enhancing throughput.
  • Especially for the application that uses large amounts of virtual memory with many TLB misses (translation lookaside buffers).
  • TLB is a memory cache used to reduce the time taken by the system for translating the program's virtual addresses for instructions.
  • A single large page that is being constantly referenced remains in memory, eliminating the possibility of swapping out several small pages.
  • The larger page size can benefit workloads that use large chunks of data like database servers.
  • Some application shows noticeable improvement when using large pages.
  • The overall CPU consumption for a given task is reduced.
  • Less time spent spinning on locks makes more real work gets done.

 

In order to enable the large page support, a configuration is needed for both the operating system and the application, by involving an environment variable called LDR_CNTRL or by using the ldedit command then configuring (building) a large page pool by using the vmo tool, check details in the following section. Check also Large page best practices, considerations, and monitoring for more information about the need for the configuration, look in the section "Configuration is needed to use large pages"

  

Application configuration for large page  -  marking application to use large pages
In order to let an application uses the large page support, you need to use one of the following options or the application might need both to be configured:
  • Application works with large pages via identifying its executable file (marking the executable file header to request a large page), the ldedit command makes it possible to mark or unmark an application as a large-page program.
  • Application works with large pages also by setting the environment variable LDR_CNTRL to request large page support.
Ldedit command options:
The ldedit command can mark a binary header to use the large page support, the same command with another flag can unmark the marked binary.
  • # ldedit -blpdata, ​enables large page support by marking a specific binary (executable) to request large pages.
  • # ldedit -bnolpdata, disables large page support by clearing the large page flag.
LDR_CNTRL variable options:
The environment variable LDR_CNTRL could be used to enable/disable the large page support as well.
  • Use LDR_CNTRL=LARGE_PAGE_DATA=Y to enable large page support.
  • Use LDR_CNTRL=LARGE_PAGE_DATA=N to disable large page support.
  • The LDR_CNTRL=LARGE_PAGE_DATA=M enables large page support in mandatory mode, this option makes the application that is executed uses large pages in the mandatory mode for its data and heap segments, check details and the difference in the following section.
LDR_CNTRL multiple options:
  • You can separate multiple options on the LDR_CNTRL environment variables by using an ‘@’ character.
  • The following LDR_CNTRL variable setting requests large page usage along with the maxdata option:
    LDR_CNTRL=MAXDATA=0x80000000@LARGE_PAGE_DATA=Y
Large pages modes:
When you use large page support, the application's heap segments could be backed by large pages, and some of them backed by 4 KB pages.
  • Advisory mode: When there are not enough large pages available to back the application's heap segments. This mode allows the application to have some of its heap segments backed by large pages and some of them backed by 4 KB pages. So the application doesn't crash due to lack of large pages. This mode is applied by using the LDR_CNTRL environment variable with LARGE_PAGE_DATA=Y.
  • Mandatory mode:  In this mode, the heap segments are not allowed to use the 4 KB pages, if an application requests a heap segment and there are not enough large pages to satisfy the request. The allocation request fails, which causes most applications to terminate with an error. This mode is applied by using LDR_CNTRL=LARGE_PAGE_DATA=M.
LDR_CNTRL vs ldedit:
The differences between the environment variable LDR_CNTRL and the ldedit command and how they affect your environment.
  • Marking the needed binary files that need to use the large page support by using the ldedit command is better than using the environment variable LDR_CNTRL system-wide, so you can control large pages usage per command.
  • Users are advised to be cautious in their use of the environment variable LDR_CNTRL to specify large page usage.
  • Performance tests showed there can be a significant performance loss in environments where a number of shell scripts or small, short-running applications are invoked.
  • One example showed a shell script’s execution-time increase over 10 times when the large page environment variable was specified.
  • Customers are advised to set the large page environment variable around specific applications that can benefit from large page usage.
  • Check Large page best practices, considerations, and monitoring for more information about how to configure your LDR_CNTRL environment variable with the large page support appropriately, look in section "LDR_CNTRL and non-large pages programs notes"

LDR_CNTRL=LARGE_PAGE_DATA=N option overrides blpdata flag that could be marked on a binary by ldedit command. So if a specific binary uses large pages by using ldedit command with blpdata option, you still can override this setting by using LDR_CNTRL environment variable with LARGE_PAGE_DATA=N option while the executable file header remains marked as a large-page program from ldedit command, you might run ldedit command later with bnolpdata option just to change it back to the default setting (as a non-large-page program).

Application configuration summary:
LDR_CNTRL variable and ldedit command settings
LDR_CNTRL=LARGE_PAGE_DATA=Y
  • Enables large page support in advisory mode.
  • Better for applications not to crash due to lack of large pages.
LDR_CNTRL=LARGE_PAGE_DATA=N
  • Disables large page support.
  • Overrides blpdata flag set by ldedit.
LDR_CNTRL=LARGE_PAGE_DATA=M
  • Enables large page support in mandatory mode.
  • The heap segments are not allowed to use the 4 KB pages. 
ldedit -blpdata <command> Enables large pages for a binary by marking its header to use the large pages.
ldedit -bnolpdata <command> Disables large page support for a binary by removing the header set by ldedit -blpdata 
Shared memory segments:
AIX® provides a shared memory facility for use by applications consisting of multiple processes that need to manipulate the same data.
  • To use large page for shared memory segments, use # vmo -p -o v_pinshm=1
  • This option allows the pinning of shared memory segments.
Now, as we have discussed the application configuration needed to make a specific application a large-page program, you still need to configure your operating system to work with the large page support via creating a large page pool, check details hereinafter.
System configuration for large page  -  building a large page pool:
  • Configuring your system to use a large page by using vmo command is a must.
  • Specify the amount of memory that needs to get allocated for a large page pool.
  • lgpg_regions and lgpg_size parameters are used to specify the large page memory amount.
  • The system doesn't hold any memory allocated for the large page physical pool by default.
  • Use the vmo command to configure the size of the large page needed.
  • See the following example, the command allocates 1 GB of memory for the large page pool:
   # vmo -r -o lgpg_regions=64 -o lgpg_size=16777216
  • No reboot is needed.
  • Note: 16777216 Bytes x 64 = 1 GB
  • Run the bosboot command so that the vmo command that you previously ran takes effect following the next system boot.
 
lgpg_regions:
  • Specifies the number of large pages to get reserved.
  • The default value is 0, which means no large page configured.
  • Check the output of # vmo -h lgpg_regions for more information.
  • It's a dynamic parameter, hence no reboot is needed.
  • lgpg_size parameter must also be configured in addition to this parameter.
lgpg_size:
  • Specifies the size in bytes of the hardware-supported large page used.
  • See the output of # vmo -h lgpg_size
  • It's a dynamic parameter, hence no reboot is needed.
  • Supported on systems from Power4 onwards.
  • lgpg_regions must be set to a nonzero value in addition to this parameter.
A figure shows a large page pool with n large pages
image 7671
  • The large page pool consists of multiple of large page chunks.
  • The single large page size is 16 MB.
  • This size is specified with the tunable lgpg_size.
  • With large page, lgpg_size will be 16777216 = 16 MB.
  • The number (n) represents the number of these large pages.
  • This number is specified with the tunable lgpg_regions.
  • Large page pool size = lgpg_size x lgpg_regions = 16MB x Large pages count = 16MB x n
A figure summarizes the large page support configuration
image 7657

Cheers, Mahmoud M. Elshafey

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"ARM Category":[{"code":"a8m0z0000001fMuAAI","label":"AIX General Support"}],"ARM Case Number":"","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Document Information

Modified date:
11 January 2021

UID

ibm16371654

Page Feedback