Enabling, disabling and testing the large page support

How To

Summary

The system is not configured for large page support by default, a configuration is needed on both the operating system and application.

Steps

This document discusses how to enable and disable the AIX® large page support with examples, make sure to test the large page use impact on both the operating system performance and application throughput.

Testing the large page
Enabling large page
Disabling large page
Database procedures

Testing the large page support

First, you need to check the total assigned memory on your server to confirm if this server can satisfy the size of a new large page pool that will be pinned in memory.

Using the following command to get the currently assigned memory on your partition (svmon and other tools could be used as well to get the same information):

^{# prtconf -m

Memory Size: 8192 MB}

Confirm that your system supports large page by using pagesize command (the output shows 16M, means this system supports large page):

^{# pagesize -af

4K

64K

16M

16G}

Use vmstat -l to display the large pages total pool size and available pages statistics. You will find a new section displayed called large-page at the end of the output.

We see alp and flp both on 0 (means no large pages configured):

alp indicates the number of large pages currently in use.

flp indicates the number of large pages on the large page freelist.

^{kthr    memory           page                   cpu        large-page

------- ---------------- ---------------------- ------------------- ----------

r   b      avm     fre re pi po fr sr cy us sy id wa pc ec alp    flp

5   1 16446108 5241821 0   0   0   2   4   0 2 1 97 0 0.3 5.4   0      0}

Use the following command in the below snippet as well to check the lgpg_regions and lgpg_size tunables current values, both tunables on 0 indicates the large page support is disabled.

^{# vmo -a |egrep "lgpg_regions|lgpg_size"}

^{lgpg_regions = 0

lgpg_size = 0}

For more details about the current, default, minimum, and maximum values for both tunables, use # vmstat -L, check the following command output in the below snippet:

We see each tunable is depending on the other tunable, which means large page support needs both tunables to be configured.

^{# vmo -L lgpg_regions -L lgpg_size}

^{NAME CUR DEF BOOT MIN MAX UNIT TYPE DEPENDENCIES

lgpg_regions 0 0 0 0 8E-1 D lgpg_size

lgpg_size 0 0 0 0 16M bytes D lgpg_regions}

How to enable large page support?

Assign an appropriate enough memory for the new large page pool.
Configure the large page pool with both tunables lgpg_regions and lgpg_size.
Configure ldedit or LDR_CNTRL or both, confirm with your application team.
Set the needed user(s) capabilities for full large page support configuration.
Allow pinning of shared memory segments if needed.

The system is not configured for large page support by default, hence you need to specify the amount of memory needs to get allocated for large pages via both tunables lgpg_regions and lgpg_size.

Using the vmo command dynamically (building the large page pool) (No need to reboot the system to work with large page support).

The following example allocates 1 GB of memory to the large page physical memory pool. The output will be similar to the following:

^{# vmo -p -o lgpg_regions=64 -o lgpg_size=16777216}

^{Setting lgpg_size to 16777216 in nextboot file

Setting lgpg_regions to 64 in nextboot file}

^{Warning: although the change concerns a dynamic tunable,

bosboot should be run to optimally restore these settings at reboot}

^{Setting lgpg_size to 16777216

Setting lgpg_regions to 64}

Note the -p flag makes changes apply to both current and reboot values.

Run the bosboot command so that the vmo command that you previously ran takes effect following the next system boot.

Run vmstat again to check the changes, you can see that there are 0 active large pages (alp), and 64 free large pages (flp), this means we have configured this system with a new large page pool as 1 GB, but still all these pool pages are free as alp is 0.

^{# vmstat -l}

^{kthr memory page cpu large-page

----- ------------- ----------------------- ---------------------- ----------

r b avm fre re pi po fr sr cy us sy id wa pc ec alp flp

1 1 445723 49009 0 0 0 1 1 0 0 0 99 0 0.01 0.6 0 64}

We can also use KDB with vmstat subcommand to get the same information (it displays the large page pool size in GB):

^{(0)> vmstat | grep 16M}

^{16M number of frames : 00000040 1.0GB}

^{16M frames pinned : 00000040 1.0GB

16M system pinnable frames remaining: 00000000 0.0MB

16M user pinnable frames remaining : 00000000 0.0MB}

The following is just for test purpose, the large pages are supposed to be working with a production application or databases that support large pages use.

Using Ldedit

To enable a specific command to use large pages using the ldedit command, use the following procedure:

Testing the large page with sar command (sar collects, reports, or saves system activity information).

Run ldedit command (marking sar binary header as a large-page program) to let sar uses large page support:

^{# ldedit -blpdata /usr/sbin/sar

ldedit: File /usr/sbin/sar updated.}

Now sar command is marked to used large pages, start a test with sar command, command in below snippet gets 600 snapshots (each every one second) and execution in the background, we see the command process ID as 7536648:

^{# sar 1 600 > /tmp/sar.test&

[1] 7536648}

Use tail command against sar.test file to confirm we're having a running sar in the background:

^{# tail -f /tmp/sar.test}

^{09:11:13 0 0 0 100 0.00 0.5

09:11:14 0 0 0 100 0.00 0.4

09:11:19 0 0 0 100 0.00 0.4

09:11:20 0 0 0 100 0.00 0.4}

Check the following output from the vmstat -l command. We can see alp with 16 (means there are 16 large pages used by the application):

^{# vmstat -l}

^{kthr memory page cpu large-page

----- ------------- ----------------------- --------------------- -----------

r b avm fre re pi po fr sr cy us sy id wa pc ec alp flp

1 1 445723 49009 0 0 0 1 1 0 0 0 99 0 0.01 0.6 16 48}

The command svmon can be used as well, it shows the same large pages consumption from vmstat -l.

Check the PageSize section (the line starts with L), this line is confirming that sar command is using 16 large pages).

^{# svmon -Pnsm 7536648}

^{-------------------------------------------------------------------------------

Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB

7536648 sar 94116 80628 0 94081 N N Y

PageSize Inuse Pin Pgsp Virtual

s 4 KB 52 4 0 17

m 64 KB 1783 943 0 1783}

^{L 16 MB 16 16 0 16}

^{S 16 GB 0 0 0 0}

Note that 16MB changes from ‘N’ (not used) to ‘Y’.

M would indicate that the application has made 16M page usage Mandatory.

Using LDR_CNTRL

You can use the LDR_CNTRL environment variable to enable the large page support. Checking the current large pages statistics (output shows no large pages used):

^{# vmstat -P 16M

pgsz memory

siz avm

16M 128 0}

Creating a test user named testlp and switch into its shell:

^{# mkuser testlp

# su - testlp}

Exporting the LDR_CNTRL environment variable. Capabilities of the user are set by the parameters CAP_BYPASS_RAC_VMM and CAP_PROPAGATE:

^{# export LDR_CNTRL=LARGE_PAGE_DATA=Y}

^{# chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE testlp}

Testing normal operating system binaries, testing sar and lparstat commands in a 100 seconds window so we can find the time to capture their memory pages consumption by using svmon command with -P flag (both commands are using large pages as seen in the line starting with L):

^{# sar 1 100 > sar1.out&

[7] 6684860}

^{# lparstat 1 100 > laprstat1.out &

[8] 4456652}

^{# svmon -P 6684860

Pid Command Inuse Pin Pgsp Virtual 64-bit

6684860 sar 93244 80308 0 93212 N

PageSize Inuse Pin Pgsp Virtual

s 4 KB 10076 9524 0 10044

m 64 KB 1102 328 0 1102

L 16 MB 16 16 0 16

S 16 GB 0 0 0 0}

^{# svmon -P 4456652

Pid Command Inuse Pin Pgsp Virtual 64-bit

4456652 lparstat 93235 80308 0 93211 N

PageSize Inuse Pin Pgsp Virtual

s 4 KB 10067 9524 0 10043

m 64 KB 1102 328 0 1102

L 16 MB 16 16 0 16

S 16 GB 0 0 0 0}

Executed other programs until reaching the maximum utilization of the large page pool (check below vmstat output, it shows no free large pages, avm = siz):

^{# vmstat -P 16M

pgsz memory

siz avm

16M 128 128}

Running additional program to check the pages type it uses (the output shows L Inuse as 0 hence this process doesn't use large pages, this is due to the large page being full already from previous programs sar and lparstat)

^{# mpstat 1 200 > mpstat.out&

[13] 9699408}

^{# mpstat -P 9699408

Pid Command Inuse Pin Pgsp Virtual 64-bit

9699408 sar 27742 14772 0 27708 N

PageSize Inuse Pin Pgsp Virtual

s 4 KB 10078 9524 0 10044

m 64 KB 1104 328 0 1104

L 16 MB 0 0 0 0

S 16 GB 0 0 0 0}

This means the non-large-pages programs have used the large pages while they are not supposed to.

Testing a non-large page program

Testing an operating system binary, the cfgmgr command (this is just for test purpose)

In this test, we involved cfgmgr binary (a very basic operating system command) to use the large pages and if the execution time of the cfgmgr command changed when using the large pages.

The time command output shows 20.06 seconds as total execution time for a cfgmgr command using the default operating system pages.

^{# time /usr/sbin/cfgmgr}

^{real 0m20.06s}

Now, mark the cfgmgr binary header to let it use the large pages using ldedit command with blpdata flag:

^{# ldedit -blpdata /usr/sbin/cfgmgr

ldedit: File /usr/sbin/cfgmgr updated.}

Execution of the cfgmgr command shows a higher execution time (101.07 seconds) when using the large pages exceeding 5 times the time taken when using the default pages (the non-large-pages).

^{# time /usr/sbin/cfgmgr}

^{real 0m101.07s}

Setting cfgmgr binary back as non large-page program (default) (unmarking the binary header):

^{# ldedit -bnolpdata /usr/sbin/cfgmgr

ldedit: File /usr/sbin/cfgmgr updated.}

Cfgmgr command works better with the default memory pages than with the large pages.

Check Large page best practices, considerations, and monitoring for more information about how to configure your LDR_CNTRL environment variable with the large page support appropriately, look in section "LDR_CNTRL and non-large pages programs notes"

How to disable large page support?

Terminate any applications use large pages or wait until they end.
Unset any LDR_CNTRL environment variable configuration.
Unmark all binaries marked with ldedit -blpdata.
Set back both lgpg_regions and lgpg_size to 0.
Remove the configured user(s) capabilities back to default.
Set back pinning of shared memory segments to default.

Use the following vmo command to disable large page support (set both lgpg_regions and lgpg_size to 0, this will be deleting the large page pool freeing the pinned memory portion).

You need to be sure the applications that are using the large pages are terminated (execution ended) already, to be able to disable the large pages.

^{# vmo -p -o lgpg_regions=0 -o lgpg_size=0}

^{Setting lgpg_size to 0 in nextboot file

Setting lgpg_regions to 0 in nextboot file}

^{Warning: although the change concerns a dynamic tunable,

bosboot should be run to optimally restore these settings at reboot}

^{Setting lgpg_size to 0

Setting lgpg_regions to 0}

Run the bosboot command so that the vmo command that you previously ran takes effect following the next system boot.

What if any application is currently using large pages?

Setting both tunables lgpg_regions and lgpg_size to 0 while an application uses large pages will fail with the following error:

vmo: 1485-124 Error setting lgpg_size lgpg_regions
The system was unable to completely satisfy the request

However, this change will set the next boot values to 0 successfully.

Setting lgpg_size to 0 in nextboot file
Setting lgpg_regions to 0 in nextboot file

Also, note that this change indeed decreased the large page pool size by decreasing the lgpg_regions tunable to a number equivalent to the alp value (the number of large pages currently in use)

You still will have your system with large page configuration enabled and you still will need to wait until the execution of this application that uses the large pages ends then to re-run the # vmo -p -o lgpg_regions=0 -o lgpg_size=0 command again to completely delete the large page pool freeing the memory portion that was pinned.

Run the bosboot command so that the vmo command that you previously ran takes effect following the next system boot.

^{# bosboot -a

bosboot: Boot image is 51228 512 byte blocks.}

Using the following command to confirm this partition is large page support disabled (both tunables lgpg_regions and lgpg_size = 0):

^{# vmo -a |egrep "lgpg_regions|lgpg_size"}

^{lgpg_regions = 0

lgpg_size = 0}

Test again the enabled large page binary in the example above (sar).

^{# sar 1 600>/tmp/sar.out&

[1] 8192248}

Using the svmon command, we see 0 Inuse within 16 MB large page, even if the binary is large page support enabled, sure it doesn't use large page without large page pool configured:

^{# svmon -Pnsm 8192248}

^{Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB

8192248 sar 28627 15092 0 28593 N N N

PageSize Inuse Pin Pgsp Virtual

s 4 KB 51 4 0 17

m 64 KB 1786 943 0 1786

L 16 MB 0 0 0 0

S 16 GB 0 0 0 0}

Note that 16MB changed from 'Y' to ‘N’ (not used).

Set back sar command to large page support disabled so you can completely remove all related large pages configuration from your system (un-marking the binary header):

^{# ldedit -bnolpdata /usr/sbin/sar

ldedit: File /usr/sbin/sar updated.}

If you set the LDR_CNTRL environment variable with LDR_CNTRL=LARGE_PAGE_DATA=Y or LDR_CNTRL=LARGE_PAGE_DATA=M and you need to disable the large page support, you will need to unset the value using LDR_CNTRL=LARGE_PAGE_DATA=N, then to remove the configured user capabilities back to default.

NOTES!

Any test for a specific command to use the large page support in this document was a test just to explain the large page configuration needed from both the application and operating system sides.
The large page support is used with production applications that are certified to use large pages.
The sar command works better with non-large pages, the same with all operating system programs.
Contact IBM AIX® Support for any large page issues.

Configuration summary in this table:

Large page status	Enabled	Disabled
ldedit or LDR_CNTRL	Use ldedit command with blpdata option (this will mark individual binaries as needed) OR Use LDR_CNTRL=LARGE_PAGE_DATA=Y (advisory mode) The LDR_CNTRL=LARGE_PAGE_DATA=M (mandatory mode)	If a specific binary header marked with the ldedit -blpdata option, unmark the header with ldedit -bnolpdata OR If LDR_CNTRL variable configured, use LDR_CNTRL=LARGE_PAGE_DATA=N LDR_CNTRL=LARGE_PAGE_DATA=N overrides ldedit command blpdata flag
vmo	lgpg_regions and lgpg_size tunables must be configured.	Set lgpg_regions and lgpg_size tunables to 0.

Procedures to enable large page support for database

Confirm with the database team the appropriate number of the large pages database can work with, there are some calculations depending on the SGA size to get the number of the large pages needed to be configured.
Additional large pages could still be needed to be configured for other segments of the database binary.
Configure the large page pool using the following command:

^{# vmo -p -o lgpg_regions=<num_of_large_pages+num_of_additional_pages> -o lgpg_size=16777216}

Run the bosboot command so that the vmo command that you previously ran takes effect following the next system boot.
Marking the main database binary XCOFF header might be needed to be configured using ldedit command with blpdata option.

^{# ldedit -blpdata <database_binary>}

Export the environment variable LDR_CNTRL, confirm with the database team if there are further LDR_CNTRL options that might be needed and if the database should be working with mandatory or advisory mode, see below example using mandatory mode:

^{#export LDR_CNTRL=LARGE_PAGE_TEXT=Y@LARGE_PAGE_DATA=M}

Specifying the LARGE_PAGE_TEXT=Y option indicates that the loader might attempt to use large pages for the text segment of the process.
The LARGE_PAGE_TEXT=Y option is only valid for 64-bit processes on a 64-bit kernel.
The data of an application that is backed by large pages remain in physical memory until the application completes.
If you increased the large page pool size, you might need to restart the application to pick up the additional large pages.
Allow pinning of shared memory segments, using the following command:

^{# vmo -p -o v_pinshm=1}

Capabilities of the user are set by the parameters CAP_BYPASS_RAC_VMM and CAP_PROPAGATE.
For a non-root user, you must enable the CAP_BYPASS_RAC_VMM capability also CAP_PROPAGATE with the chuser command in order to use large pages.
The following example demonstrates how to grant the CAP_BYPASS_RAC_VMM capability also CAP_PROPAGATE, allowing the database to use Large Page Support, where <user> is the non-root user or the database user (database admin) that needs to use the large page support.

^{# chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE <user>}

A security access control mechanism prevents unauthorized applications from using large page physical memory. It also prevents unauthorized users from using large pages for their applications.
If large pages (16 MB) are used for SGA pinning, any database admin changes to SGA size are tightly coupled to corresponding AIX® vmo lgpg_regions changes.
An imbalance between SGA size and the lgpg_regions allocation might result in suboptimal performance.
Contact the database support team to confirm the appropriate procedures valid for your environment.

Procedure for Db2 database

When large pages are enabled for the database memory area, the self-tuning memory manager (STMM) does not tune the overall database memory configuration. Assign a specific setting to DATABASE_MEMORY based on performance requirements. STMM tunes the internal memory areas (bufferpool, locklist, shared sort, package cache) to make optimal use of the memory that is assigned to the database memory area.

Because database memory tuning that STMM normally performs is disabled, enabling large pages should only be considered for well-defined workloads that have relatively static database memory requirements.

To enable large page support in Db2 database systems on AIX operating systems:

Configure your AIX server for large page support by issuing the vmo command (build the large pool).
Run the bosboot command so that the vmo command that you previously ran takes effect following the next system boot.
Issue the vmo command with this flags: # vmo -p -o v_pinshm=1
Use the db2set command to set the DB2_LARGE_PAGE_MEM registry variable to DB, then start the Db2 database manager.
- For example:
- db2set DB2_LARGE_PAGE_MEM=DB
- db2start
Contact the Db2 support team to confirm the appropriate procedures valid for your environment.

Cheers, Mahmoud M. Elshafey

Document Location

Worldwide

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"ARM Category":[{"code":"a8m0z0000001fMuAAI","label":"AIX General Support"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}}]

Tips

Enabling, disabling and testing the large page support

How To

Summary

Steps

Document Location

Was this topic helpful?

Document Information

UID

Share your feedback

Need support?