We've had numerous observations roll into our team that using libhugetlbfs is just too complicated. We don't really think so, so here we summarize the simple steps of using libhugetlbfs. This is intended to follow the KISS principle (short and simple). We do not explain the why's and where-fore's and the more flexible options available in the library.

This page is focused on Power systems.

For additional reading:

See a more in-depth example if you want more details in a step-by-step guide for tuning an application program.
For details on how to leverage libhugetlbfs in a production environment, see this DeveloperWorks paper.
And we've started a simple libhuge FAQ page
The libhugetlbfs HOWTO page is tucked into the tar ball, so it's hard to link to from here. We'll try to get that split out.

Simplistically, there are two common uses.

Backing mallocs or allocates used in the program (done with environment variables)
Backing the uninitialized arrays in a program (done with a couple of re-link directives added to the build step)

Here's some common notes.

No application source code changes are needed. Really.
SLES 10 sp1 and RHEL 5.1 are recommended. libhugetlbfs is not supported on SLES 9 or RHEL 4.
Large pages are 16MB pages on Power systems.
- The terms "large pages" and "huge pages" are the same thing in this context.
- AIX users are used to "large pages".
- Linux users are used to "huge pages".
- Be flexible.
16GB pages (coming later) can be referred to as absurdly large/huge/gigantic pages.
You need to be root to install and set up libhugetlbfs, but not root to use. We assume you have root access here.
There's no simple way to determine how many large pages your application needs. Allocate more than you need when you test. Then adjust.
Using 16MB large pages will not always improve performance. Sometimes it does. Sometimes it doesn't.

If you're curious about 64KB page memory support (on Power systems):

If you're using RHEL 5.1 on Power5+ or Power6 systems, you're automatically using 64KB pages for memory. The 64KB pages provides the majority of the performance gains you might get from using libhugetlbfs. Using libhugetlbfs can provide a couple of percent extra faster performance.

The 64KB page support in the mainline Linux kernel missed the deadlines for being included in SLES 10. SLES 10 remains based on the normal 4KB pages for memory. Changing the default page size is something that would be done on new version boundaries.

AIX provides for the flexible mix and matching of 4KB, 64KB, and 16MB pages. Work continues in the Linux community to add more flexibility to the Linux implementation.

The following are the quick steps for using libhugetlbfs:

libhugetlbfs - short and simple

libhugetlbfs - short and simple

Install libhugetlbfs

On SLES 10 sp1, install the two rpms.

libhugetlbfs-1.0.1-1.4.ppc.rpm
libhugetlbfs-64bit-1.0.1-1.4.ppc.rpm

On RHEL 5.1, install the five rpms

libhugetlbfs-1.0.1-1.el5.ppc.rpm
libhugetlbfs-devel-1.0.1-1.el5.ppc.rpm
libhugetlbfs-devel-1.0.1-1.el5.ppc64.rpm
libhugetlbfs-lib-1.0.1-1.el5.ppc64.rpm
libhugetlbfs-lib-1.0.1-1.el5.ppc.rpm

Or, from sourceforge, download the tar-ball and install. Get the libhugetlbfs tar ball from sourceforge.net into a working directory, for example /usr/local/src

# tar -zxf libhugetlbfs-1.0.1.tar.gz
# ls
.  ..  libhugetlbfs-1.0.1  libhugetlbfs-1.0.1.tar.gz
# cd libhugetlbfs-1.0.1
# make
# make install PREFIX=/usr

There are newer versions of libhugetlbfs on SourceForge which are regularly used by the performance teams. Check with your service provider on whether the newer versions can be used and supported in business critical environments.

Allocate the large pages

In this case, we set up thirty (30) 16MB hugepages. We generally recommend clearing the number of huge pages first (in case there were hugepages previously allocated).

However, if you're looking at using massive numbers of hugepages, this may need to be done on the system boot.

Note that we check the result after we set the thirty (30) 16MB pages. The operating system will try to get as many pages as it can. In this case, 30 is easy to get. If you were to try to get several thousand 16MB pages, you would need that many pages free.

# cat /proc/meminfo | grep HugePage
HugePages_Total:   0
HugePages_Free:    0

# echo  0 > /proc/sys/vm/nr_hugepages
# echo 30 > /proc/sys/vm/nr_hugepages

# cat /proc/meminfo | grep HugePage
HugePages_Total:   30
HugePages_Free:    30

Booting with hugepages

You can of course specify the number of hugepages on the Linux boot command. For example, on a SLES10 Power system, we add another Linux boot option (LinuxLP) in yaboot.conf. This then can be selected on the boot. In this example, we tuck away 100 hugepages on the boot.

# header section
partition = 2
timeout = 80
default = linux
# image section
image = /boot/vmlinux-2.6.16.46-0.12-ppc64
    label = linux
    root = /dev/disk/by-id/scsi-35000c5000639376f-part2
    initrd = /boot/initrd-2.6.16.46-0.12-ppc64
image = /boot/vmlinux-2.6.16.46-0.12-ppc64
    label = linuxLP
    root = /dev/disk/by-id/scsi-35000c5000639376f-part2
    initrd = /boot/initrd-2.6.16.46-0.12-ppc64
    append = "hugepages=100"

Create the necessary pseudo-filesystem

The mkdir is done once on a system. You will need to re-mount the directory after a reboot.

# mkdir /libhugetlbfs
# mount -t hugetlbfs hugetlbfs /libhugetlbfs

Open a second window

To watch the number of HugePages change, open a second window and execute this command:

# watch cat /proc/meminfo

Run your program

Backing the mallocs or allocates

Assuming your program is already built and ready to go. And uses mallocs or allocates. Note that we use "/usr/bin/time -p" to time the execution of the program.

# LD_PRELOAD=libhugetlbfs.so HUGETLB_MORECORE=yes   /usr/bin/time -p ./my_progam

Watch the meminfo window to confirm whether HugePages are being used. You want to allocate more pages than you need the first time you try it, then you can adjust.

Running without the libhugetlbfs backing, you can compare the run-times.

# /usr/bin/time -p ./my_progam

Backing the un-initialized arrays

If you're using the IBM compilers:

# xlc -B/usr/share/libhugetlbfs/ -tl -Wl,--hugetlbfs-link=BDT  my_program.c  -o my_program
# /usr/bin/time -p ./my_program

or if you're using gcc:

# gcc -B/usr/share/libhugetlbfs/ -Wl,--hugetlbfs-link=BDT  my_program.c  -o my_program
# /usr/bin/time -p ./my_program

Watch the meminfo window to confirm whether HugePages are being used. You want to allocate more pages than you need the first time you try it, then you can adjust.

Special Notes

Shared memory with large pages

When it is desired to use shared memory, and 16MB pages, and execute as a non-root user, you will need to add the GroupID (gid) in the /proc/sys/vm/hugetlb_shm_group file (the numeric value) and update the /etc/security/limits.conf file to allow the usage.

Keep in mind that the non-root user will need to be defined as a member of the group defined for access to these large pages.

#<domain>      <type>  <item>         <value>
#
*               soft    memlock         -1
*               hard    memlock         -1

Reserving lots of pages at system boot

If you're using a lot of 16MB pages, you may need to reserve them on the system boot. The longer the operating system runs, the more likely it is that the Linux kernel will touch and use a portion of a 16MB page, which means that 16MB page can no longer be used for large pages.

16MB pages don't always help

Don't be surprised if the 16MB pages don't improve performance. Some programs run faster. Some do not.