Getting more memory in AIX for your Java applications


This article explains how 32-bit Java allocates its heap in AIX, and pertains to all the latest versions of 32-bit IBM Java available for AIX versions 4.3.3, 5.1, and 5.2.

You will learn how to distinguish between various memory areas of a Java process under AIX, and how to tweak their sizes to suit your needs. The information in this article will allow your Java-based applications to break the barrier of 1 GB Java heap with versions of Java prior to 1.4; I'll also describe how Java 1.4 does it automatically for you. For applications that use databases and middleware, this article provides the basis for advanced topics, such as segment collision avoidance.

This article has been updated to include information about Java 1.4, and also to give examples of commands you can use to increase your understanding of the topic. The fence analogy, and much of the enhanced description, were provided by Mark Bluemel. Along with Mark, Ashok Ambati and Greg Hodgins reviewed the article to make it much more readable. The update itself was triggered by feedback from readers like you; thanks to everyone who sent their comments.

Native versus Java heaps

A heap is a storage management structure for tracking and allocating memory. When we talk about Java processes, there are two different types of memory areas, and hence two types of heaps, that must be distinguished.

The Java Virtual Machine (JVM) has an address space, just like any process, from which it can allocate chunks of memory for various purposes. Probably the most important (and definitely the most familiar to Java programmers) of these chunks is the Java heap. The Java heap is used for allocating the Java objects, and you can monitor its usage through verbosegc traces. The size of a Java heap can be explicitly controlled using -Xms and -Xmx command line arguments. This is true for all platforms where Java is supported, hence the name Java heap.

The rest of the chunks are allocated by the JVM on an as-needed basis, and are usually platform dependent. We refer to them collectively as data. From a Java point of view, a more appropriate term for these chunks is native heap, due to its platform-specific nature. For example, on AIX the native heap contains several critical components of a JVM process:

  • Thread stacks for every thread except primordial
  • Buffers, lookup tables, and so on for ZIP-related operations (such as GZIPOutputStream methods)
  • Buffers and structures related to native GUIs underlying Swing/AWT (for example, Motif)
  • Data used by JNI code, such as native database drivers, MQ-Series, and so on
  • Just-in-time (JIT) compiler and Mixed-Mode-Interpreter (MMI) support routines
  • Executable code for JIT-compiled methods

and more. The native heap size cannot be controlled through a command line argument.

On AIX, depending on the Java version in use and the Java heap size selected, the Java heap may or may not itself be allocated from the native heap.

In this article, the term heap, if not further qualified, refers to the Java heap, not the process (native) heap. Also, the term data is used interchangeably with native heap.

The o_maxdata on AIX

Before we can discuss how to allocate larger heaps using Java, we have to understand the process address space in AIX. The information in this section applies to all processes under AIX, including the JVM. See Related topics for links to more information on this topic.

On AIX, the 32-bit address space (= 232 or 4 GB) is divided into 16 segments, each 256 MB (256 MB*16 = 4 GB). When a 32-bit process is loaded in memory, its address space looks Figure 1.

Figure 1.Segments in 32-bit AIX model
Segments in 32-bit AIX model

In the default address-space model, segment 2 contains the user stack, kernel stack and u-block, as well as data (area used by malloc()). This translates to a limit of 256 MB for data, stacks, and u-block for the whole process. Segments 3 through C are free to be used as shared memory, using shmat() or mmap().

If a process needs to allocate more data, it can switch to the large space model, where the native heap starts from segment 3 onwards, and grows to an offset specified by the o_maxdata value in the XCOFF header to a non-zero value. It is essential to understand the significance of o_maxdata to learn how to allocate larger Java heaps.

You can think of o_maxdata to be setting a "fence" in segments 3 through A. Figure 2 shows the fence in action (segments 0, 1 and D-F omitted for clarity).

Figure 2. The fence
The fence
The fence

If the fence moves "up" (that is, the Fence line moves towards Segment 2), we get more space for shared memory and, correspondingly, less space for the native heap. Moving the fence down allows the native heap to grow, while reducing shared memory. For a setting of o_maxdata = N, the fence is placed at 0x30000000+N. For several good reasons, it is recommended to set o_maxdata to a value that is the start of a particular segment, such as 0xn0000000. In this case, the fence sits between segments 2+n and 3+n, which translates to n segments for the native heap, and 10-n segments for shared memory.

If o_maxdata is zero, the entire native heap is allocated from segment 2. This is the default space model; here, all segments from 3 through C (10 segments = 2.56 GB) can be used for shared memory. The native heap size is constrained by the available memory in segment 2, (less than 256 MB). When o_maxdata is set to a non-zero value, we switch to the large space model, which implies the native heap is allocated from segment 3 onwards. The top-most position of the fence in Figure 2 shows the scenario where o_maxdata is greater than zero. Other structures, such as stacks and u-blocks, still get allocated from segment 2.

Segments B and C are used only for shared memory, so o_maxdata can move the fence all the way down to segment A, depicted as the bottom-most position of the fence, leaving segments 3 through A for native heap (8 segments = 2 GB), and B & C (2 segments =512 MB) for shared memory.

As an example, setting n=2 (o_maxdata = 0x20000000) would mean that the fence sits at segment 5, leaving 2 segments (=512 MB) for native heap, and 8 segments (=2 GB) for shared memory.

Java heap allocation values

The preceding discussion might make you think that if you want a large Java heap, you must patch the Java binary to have o_maxdata set to 0x80000000. Unfortunately, things are not so simple.

If you look at the 32-bit Java binary, using the dump command, o_maxdata is already set to 0x80000000. The example listing below shows the output for Java 1.3.1.

$ dump -ov /usr/java131/jre/bin/java


                        ***Object Module Header***
# Sections      Symbol Ptr      # Symbols       Opt Hdr Len     Flags
         4      0x000079bc            627                72     0x1002
Timestamp = "Aug 07 03:17:57 2003"
Magic = 0x1df  (32-bit XCOFF)

                        ***Optional Header***
Tsize        Dsize       Bsize       Tstart      Dstart
0x000056df  0x000002f1  0x00000818  0x10000128  0x20000807

SNloader     SNentry     SNtext      SNtoc       SNdata
0x0004      0x0002      0x0001      0x0002      0x0002

TXTalign     DATAalign   TOC         vstamp      entry
0x0005      0x0003      0x200009e0  0x0001      0x20000984

maxSTACK     maxDATA     SNbss       magic       modtype
0x00000000  0x80000000  0x0003      0x010b        UR

o_maxdata shows up in the last line of the output, and is labeled maxDATA. Yet, an attempt to allocate a heap larger than 1 GB will fail. You can test this very quickly by using the command java -mx1200m -version. This command will fail with the default JVM settings. Note that older versions of Java 1.1.8 and 1.2.2 differed in this behavior. If you are not using the latest service refreshes of your JVM version, you might observe a different behavior.

The variability is because of the dual-personality heap allocation method used by Java. For up to 1 GB of heap, Java uses malloc() to allocate the Java heap. The Java heap is allocated from the native heap. But if you go above 1 GB, or if the environment variable IBM_JAVA_MMAP_JAVA_HEAP is defined, Java reverts to mmap() for the Java heap, which means it uses shared memory. This allows Java to allocate a heap as large as the maximum size of shared memory, or 2.56 GB.

By default, o_maxdata is set to 0x80000000, leaving 2 GB for native heap and 512 MB for shared memory. If you attempt to allocate a Java heap larger than 1 GB, it fails because Java tries to use shared memory for heap, and there is only 512 MB of shared memory available. If you set IBM_JAVA_MMAP_JAVA_HEAP in the environment and try to allocate a heap larger than 512 MB, JVM will be unable to allocate the heap. The solution is to adjust o_maxdata in such a way that the size of shared memory grows large enough to accommodate the Java heap. The next section shows you how to do this.

Stretching the limits

So how do you go to a larger Java heap? You need to change o_maxdata to increase the amount of shared memory address space. You can use the following calculations to come up with the appropriate value for o_maxdata. Supposing you need a maximum heap size of J bytes, you would invoke Java as

java -mxJ <other arguments>

If J is less than 1 GB, and IBM_JAVA_MMAP_JAVA_HEAP is not set, the default setup will suffice. If J is > 1 GB, or if IBM_JAVA_MMAP_JAVA_HEAP is set, use

o_maxdata = 0xn0000000


n = (10 - ceil(J/256M)) 
or 8

, whichever is smaller. The function ceil rounds up the argument to the next integer.

For example, if you need to allocate 1500 MB of heap, we have n = (10 - ceil(1500M/256M)) = (10 - 6) = 4. If you set o_maxdata = 0x40000000, you will be able to allocate the needed size of heap. To change o_maxdata, set the following environment variable:

LDR_CNTRL=MAXDATA=<new o_maxdata value>

The above example would set the following environment variable:


To verify that your calculation is accurate, you can try the following commands:

$ export LDR_CNTRL=MAXDATA=0x40000000 
$ java -mx1500m -version

The command works, indicating that your calculations were accurate. Change 4 to 3 in the above example, and you will see that Java fails to load.

If you need a heap larger than 2.25 GB, some older releases of AIX had problems interpreting 0x00000000. You can use LDR_CNTRL=MAXDATA=0 instead.

So what would be the maximum heap size we would be able to allocate with Java on AIX? Setting o_maxdata to zero will allow all 10 segments to be used for shared memory, letting us allocate up to 2.56 GB of heap. (And even more for Java 1.4!) But, see Balancing memory for some caveats associated with very large heaps.

Reverse-engineering a running process

If you use svmon on a running JVM, you can get a picture that is quite a parallel of Figure 2. For example, given a JVM process with process ID 389354, the following is the output of svmon -P 389354 -m :

  Pid Command          Inuse      Pin     Pgsp  Virtual 64-bit Mthrd LPage
  389354 java             14837     3155      951    14566      N     Y     N

    Vsid      Esid Type Description              LPage  Inuse   Pin Pgsp Virtual
   289aa         d work loader segment               -   6899     0    0  6899
       0         0 work kernel segment               -   6375  3144  951  6375
   1c324         3 work working storage              -    993     0    0   993
   4d8b3         - pers large file                   -    236     0    -     -
   4b772         - work                              -    135     0    0   135
   7c2dc         f work shared library data          -    103     0    0   103
   102e7         2 work process private              -     38     4    0    38
   13504         - work                              -     23     7    0    23
   46471         - pers large file                   -     13     0    -     -
   7637d         - pers large file                   -     10     0    -     -
    4322         1 pers code,large file              -      9     0    -     -
   55f95         - pers large file                   -      2     0    -     -
   3956e         - clnt                              -      1     0    -     -
   23608         7 mmap mapped to sid 4b772          -      0     0    -     -
   5b536         - work                              -      0     0    0     0
   1b666         a mmap mapped to sid 82e1           -      0     0    -     -
   5f557         8 mmap mapped to sid 702df          -      0     0    -     -
   2b76a         - work                              -      0     0    0     0
    82e1         - work                              -      0     0    0     0
   702df         - work                              -      0     0    0     0
   521d6         6 work working storage              -      0     0    0     0
    7701         5 work working storage              -      0     0    0     0
   777bd         b mmap mapped to sid 5b536          -      0     0    -     -
   737bc         9 mmap mapped to sid 2b76a          -      0     0    -     -
   242ca         4 work working storage              -      0     0    0     0

Can you tell what the value of o_maxdata is, by looking at the above output? The segment numbers show up under Esid; the above shows segments 3, 4, 5, and 6 being used for "work," and segments 7, 8, 9, a and b are using mmap. So, svmon seems to indicate that this process was using o_maxdata set to 0x40000000, and was using a heap larger than 1 GB but smaller than 1.25 GB (since only 7 through B are used, hence 5 segments). This is exactly right; the process was started with o_maxdata set to 0x40000000, and with a maximum heap size of 1100m. See Related topics for more examples with svmon.

Java 1.4

If you are new to AIX platform, and the first version of Java you have used on AIX is Java 1.4, you may be wondering what I am talking about. Java 1.4 is more intelligent than its predecessors; it no longer requires you to do calculations and set environment variables. Based on the requested heap size, Java 1.4 launcher sets the appropriate o_maxdata value automatically. The exact behavior of Java 1.4 is a bit different from the one I explained above, and it may change in future releases. You can use svmon to find out for yourself how Java 1.4 and above manipulate o_maxdata. Note, though, that you can still use the techniques you learnt in this article with Java 1.4. If any LDR_CNTRL=MAXDATA values are set in the environment, Java 1.4 will not override it, even if it results in an error during startup. Conversely, if you do want Java 1.4 to manage the heap sizing for you, make sure you remove any LDR_CNTRL=MAXDATA entries from the environment settings.

If you use Java 1.4 on AIX 5.2, another new feature becomes available. Let's see if you can locate what the size of heap is, from the svmon output below (I have removed all lines with Esid marked "-" for clarity):

     Pid Command          Inuse      Pin     Pgsp  Virtual 64-bit Mthrd LPage
  507904 java             58146     3155      951    52487      N     Y     N

    Vsid      Esid Type Description              LPage  Inuse   Pin Pgsp Virtual
   2f549         2 work process private              -  45801     4    0 45801
       0         0 work kernel segment               -   6383  3144  951  6383
   3966e         1 clnt code                         -     18     0    -     -
   5bfb4         c work overflow heap                -     17     0    0    17
   2ffa9         c mmap mapped to sid 23faa          -      0     0    -     -
    bda0         e mmap mapped to sid 1bfa4          -      0     0    -     -
   27fab         b mmap mapped to sid 3bfac          -      0     0    -     -
   2f409         5 mmap mapped to sid 1b544          -      0     0    -     -
   47fb3         7 mmap mapped to sid 7ffbd          -      0     0    -     -
   4bfb0         8 mmap mapped to sid 43fb2          -      0     0    -     -
   3ffad         9 mmap mapped to sid 37faf          -      0     0    -     -
   57fb7         4 mmap mapped to sid 53fb6          -      0     0    -     -
   3f42d         3 mmap mapped to sid 5ffb5          -      0     0    -     -
   2bfa8         a mmap mapped to sid 33fae          -      0     0    -     -
   63fba         6 mmap mapped to sid 6bfb8          -      0     0    -     -
   23f8a         f mmap mapped to sid 13f86          -      0     0    -     -
   13fa6         d mmap mapped to sid 1ffa5          -      0     0    -     -

Except for 0, 1 and 2, all the other segments are being shown as "mmap". This is the Very Large Memory Model (VLMM). By setting LDR_CNTRL=MAXDATA to a "DSA" value, the segments D, E and F can also be used for mmap. Though VLMM is available in AIX 5.1 as well, Java uses it only in AIX 5.2 onwards. This results in a total of up to 3.25 GB available for Java heap. To see this in action, try the following command:

 java -mx3328m -version

Congratulations, you just broke the 3 GB barrier! You need to be on at least AIX 5.2, at least Java 1.4, and LDR_CNTRL=MAXDATA must not be set.

If the above svmon output is compared to Figure 1, you will note that the shared library code and data are no longer in segments D and F respectively. So if you are trying to use a tool like tprof to profile the application, you will no longer be able to get library- and subroutine- specific information. Plus, all the caveats associated with large heaps, covered by the next section, are applicable here as well.

See the SDK Guide accompanying Java 1.4 for more details.

Balancing memory

There are at least three factors you need to consider before tweaking the heap size to maximum. In no particular order, they are:

Garbage collection (GC) performance
The GC needs to scan the whole heap, so the time taken to scan the heap grows as the heap gets larger. See Related topics for information on GC tuning, including tuning the heap size as one of the critical parts.
Segment collisions
If an in-process component of the JVM attempts to use shared memory, and the Java heap is using all the available shared memory, you can run into a situation where JVM or the other component overwrite each others' structures. This can lead to misleading error messages and random crashes (or worse). Segment collision is a major topic by itself, and is beyond the scope of this article. Stay tuned for an article on segment collisions, soon to be published here at eServer Developer Domain.
Running out of native heap
Occasionally, you may run out of native heap. If you set o_maxdata to 0, only a portion of segment 2 is available, which may not be enough for the stuff Java needs to allocate. If you are using JNI code that consumes a lot of memory, even larger values of o_maxdata may not prove sufficient.

The symptoms of running out of native heap vary. You might not be able to start threads beyond a certain limit, or you may get error messages indicating allocation failures in native code. The verbosegc output usually shows Java heap to still be available, but OutOfMemory exceptions are still thrown.

To find out if you are running low on native heap, observe the "InUse" column of svmon output. This column lists the pages, each of size 4 KB, being used. Observing the value for each segment belonging to native heap gives you a good idea of native heap usage. The maximum value possible for InUse for a segment is 256MB/4KB = 65536. A good debugging technique for detecting memory leaks in native code is to use svmon -P <pid> -m -i <interval>, which takes a snapshot at a specified interval.

In some cases, you can tune the configuration to avoid running out of native heap:

  • Try reducing the stack size for threads (the -Xss parameter). However, deeply nested methods may force a thread stack overflow in case of insufficient stack size.
  • For middleware products, if you are using an in-process version of the driver, it is usually possible to find an out-of-process driver that can have a significant effect on the native memory requirements. For example, you can use Type 4 JDBC drivers (DB2's "Net" drivers, Oracle's "Thin" drivers), MQ-Series can be switched from Bindings mode to Client mode, and so on. Refer to the documentation for the products in question for more details. Also see "Segment collisions" above.
Alternatively, if you are using a pure Java application and are on AIX 5.1 or higher, you can use 64-bit Java instead. The 64-bit Java is available for both Java 1.3.1 and Java 1.4, and you won't have to worry about running out of memory again for quite some time.


Striking the right balance between Java and native heap becomes unavoidable as you go to larger heaps. Java 1.4 goes a long way towards making the configuration work easier for you. But if multiple components are involved, you might still have to take the address space management into your own hands. I hope this article has provided you with enough information to get started.

Downloadable resources

Related topics

  • The IBM Diagnostics Guides provide exhaustive details about Java debugging on all supported platforms, straight from the IBM Java team.
  • You can find the developer kits, tutorials, whitepapers and more at the developerWorks Java Technolozy zone. The Java 1.4 SDK Guide (or README in earlier versions) accompanies the Java Developer Kits and provides up-to-date information about each Java release.
  • AIX Documentation deals with the memory models and various APIs mentioned in this article.
  • Looking for help? Visit IBM eServer technical support for a comprehensive set of resources, all focused on helping you learn about, choose, implement and use the right IBM e-server solution for your IT infrastructure needs.


Sign in or register to add and subscribe to comments.

Zone=AIX and UNIX
ArticleTitle=Getting more memory in AIX for your Java applications