Topic
9 replies Latest Post - ‏2010-10-01T00:00:41Z by Horace.Imbruglia
SystemAdmin
SystemAdmin
2736 Posts
ACCEPTED ANSWER

Pinned topic Linux IBM JVM 1.5.0 does not release pages back to OS when heap shrinks

‏2006-03-14T19:12:29Z |
I've noticed that the the IBM 1.4.2 and 1.5.0 JVMs are good at proactivly shrinking the heap if the space is nolonger needed. This is especially helpful for people who run multiple JVMs on a single machine. JVMs may run for months on end, and one JVM may only need extra memory for only an hour... Leaving the JVM memory footprint at the highest-spike over the entire month makes provisioning memory for servers with multiple JVMs very very conservative.

Unfortunately, although the IBM JVM shrinks it's heap, it does not appear to ever release this memory back to the operating system.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14458 kislo 16 0 1097m 836m 3664 S 0.0 21.7 0:07.28 java

836megs of resident memory, but the heap has acctually been shrunk down to 4megs over the past 20 minutes. Is there any way the JVM can release these pages back to the OS so they can be reclaimed? I realize eventually the OS will swap them to disk since they're being actively accessed... but the kernel is shrinking the disk-cache to unacceptably small levels before paging the unused JVM pages out to disk. The total throughput of the system would be dramatically increased with the unused JVM pages going towards disk cache immediately.

I have created some dummy code to simulate this, you need to run it as:

java -Xmx1024M Test4

Test4.java:

import java.util.*;

public class Test4 {

public static final void main(String[] s) {
Runtime rt = Runtime.getRuntime();
Vector v = new Vector();
int count=0;
while (true) {
while (true) {
count++;
StringBuffer sb = new StringBuffer(1048576);
for (int i=0; i<104857; i++) {
sb.append("0123456789");
}
v.addElement(sb);
System.out.println("Allocated "1*count*2"megs"); //unicode
if ((count*2)>800) {
break;
}
}
System.out.println("Sleeeping!");
System.out.println("FreeMemory:"+rt.freeMemory());
System.out.println("TotalMemory:"+rt.totalMemory());
System.out.println("UsedMemory:"+(rt.totalMemory()-rt.freeMemory()));
try {Thread.sleep(10000);} catch (InterruptedException ie){}
v = new Vector();
count=0;
while (true) {
System.gc();
System.out.println("GCED!");
System.out.println("FreeMemory:"+rt.freeMemory());
System.out.println("TotalMemory:"+rt.totalMemory());
System.out.println("UsedMemory:"+(rt.totalMemory()-rt.freeMemory()));
try{Thread.sleep(10000);} catch (InterruptedException ie){}
}
}
}

}


I believe what might be happening is the IBM JVM allocates the entire -Xmx1024M + some native memory from the OS, and writes it's stack pages at the end of the space. The OS only allocates pages to the JVM on-demand, when they are written to... but when the JVM stops using them (when it shrinks it's heap) the OS has no idea (since they're in the middle of the JVM allocated space) that they can be reclaimed. Unfortunately... if the pages aren't ever released back to the OS, that's going to make major memory scalability issues for us with the IBM JVM.

Thanks,
-Joe
Updated on 2010-10-01T00:00:41Z at 2010-10-01T00:00:41Z by Horace.Imbruglia
  • ChrisBailey
    ChrisBailey
    190 Posts
    ACCEPTED ANSWER

    Re: Linux IBM JVM 1.5.0 does not release pages back to OS when heap shrinks

    ‏2006-03-14T23:41:28Z  in response to SystemAdmin
    Hi Joe:

    You are correct, the Java heap is allocated as a single memory block large enough to cover the maximum heap size setting (-Xmx). We initially only commit to memory for the the minimum heap size. As heap expansion occurs, the artificial "heap size" boundary is moved upwards as we commit to more of the allocated memory.
    When heap shrinkage occurs we de-commit to the memory, but do not free it - we can't do this as the Java heap is required to be contiguous memory and we need to own the same memory in case we expand later.

    When sizing the memory usage for a server, it is imperative (for GC performance) that no paging/swaping occurs, and therefore there needs to be enough physical memory to cover the virtual memory needs of all of the processes. In the case of a Java process, this is the maximum Java heap size plus memory to cover the VMs requirements and memory to underpin some of the Java objects used by the application (Threads, Inflaters/Deflaters, etc).

    Chris
    • SystemAdmin
      SystemAdmin
      2736 Posts
      ACCEPTED ANSWER

      Re: Linux IBM JVM 1.5.0 does not release pages back to OS when heap shrinks

      ‏2006-03-15T16:48:05Z  in response to ChrisBailey
      Ouch.. That's going to be tough for us. We've been big fans of the IBM's JVM since 1.1.8, but this one is going to be tough to swallow. The memory footprint of the 1.5.0 JVM on 64bit is sooo much higher than we're used to (1.3.0 jvm on 32bit). I realize 64bit is going to be much higher, but initial tests looks like we would be looking atleast quadrupling the memory requirement, and these machines only hold a max of 16GB physical memory. Ouch! Do you know if there is any chance this requirement that the pages be contiguous might be going away in the future? Or some workaround to get the pages back to the OS? The sun JVM is obviously designed very differently, but it does seem to be contracting the OS memory commitment when shrinking the heap.

      I do have a follow up question, how does native memory / JIT memory usage get allocated? The whole reason I got onto this was because we tried converting a production server to the 64bit 1.5.0 IBM JVM from 32bit 1.4.2 IBM JVM. It didn't go so well. After 12 hours the 1.5.0 setup was running 3.2 gigs physical and 1.6 gigs swap (it was NOT a pretty sight). The "java heap" peaked around 800megs for a few minutes, but settled around 400megs. Putting the Sun 1.5.0 64bit JVM in there the OS footprint bounces around, but averages 800megs. (huge difference!). The application doesn't use any native code. It does make AWT calls though. I'm running against an Xvfb server (using -Djava.awt.headless=true resulted in even worse memory usage). The other thing the application does, is an enormous amount of rhino execution (http://www.mozilla.org/rhino/). Rhino does have it's own optimizer, which creates many many custom classes from different classloaders. In our case, it's probably creating new classes (and very quickly destroying) for most exceute runs. I checked with a kill -QUIT, and the javaheap dump shows that nearly all (except for the most recent) rhino compiled classes have been GCed... But it seems like the OS footprint of the JVM keeps going straight up, while the java heap is staying fairly stable.

      Does memory allocated for JIT code on very temporal classes, not get released back? I don't have definitive proof that this is what's happening, but it's one of like 3 theories I have as to where all the memory outside the java heap could be going. The fact that sun's jvm stays pretty stable at 800megs after 48 hours is a good sign it's something specific with the IBMJVM. I also saw nearly identical memory footprint on the 64bit IBM 1.4.2 JVM compared to the 64bit IBM 1.5.0 JVM.

      Thanks,
      -Joe
      • ChrisBailey
        ChrisBailey
        190 Posts
        ACCEPTED ANSWER

        Re: Linux IBM JVM 1.5.0 does not release pages back to OS when heap shrinks

        ‏2006-03-20T22:11:04Z  in response to SystemAdmin
        Hi Joe:

        The way memory is allocated to back the maximum Java heap size has not changed - the allocation has worked this way certainly as far back as 1.1.8 (the earliest release I worked on). I would expect most, if not all, other vendors to work the same way.
        In order to have non-contiguous heap segments you really need to have the ability to have non-contiguous objects. This is a much discussed subject but not something that is currently implemented. Should that change then the requirement on a contiguous memory could be removed.

        Its interesting that you mention that use use Rhino, which creates generated classes.
        One of the changes between 1.4.2 and 5.0 is in the way that class unloading is handled. Classes are unloaded to a certain extent in "batch", which means that they aren't necessarily removed the moment they are eligable to be. This means that additional memory can be held in terms of underlying native class structures and associated JIT compiled code. It should be possible to see if this is the case or not from the Verbose GC output (which lists the number of classloaders unloaded).
        I would also suggest upgrading to 5.0 Service Release 1, which is now available. There's been some other changes to class loading which may make a difference.

        Chris
      • ChrisBailey
        ChrisBailey
        190 Posts
        ACCEPTED ANSWER

        Re: Linux IBM JVM 1.5.0 does not release pages back to OS when heap shrinks

        ‏2006-03-21T14:25:35Z  in response to SystemAdmin
        Hi Joe:

        I forgot to ask, do you have multiple instances of the JVM on the same partition/machine yourself?
        If you do, you'll benefit in terms of memory footprint by using shared classes.

        Chris
    • SystemAdmin
      SystemAdmin
      2736 Posts
      ACCEPTED ANSWER

      Re: Linux IBM JVM 1.5.0 does not release pages back to OS when heap shrinks

      ‏2008-10-13T16:30:50Z  in response to ChrisBailey
      I have a related question regarding physical memory requirements for servers running the IBM JVM 1.5 64-bit and hope this an appropriate place to ask it (I'm new to developer works). We are moving from 32bit JVM 1.4 to 64bit JVM 1.5 in an AIX 5.3 environment & using Webmentods (going from 6.1 to 7.1). We are trying to estimate the server physical memory requirements in the target environment. I'm assuming the your comments about having enough phys. memory to avoid paging/swapping hold in our environ. as well.

      My question is this - We've estimated the virtual memory sizes for the heap and native memory for each of the 64bit jvms on the server, and totaled these virtual memory requirements for the server. Are there guidelines for using this info to estimate the phys memory requirement? Would we be allocating excessive physical memory by providing enough phys memory to back the sum of the jvms' virtual memory on a 1:1 basis? I know that some phys memory will also be used for the aix operating system and other functions, so this approach is not trully a 1:1 memory allocation, but I still can't help but think that this approach is an overallocation of physical memory. And yes I know that more physical memory can only help performance but our goal is to provide the minimum amount of phys memory needed - ie not overallocate it. Any help or guidelines would be approieciated.
    • SystemAdmin
      SystemAdmin
      2736 Posts
      ACCEPTED ANSWER

      Re: Linux IBM JVM 1.5.0 does not release pages back to OS when heap shrinks

      ‏2009-04-02T15:31:39Z  in response to ChrisBailey
      Chris:
      Does this apply to all IBM JDK for alll platforms? or just on LINUX?
      • SystemAdmin
        SystemAdmin
        2736 Posts
        ACCEPTED ANSWER

        Re: Linux IBM JVM 1.5.0 does not release pages back to OS when heap shrinks

        ‏2009-04-03T11:00:27Z  in response to SystemAdmin
        I'm assuming you are asking about shared classes here.

        Shared classes is available on all the IBM 5.0 & 6.0 SE Java Runtimes.

        Ben Hardill
        • SystemAdmin
          SystemAdmin
          2736 Posts
          ACCEPTED ANSWER

          Re: Linux IBM JVM 1.5.0 does not release pages back to OS when heap shrinks

          ‏2009-04-03T13:18:28Z  in response to SystemAdmin
          Sorry for not making it clear. I was referring to memory not being released after shrinkage. is this the case for all platform.
  • Horace.Imbruglia
    Horace.Imbruglia
    1 Post
    ACCEPTED ANSWER

    Re: Linux IBM JVM 1.5.0 does not release pages back to OS when heap shrinks

    ‏2010-10-01T00:00:41Z  in response to SystemAdmin
    ChrisBailey wrote:
    {$content}

    Thanks for your effort! It's comprehensive.