We've seen some odd GC behaviour on our production servers and I was wondering if anyone could point me towards information that explains how the garbage collector works in overload conditions?
Briefly, we're using Websphere 6 on IBM JVM J9 1.5.0, v2.3. The "gencon" garbage collection strategy is in use.
At one point, the application suddenly allocated a very large amount of heap memory and what seemed to happen was that the JVM started a huge garbage collection run that took all of the CPU, effectively hanging the JVM. This continued for several minutes, at which point the JVM was killed and restarted to restore service.
A thread dump revealed that although the maximum Java heap size was 8192m, the "Free Java heap size" was "0 bytes" and the "Allocated Java heap size" was "2,267,613,184" bytes (so approximately 2GB).
Unfortunately verbose GC logging wasn't enabled but the summary information in the thread dump showed the message "j9mm.63 - Set scavenger backout flag=true" repeated 64 times (within 1 second).
I would have expected the heap to grow towards the -Xmx value before such dramatic GC behaviour and I can't find any documentation to explain what the "Set scavenger backout" message really means.
Can anyone shed any light on this?
Lead Architect, Global Synthetic Equity Technology
UBS Investment Bank, London
Pinned topic J9 Garbage Collection Behaviour with Full Heap
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
Updated on 2013-03-12T13:44:16Z at 2013-03-12T13:44:16Z by SystemAdmin
Re: J9 Garbage Collection Behaviour with Full Heap2011-06-15T12:31:29ZThis is the accepted answer. This is the accepted answer.Eoin,
What you are seeing here is the JVM managing to recover just enough space for a single new object and then having to do the same thing again for the next new object. In the worst cases this can go on for a long time as it never fails to allocate.
The IBM JVM will actually force an OutOfMemoryError when it finds it is spending more than 90% of it's time doing GC, which helps to reduce the length of time the JVM will thrash in this way.
If you can attach the javacore file to a post to the forum we can check if this is indeed the case.
Eoin-Woods 2700049AQM2 Posts
Re: J9 Garbage Collection Behaviour with Full Heap2011-06-16T08:24:02ZThis is the accepted answer. This is the accepted answer.
- SystemAdmin 110000D4XK
Re: J9 Garbage Collection Behaviour with Full Heap2013-03-09T10:14:03ZThis is the accepted answer. This is the accepted answer.
- Eoin-Woods 2700049AQM
Did you resolve this issue ? I am also facing the same issue. Can you please let me know how it got resolved. Thanks in advance.
Re: J9 Garbage Collection Behaviour with Full Heap2013-03-12T13:44:16ZThis is the accepted answer. This is the accepted answer.
- SystemAdmin 110000D4XK