We've seen some odd GC behaviour on our production servers and I was wondering if anyone could point me towards information that explains how the garbage collector works in overload conditions?
Briefly, we're using Websphere 6 on IBM JVM J9 1.5.0, v2.3. The "gencon" garbage collection strategy is in use.
At one point, the application suddenly allocated a very large amount of heap memory and what seemed to happen was that the JVM started a huge garbage collection run that took all of the CPU, effectively hanging the JVM. This continued for several minutes, at which point the JVM was killed and restarted to restore service.
A thread dump revealed that although the maximum Java heap size was 8192m, the "Free Java heap size" was "0 bytes" and the "Allocated Java heap size" was "2,267,613,184" bytes (so approximately 2GB).
Unfortunately verbose GC logging wasn't enabled but the summary information in the thread dump showed the message "j9mm.63 - Set scavenger backout flag=true" repeated 64 times (within 1 second).
I would have expected the heap to grow towards the -Xmx value before such dramatic GC behaviour and I can't find any documentation to explain what the "Set scavenger backout" message really means.
Can anyone shed any light on this?
Lead Architect, Global Synthetic Equity Technology
UBS Investment Bank, London
This topic has been locked.
4 replies Latest Post - 2013-03-12T13:44:16Z by SystemAdmin
Pinned topic J9 Garbage Collection Behaviour with Full Heap
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
Updated on 2013-03-12T13:44:16Z at 2013-03-12T13:44:16Z by SystemAdmin
Re: J9 Garbage Collection Behaviour with Full Heap2011-06-15T12:31:29Z in response to Eoin-WoodsEoin,
What you are seeing here is the JVM managing to recover just enough space for a single new object and then having to do the same thing again for the next new object. In the worst cases this can go on for a long time as it never fails to allocate.
The IBM JVM will actually force an OutOfMemoryError when it finds it is spending more than 90% of it's time doing GC, which helps to reduce the length of time the JVM will thrash in this way.
If you can attach the javacore file to a post to the forum we can check if this is indeed the case.
Eoin-Woods 2700049AQM2 PostsACCEPTED ANSWER
Re: J9 Garbage Collection Behaviour with Full Heap2011-06-16T08:24:02Z in response to SystemAdminThanks for the reply Ben. I've attached one of the javacore files we collected during these incidents. Any thoughts most appreciated.