Topic
4 replies Latest Post - ‏2013-03-12T13:44:16Z by SystemAdmin
Eoin-Woods
Eoin-Woods
2 Posts
ACCEPTED ANSWER

Pinned topic J9 Garbage Collection Behaviour with Full Heap

‏2011-06-09T12:58:51Z |
We've seen some odd GC behaviour on our production servers and I was wondering if anyone could point me towards information that explains how the garbage collector works in overload conditions?

Briefly, we're using Websphere 6 on IBM JVM J9 1.5.0, v2.3. The "gencon" garbage collection strategy is in use.

At one point, the application suddenly allocated a very large amount of heap memory and what seemed to happen was that the JVM started a huge garbage collection run that took all of the CPU, effectively hanging the JVM. This continued for several minutes, at which point the JVM was killed and restarted to restore service.

A thread dump revealed that although the maximum Java heap size was 8192m, the "Free Java heap size" was "0 bytes" and the "Allocated Java heap size" was "2,267,613,184" bytes (so approximately 2GB).

Unfortunately verbose GC logging wasn't enabled but the summary information in the thread dump showed the message "j9mm.63 - Set scavenger backout flag=true" repeated 64 times (within 1 second).

I would have expected the heap to grow towards the -Xmx value before such dramatic GC behaviour and I can't find any documentation to explain what the "Set scavenger backout" message really means.

Can anyone shed any light on this?

Best wishes,

Eoin.
--
Eoin Woods
Lead Architect, Global Synthetic Equity Technology
UBS Investment Bank, London
Updated on 2013-03-12T13:44:16Z at 2013-03-12T13:44:16Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    2736 Posts
    ACCEPTED ANSWER

    Re: J9 Garbage Collection Behaviour with Full Heap

    ‏2011-06-15T12:31:29Z  in response to Eoin-Woods
    Eoin,

    What you are seeing here is the JVM managing to recover just enough space for a single new object and then having to do the same thing again for the next new object. In the worst cases this can go on for a long time as it never fails to allocate.

    The IBM JVM will actually force an OutOfMemoryError when it finds it is spending more than 90% of it's time doing GC, which helps to reduce the length of time the JVM will thrash in this way.

    If you can attach the javacore file to a post to the forum we can check if this is indeed the case.

    Ben Hardill
    • Eoin-Woods
      Eoin-Woods
      2 Posts
      ACCEPTED ANSWER

      Re: J9 Garbage Collection Behaviour with Full Heap

      ‏2011-06-16T08:24:02Z  in response to SystemAdmin
      Thanks for the reply Ben. I've attached one of the javacore files we collected during these incidents. Any thoughts most appreciated.

      Cheers,

      Eoin.
      • SystemAdmin
        SystemAdmin
        2736 Posts
        ACCEPTED ANSWER

        Re: J9 Garbage Collection Behaviour with Full Heap

        ‏2013-03-09T10:14:03Z  in response to Eoin-Woods
        Hi Eoin,
        Did you resolve this issue ? I am also facing the same issue. Can you please let me know how it got resolved. Thanks in advance.

        Thanks
        Mahesh
        • SystemAdmin
          SystemAdmin
          2736 Posts
          ACCEPTED ANSWER

          Re: J9 Garbage Collection Behaviour with Full Heap

          ‏2013-03-12T13:44:16Z  in response to SystemAdmin
          If you can share the logs(javacore & -verbose:gc), we can help you with problem analysis.

          Thank you