webMethods Knowlegebase : OOM when purging during load-testing using UM v9.7 Build 76185 (1766553)

Problem

Customer had a load test in their Beta environment that was going well until a queue (/esb/testQA/testReply) was attempted to be purged.

It had roughly 30,000 messages in it at the time.

This caused the realms to be unresponsive and one (UM node1) to shut down with an OOM error.

The heap dump MAT reports one instance of:

"com.pcbsys.foundation.persist.fMixedEventManager" loaded by "sun.misc.Launcher$AppClassLoader @ 0x6800023a8" occupies 4,962,763,328 (98.74%) bytes.

The memory is accumulated in one instance of "com.pcbsys.foundation.collections.fPageArray$fPage[]" loaded by "sun.misc.Launcher$AppClassLoader @ 0x6800023a8".

Further Information via Heap Dump:

Looking through the heap dumps we have the following traces which means there was certainly rest being used.

It looks based on timestamps that the heap dump actually sent the realm into a OOME, but the heap was already fragile and by the looks heavily fragmented.

There were events being recovered via the cluster which seem to be persistent then there were a whole raft of reliable messages sitting in the heap which cannot go anywhere until a purge is done.