Working Around Long Class Unload Times In WebSphere Application Server

Body

Upon investigation of a WebSphere Application Server performance issue, you may find that the poor performance correlates with long classloader unload times. To check if this is indeed a problem, enable verbose GC and note the times of poor performance. Then correlate the times of poor performance to the time stamps in the verbose GC data:

Example of long class unload time as seen in the verbose GC output (default log is the native_stderr.log in WebSphere):

...
<classunloading classloaders="3058" classes="6490" timevmquiescems="0.000" timetakenms="25253.772" />
...
 

In this example, 25253ms=25seconds. To determine if a response time is too long, consider the end user of the application. Is the pause time unacceptable for a user to wait for a response? If the answer is "yes", then the pause time is too long.

If you find that time spent unloading classes is excessive ("classunloading"), you may be inclined to add "-Xdisableexplicitgc".  However, this is likely to lead to a very large memory footprint and a possible OutOfMemory problem due to native memory.  Instead, try the following:

1) Check for:

APAR IV49664: SLOW CLASS UNLOADING SCAN TIME

2) Check for:

APAR IV47984: LONG GC PAUSE TIMES WHEN USING WIDE CLASS HIERARCHIES

3) Try decreasing the number of reflection classloaders:

-Dsun.reflect.inflationThreshold=0
IBM Potential native memory use in reflection delegating classloaders

4) Try triggering tenured GC when nursery becomes full by setting the following generic JVM argument:

-Xgc:classUnloadingKickoffThreshold=N
Garbage collection in WebSphere Application Server V8, Part 1: Generational as the new default policy

5) Try forcing System.gc()s after every N scavenges by setting the following generic JVM argument:

-Xgc:maxScavengeBeforeGlobal=N
Excessive Native Memory Usage by DirectByteBuffers

6) Try removing JIT optimizations that add to class overhead. Set the following generic JVM argument:

Java 7.1 and Newer:

-Xjit:disableCHOpts

Java 7.0 and Older (**The JVM will fail to start if the following argument is enabled on Java 7.1 or Newer):

-Xjit:disableCHOpts,disableCHTable

7) Trace for the reason behind the abundance of classes by setting the following generic JVM argument (Note: This will incur performance overhead and is only meant for diagnostic purposes):

-Xtrace:trigger=method{java/lang/Runtime.gc,jstacktrace},print=mt

8) Try a different GC policy, (perhaps balanced):

-Xgcpolicy:<value>

Topics in the product documentation:
Garbage Collection Options
When to use the Balanced garbage collection policy

9) Write your own classloader:

Why write your own class loader?
Demystifying class loading problems, Part 1: An introduction to class loading and debugging tools

title image (modified) credit: (cc) Some rights reserved by Mysitemyway.com