Is your WebSphere Application Server not starting due to shared class cache problem?

Body

I've seen some problems come in to WebSphere Application Server L2 for support that are related to shared class cache so I wanted to talk about the shared class cache a little and the main problem we see and offer a potential solution.  Shared class cache was introduced in WebSphere Application Server 6.1 and this information applies to that version and versions up to 8.0.

First, the main benefits of sharing class cache is to help improve start up time. It's possible you could also see a performance improvement depending on your setup and JVM (Java^(tm) Virtual Machine) usage. To better understand how things work, it's best to know the lookup order.  A JVM running with Class Sharing uses the following order:

   1. Classloader cache
   2. Parent
   3. Shared cache
   4. Filesystem

A JVM without class cache sharing would be the same except you would omit #3.  Classes are read from and written to the cache using the public Helper API, which has been integrated into the IBM implementation of java.net.URLClassLoader. As a result, any classloader that extends java.net.URLClassLoader gets class sharing support for free.

Being the main benefit is to help improve start up time, you can probably guess that any problems with the cache would likely be apparent on start up.  The main problem you could see would be a hang on start up. A hang can occur when the shared cache becomes corrupt.  It's difficult to determine what corrupted the cache, so the simplest remedy is to disable shared caching or destroy and recreate it if there are no JVMs attached.  You should not assume a hang on start up means the cache is corrupt.  There's a few things to check (that I'll talk about a few paragraphs later) that can help determine that it's corrupt.  The most likely candidate for corruption is an application or SDK crashing while the cache is being updated. "Crashing" could also mean forced shutdown; "kill -9" for example.

It's reasonable to think a utility such as printStats or printAllStats could give you an indication if the cache is corrupt or not.  It could give you an indication, but that would not always be the case. The best method to determine if the cache is corrupt is to check the stderr for a couple error messages.  Although these error messages are not always present, if you do see them, you know there is corruption.  

JVMSHRC247E Error shared class cache file header eyecatcher is invalid
JVMSHRC097E Shared cache "<cachename>" is corrupt. No new JVMs will be allowed to connect to the cache. Existing JVMs can continue to function, but cannot update the cache.
JVMSHRC256I Persistent shared cache "<cachename>" has been destroyed

Checking the integrity of the shared class cache on start up would not make sense being the purpose of the cache is to improve start up times.  

Long story short, if you see the startup of the Deployment Manager (DMGR), Server or Node Agent hang it could be due to a corrupted shared class cache.  Look in the systemout.log see if the DMGR, Server or Node Agent is waiting for initialization. Sometimes you may see an actual error message pointing to the cache as the problem in the log.  Lastly, you should look to see if any of the JVM error mentioned previously can be found.  When this is the case, you could either disable class cache sharing or destroy the cache (assuming no attached JVMs) and restart to resolve the hang.  
 

• To disable shared class cache for WebSphere Application Server, add -Xshareclasses:none as a generic JVM argument through the admin console.  See more information about this in the information center on Tuning the IBM virtual machine for Java.   
• To destroy the cache, use the following command
  C:\.....\sdk\jre\bin>java -Xshareclasses:name=<cachename>,destroy
  JVMSHRC010I Shared Cache "cache1" is destroyed
  Could not create the Java virtual machine.

NOTE: You can not delete the cache while any of the JVM's are running and using the cache. If there are JVMs attached, run the following script: WAS_HOME/bin/clearClassCache.bat for Windows or WAS_HOME/bin/clearClassCache.sh for UNIX.^{(registered tm)}

Once the cache is destroyed, you can then recreate the cache or it will be created automatically on startup.

The message "Could not create the Java virtual machine" is printed by the Java launcher by each utility because a JVM is not started. This is not an error since none of the utilities (apart from expire) start the JVM. You can get a complete list of all valid suboptions to -Xshareclasses by typing
java -Xshareclasses:help. You can use some of these utilities to manage active caches.

Reference:
Java technology, IBM style: Class sharing