Hanging threads/deadlocks/infinite loops

You can find hanging threads, deadlocks, or inifnite loops through thread dumps.

In some rare exceptions, the JVM may have threads that are not progressing, possibly because of one of the following reasons:

  • Threads are deadlocked.
  • Threads are in an infinite loop.
  • Threads are waiting for an external event.
  • JVM bug.

You can often find these offending threads by taking several successive thread dumps and seeing if there are any threads that seem "stuck" in the same processing point. On Unix and Linux®, issue the following command where pid is the process id of the JVM:


   # kill -3 pid

In Windows, you have to press the CTRL+BREAK keys on the command window that started the JVM.

If you have a hanging or deadlocked thread, in the best case, all they do is tie up a number of scarce worker (execute) threads. There currently isn't any way to kill hung or deadlock threads except to schedule a restart of the JVM.

In the worst case, these offending threads hold on to crucial shared resources (such as database record locks) and are blocking other threads in this or other JVMs. This situation could lead to a system-wide slowdown as more and more threads become blocked behind these offending threads.

If you have infinite looping threads, at best, all they do is make the server node busier. In the worst case, they start to impact the performance of transaction running in that node or they hold critical resources needed by other threads.

Recommendations:

If you suspect a JVM has a hung or looping thread:

  • Take three thread dumps for that JVM. Space the thread dumps a minute apart.
  • Look at the stacktrace for the Default Queue in the successive thread dumps - see if there are any threads that are active and in the same code path in each thread dump.
  • The IBM® Thread and Monitor Dump Analyzer for Java™ is an excellent tool for analyzing the thread dumps. You can download the tool from IBM developerWorks.

    The download page contains excellent information about using the tool for analysis and troubleshooting.

If you suspect transactions are slow across many JVMs:

  • Look in your database for blocking chains - specifically, what sessions are blocking whom. Find out what servers the root blockers are coming from, the types of locks that they are holding and what was the latest SQL run.
  • Identify the JVMs that have the root blockers. You may have to shutdown those JVMs if the blocking sessions are spreading to a system-wide shutdown.
    Note: Since thread dumps are invaluable diagnostic tools, you should be very comfortable taking thread dumps when the need arises. For example, you should occasionally take thread dumps from all JVMs (e.g., all application server instances, all agent/monitor servers) during non-peak processing periods. This gives you a chance to find out where the thread dumps are written to and how to read the thread dumps.