Using OMEGAMON to Diagnose Slow JVMs Through Thread Data
bgrusky 3100021EA9 Comment (1) Visits (11890)
Java is one of the most well-known and used programming languages today, which makes Java Virtual Machines (JVMs) integral in today’s application programming. When working with a JVM, you will undoubtedly encounter performance issues at some point. When these problems arise, the thread data provided by the IBM Tivoli OMEGAMON XE on z/OS Monitoring Feature for JVM can be invaluable in resolving the issue.
JVMs split work across multiple threads that can share CPU time to maximize efficiency. To protect data from concurrent updates, these threads can be forced to run in a synchronized section of code. This process is accomplished through locking, in which a thread needs to hold a lock on an object or resource to enable reading of or writing to that resource. Once a thread holds a lock on a synchronized object, any other threads that attempt to access that object, called the contending object, will be forced to wait, and will be placed into a “BLOCKED” state until the thread holding the lock, called the contending thread, completes processing of the resource and releases the lock.
Blocked threads are not always an indicator of poor performance. Since they can be a necessity in multi-threaded execution environments, the locking scheme used should be designed to avoid a performance issue. Illustrated below are deadlocked threads which are an example of inefficient locking leading to reduced performance.
In this example, two threads each own a different lock that the other needs to continue processing. This could cause the JVM to drastically slow down, or cease processing entirely. Without any monitoring tool for diagnostics, the only apparent fix might be restarting the JVM and hoping that the problem does not resurface. However, OMEGAMON Monitoring for JVM can detect a problem like this. The following picture from the enhanced 3270 user interface (enhanced 3270UI), shows the detection of blocked threads:
Each row represents a JVM. Any JVMs with blocked threads will trigger the critical threshold. By selecting the JVM, the specific blocked thread(s) can be displayed:
You can see that both “Thread-6” and “Thread-7” are in the BLOCKED state, and that they are both listed as the Contending Thread for each other.
So far all we have done is diagnose a set of deadlocked threads as the cause of the slow performance. There is one more field which will be crucial in further investigation: the stack trace. By scrolling to the right, the stack trace for the thread can be displayed:
The stack trace tells you specifically what, within the executing code, the thread is currently processing. It also records each of the previous methods going back to the thread’s creation. In the example above, the JVM is running a trivial workload which starts two threads and causes them to deadlock. So the method that created each thread, the run() method of the DeadLockSample class caused threads “Thread-6” and “Thread-7” to enter a dead locked state at lines 11 and 25 of the code respectively. The full stack trace for a thread can be shown by selecting a value in the Stack Trace column. This will open a window with the full stack trace:
In the example above, we see the “Timer-0” thread started with the run() method of the Timer class on line 517, but it is currently within the wait() method of the Object class which invoked a Native Method. This is useful to see because perhaps the problem did not arise within the most recent method, but possibly in a previous one.
With the problem diagnosed, there is more information for the application programmers to use to fix the problem. This information can help those application programmers find synchronized threads that do not acquire locks in a consistent fashion, or threads that consistently read/write similar database entries. Again, these diagnostics are not limited to just investigating deadlocks – thread issues come in multiple forms, and many can be diagnosed with OMEGAMON Monitoring for JVM. If you are interested in learning more about the product, a great place to start is the IBM Knowledge Center (htt
OMEGAMON Monitoring for JVM is available as priced feature for OMEGAMON XE on z/OS V5.3.0 [lin