A hung thread is a common error in J2EE™ applications. According to the WebSphere Application Server Information Center:
A hung thread can result from a simple software defect, such as an infinite loop, or a more complex cause such as a resource deadlock. System resources, such as CPU time, might be consumed by this hung transaction when threads run unbounded code paths, such as when the code is running in an infinite loop. Alternately, a system can become unresponsive even though all resources are idle, as in a deadlock scenario. Unless an end user or a monitoring tool reports the problem, the system may remain in this degraded state indefinitely.
The hang detection option for WebSphere Application Server is turned on by default. You can configure a hang detection policy to accommodate your applications and environment so that potential hangs can be reported, providing earlier detection of failing servers. When a hung thread is detected, WebSphere Application Server notifies you of the event so that you can troubleshoot the problem.
The thread monitor is a new notification feature in WebSphere Application Server V5.1.1 that enables each server to detect if there is a potentially hung thread, by determining if a specific thread has been executing too long within a unit of work. The thread monitor checks all managed threads in the system, such as Web container and Object Request Broker threads. Unmanaged threads, which are threads created by applications, are not monitored by this feature.
This article will explain how to modify a Java client for WebSphere Application Server V5.1.1 to automatically help debug potentially hung threads on a server, including how to program a listener to detect the specific event notification that indicates a hung thread, and how to programmatically react to the notification. This article will also provide a skeleton administrative client and custom service that can be easily modified for any WebSphere Application Server Management Bean (MBean). For more information on configuring the hung thread detection policy, see WebSphere Application Server Information Center:
Develop a client to determine a Thread_Hung problem
The event notification that the client is to listen for is the
TYPE_THREAD_MONITOR_THREAD_HUNG (Thread_Hung) from the
com.ibm.websphere.management.NotificationConstants class. The Thread_Hung event is triggered by the detection of a potentially hung thread. When this event occurs, a thread dump will automatically execute to help you (or IBM support) determine the exact line of the code where the thread is hung. To develop a client to listen and react to a Thread_Hung notification, follow these steps:
- Create a Java client instance.
A Java client instance (either an administrative client program or a custom service) needs to be created to contain the code that registers a listener and handles the Thread_Hung event notification. The only requirement for the Java client is to implement the NotificationListener class. (See How to extend the WebSphere Management System (and create your own MBeans) for examples of a Java client.)
- Register a listener.
The Java client needs to register as a listener for the Thread_Hung notifications. Listing 1 is an example of how an object can register itself for the Thread_Hung event notifications emitted from any MBean. MBeans are the building blocks for WebSphere Application Server V5 system administration and monitoring based on the Java Management Extensions (JMX) specification.
adminServicevariable is an application server API object that interfaces with the MBean, and is either an AdminClient object for the adminstrative client program, or an AdminService object for the custom service. If you want to listen to more event notifications, you can add more
- Handle the event.
The NotificationListener class defines the handleNotification method for handling JMX event notifications. This is the key function in determining when to execute a thread dump on a potential problem server with hung threads. Listing 2 shows an implementation of handleNotification that executes a Java thread dump on the problem server.
The sample code in Listing 2 initially parses the source of the notification to determine the server that detected a possible hung thread. The program retrieves the JVM MBean of the server that sent the notification and executes the dumpThreads() method to produce a Java thread dump. The thread dump information can be found in either a new javacore file, located in the Java working directory when using the IBM JDK (for Windows, AIX and Linux), or in the native_stdout.log file located in the server logs directory for the other JDKs.
Sample client program
In the included
HungThreadDump.javadownload file is a complete administrative client program that listens for the Thead_Hung event notification and reacts by taking a thread dump on the appropriate server. Download this file to your client machine. After changing
SoapPort to the appropriate values for your configuration, you can compile and run the client program as follows:
- To compile, execute this command (set the variables
WAS_HOMEto the appropriate locations):
%JAVA_HOME%\bin\javac -classpath %WAS_HOME%\lib\admin.jar; %WAS_HOME%\lib\wsexception.jar; %WAS_HOME%\lib\jmxc.jar HungThreadDump.java
- To run, execute this command (set the variables
WAS_HOMEto the appropriate locations):
%JAVA_HOME%\bin\java" -classpath %WAS_CLASSPATH%; %WAS_HOME%\classes; %WAS_HOME%\lib\admin.jar; %WAS_HOME%\lib\wasjmx.jar;." HungThreadDump
This client will stay in a wait state listening for the notification (or for the user input) to end the program. The client is designed to output all the information about the notification and then execute a thread dump. It is not designed to work when global security is enabled, but the client can easily modify the createAdminClient function to add the security parameters to the connectProps object. (For more information, see the Application Server API section in the WebSphere Application Server V5.1 Information Center.)
Sample custom service program
In the included
CustomServiceThreadHungMBean.javadownload file is a complete custom service program that listens for the Thead_Hung notification and reacts by invoking a dumpThreads() method on the server's JVM MBean. This produces a thread dump on the appropriate server. Download this file to the WebSphere Application Server machine. No modifications are required. Compile and run this program using following commands:
- To compile, execute the following command (set the variable
WAS_HOMEto the appropriate locations):
%JAVA_HOME%\bin\javac -classpath %WAS_HOME%\lib\runtime.jar; %WAS_HOME%\lib\admin.jar; %WAS_HOME%\lib\wsexception.jar; %WAS_HOME%\lib\jmxc.jar CustomServiceThreadHungMBean.java
Package the class file into a JAR file; for example:
%JAVA_HOME%\bin\jar.exe cvf hung.jar
- To install the custom service in a WebSphere Application Server cell:
- Select the server to which you want to add the custom service (dmgr, nodeagent or application server).
- In the Additional Properties section, select Custom Services.
- To create a new custom service, select New.
- Enter the appropriate information about the MBean; for example:
- Select Startup so the custom MBean starts every time the server starts.
- Supply the Classname of this custom service:
- Display Name is user defined and can be any value.
- Classpath specifies the location of the JAR file; this value can be either an absolute path or symbolic.
Figure 1. Define custom service
If the WebSphere Application Server install is a Base install, then the custom service must be configured on each server to monitor for hung threads. If the Base installs are federated to a Network Deployment cell, then only the dmgr process requires the custom service. The custom service running in the dmgr process will monitor all managed processes (dmgr, nodeagent, application server, and JMS server). The custom service is not designed to work with Java 2 security.
A Java client Thread_Hung event notification program can help administrators automatically monitor possibly hung threads in an entire cell, as well as execute thread dumps. This tool can provide the user with the necessary information for debugging specific code problems, and can also provide the initial information IBM Support needs to begin helping to fix the potential issue. Included with this article is a client program with a skeleton JMX administrative client, and a custom service for WebSphere Application Server. These program skeletons can easily be redesigned to listen for any event notification, based on filter design and query string, and to react as required, based on handleNotification method implementation.
|Java code samples||hung_thread_java.zip ( HTTP | FTP )||4 KB|
- WebSphere Application Server V5.1 Information Center
- System Administration for WebSphere Application Server V5 -- Part 4: How to extend the WebSphere Management System (and create your own MBeans
- To learn more, visit the developerWorks WebSphere Application Server zone for technical documentation, how-to articles, education, downloads, product information, technical support resources, and more.
- Browse for books on these and other technical topics.
- WebSphere forums. Product-specific forums where you can ask questions and share your opinions with other WebSphere users.
- developerWorks blogs. Ongoing, free-form columns by software experts, with space for you to add your comments.