Developing a client to determine a hung thread problem

This article explains how to modify a Java client with a new notification feature in WebSphere Application Server V5.1.1 to automatically help debug potentially hung threads .

James Kochuba, Support Staff Engineer, IBM

James Kochuba photoJames Kochuba is the technical team leader for IBM support personnel whose primary focus is systems management and workload management. James' role requires focus on customer issues to help guide both support engineers and customers toward problem source identification and, ultimately, to problem resolution.



15 December 2004

Introduction

A hung thread is a common error in J2EE™ applications. According to the WebSphere Application Server Information Center:

A hung thread can result from a simple software defect, such as an infinite loop, or a more complex cause such as a resource deadlock. System resources, such as CPU time, might be consumed by this hung transaction when threads run unbounded code paths, such as when the code is running in an infinite loop. Alternately, a system can become unresponsive even though all resources are idle, as in a deadlock scenario. Unless an end user or a monitoring tool reports the problem, the system may remain in this degraded state indefinitely.

The hang detection option for WebSphere Application Server is turned on by default. You can configure a hang detection policy to accommodate your applications and environment so that potential hangs can be reported, providing earlier detection of failing servers. When a hung thread is detected, WebSphere Application Server notifies you of the event so that you can troubleshoot the problem.

The thread monitor is a new notification feature in WebSphere Application Server V5.1.1 that enables each server to detect if there is a potentially hung thread, by determining if a specific thread has been executing too long within a unit of work. The thread monitor checks all managed threads in the system, such as Web container and Object Request Broker threads. Unmanaged threads, which are threads created by applications, are not monitored by this feature.

This article will explain how to modify a Java client for WebSphere Application Server V5.1.1 to automatically help debug potentially hung threads on a server, including how to program a listener to detect the specific event notification that indicates a hung thread, and how to programmatically react to the notification. This article will also provide a skeleton administrative client and custom service that can be easily modified for any WebSphere Application Server Management Bean (MBean). For more information on configuring the hung thread detection policy, see WebSphere Application Server Information Center:


Develop a client to determine a Thread_Hung problem

The event notification that the client is to listen for is the TYPE_THREAD_MONITOR_THREAD_HUNG (Thread_Hung) from the com.ibm.websphere.management.NotificationConstants class. The Thread_Hung event is triggered by the detection of a potentially hung thread. When this event occurs, a thread dump will automatically execute to help you (or IBM support) determine the exact line of the code where the thread is hung. To develop a client to listen and react to a Thread_Hung notification, follow these steps:

  1. Create a Java client instance.
    A Java client instance (either an administrative client program or a custom service) needs to be created to contain the code that registers a listener and handles the Thread_Hung event notification. The only requirement for the Java client is to implement the NotificationListener class. (See How to extend the WebSphere Management System (and create your own MBeans) for examples of a Java client.)
  2. Register a listener.
    The Java client needs to register as a listener for the Thread_Hung notifications. Listing 1 is an example of how an object can register itself for the Thread_Hung event notifications emitted from any MBean. MBeans are the building blocks for WebSphere Application Server V5 system administration and monitoring based on the Java Management Extensions (JMX) specification.

    Listing 1

    Click to see code listing

    try
    {
    	//   Create an Object for the MBean search string we want to find. 
    	//   The * is used as a wildcard search allowing this to not be locked into a very specific MBean.
    	ObjectName query = new ObjectName("WebSphere:*");
    	//   Create a filter object so this listener will only listen to specific MBean notifications, Thread_Hung.
    	NotificationFilterSupport filter = new NotificationFilterSupport();
    	//   Adding the Thread_Hung notification to the filter.
    	filter.enableType(NotificationConstants.TYPE_THREAD_MONITOR_THREAD_HUNG);
    	//   Adding a listener to multiple MBeans listening only for the filtered notifications.
    	//   Note: adminService variable is the interface to the WebSphere administration functions
    	adminService.addNotificationListenerExtended(query, this, filter, null);
    }
    //Catch any exception thrown from the above actions and print to the system out to debug. 
    //Because this fails to properly add a notification to the interested MBean we exit the program.
    catch (Exception e)
    {
    	//   Print the error and stack information to the system out location for debug purposes.
    	System.out.println(e);
    	e.printStackTrace();
    	//   Exit the program, we can pass any number noting the reason.
    	System.exit(-1);
    }

    The adminService variable is an application server API object that interfaces with the MBean, and is either an AdminClient object for the adminstrative client program, or an AdminService object for the custom service. If you want to listen to more event notifications, you can add more filter.enableType code lines.

  3. Handle the event.
    The NotificationListener class defines the handleNotification method for handling JMX event notifications. This is the key function in determining when to execute a thread dump on a potential problem server with hung threads. Listing 2 shows an implementation of handleNotification that executes a Java thread dump on the problem server.

    Listing 2

    Click to see code listing

    public void handleNotification(Notification ntfyObj, Object handback)
    {
        //   Create a String object, problemServer, representing the MBean source information 
        //	  emitted in a notification that matched the filter.  The source string is a chain 
        //	  of WebSphere objects (type and name) where a comma separates each object pair for 
        //   the specific MBean. The type "process" in the source chain will determine the server
        //   process who sent the notification.
        String problemServer = ntfyObj.getSource().toString();
        //	  Find the index number in the source string where type "process" is found.
        int processIndex = problemServer.indexOf("process");
        //	  Remove all the WebSphere objects before the process index number
        problemServer = problemServer.substring(processIndex);
        //	  Find index number for the end of the object pair
        int commaIndex = problemServer.indexOf(",");
        //	  Check on the index number because the process pair might be the last object in the chain.
        //   Therefore, no parsing of the string needs to be done.
        if (commaIndex >= 0)
        {
            problemServer = problemServer.substring(0,commaIndex);
        }
        //	  Creates an Object for the JVM MBean so we can issue a thread dump.
        ObjectName MBean = null;
        try 
        {
            //   Create a String called query that will be used for a search string.
            String query = "WebSphere:type=JVM," + problemServer + ",*";
            //   Cast the query String into an ObjectName to meet the required input parameter type
            ObjectName queryName = new ObjectName(query);
            //   Returns all MBeans that meet the search string.  Note the adminService object is interface
            //   into the WebSphere MBean server.  This can be either an AdminClient or AdminService
            //   object based on the client program.
            Set s = adminService.queryNames(queryName, null);
            //   Pulls the MBean outside the Set object assuming the search string was specific enough to 
            //   only get one MBean.
            if (!s.isEmpty())
            {
                MBean = (ObjectName) s.iterator().next();
            }
            else
            {
                return;
            }
        }
         //   Catch any exception thrown from the above actions and print to the system out to debug.
        catch (Exception e)
        {
            System.out.println(e);
            e.printStackTrace();
            System.exit(-1);
        }
        try
        {
            System.out.println("Dumping Threads on " + problemServer);
            //   Executing the dumpThreads function on the JVM MBean to create the javacore
            adminService.invoke(MBean, "dumpThreads", null, null);
        }
        catch (Exception e)
        {
            System.out.println("Exception invoking JVM MBean dumpThreads for " + problemServer + "\n" + e);
            e.printStackTrace();
        }
    }

    The sample code in Listing 2 initially parses the source of the notification to determine the server that detected a possible hung thread. The program retrieves the JVM MBean of the server that sent the notification and executes the dumpThreads() method to produce a Java thread dump. The thread dump information can be found in either a new javacore file, located in the Java working directory when using the IBM JDK (for Windows, AIX and Linux), or in the native_stdout.log file located in the server logs directory for the other JDKs.


Sample client program

In the included HungThreadDump.javadownload file is a complete administrative client program that listens for the Thead_Hung event notification and reacts by taking a thread dump on the appropriate server. Download this file to your client machine. After changing HostName and SoapPort to the appropriate values for your configuration, you can compile and run the client program as follows:

  • To compile, execute this command (set the variables JAVA_HOME and WAS_HOME to the appropriate locations):
    %JAVA_HOME%\bin\javac -classpath
    %WAS_HOME%\lib\admin.jar;
    %WAS_HOME%\lib\wsexception.jar;
    %WAS_HOME%\lib\jmxc.jar HungThreadDump.java
  • To run, execute this command (set the variables JAVA_HOME and WAS_HOME to the appropriate locations):
    %JAVA_HOME%\bin\java" -classpath
    %WAS_CLASSPATH%;
    %WAS_HOME%\classes;
    %WAS_HOME%\lib\admin.jar;
    %WAS_HOME%\lib\wasjmx.jar;." HungThreadDump

This client will stay in a wait state listening for the notification (or for the user input) to end the program. The client is designed to output all the information about the notification and then execute a thread dump. It is not designed to work when global security is enabled, but the client can easily modify the createAdminClient function to add the security parameters to the connectProps object. (For more information, see the Application Server API section in the WebSphere Application Server V5.1 Information Center.)


Sample custom service program

In the included CustomServiceThreadHungMBean.javadownload file is a complete custom service program that listens for the Thead_Hung notification and reacts by invoking a dumpThreads() method on the server's JVM MBean. This produces a thread dump on the appropriate server. Download this file to the WebSphere Application Server machine. No modifications are required. Compile and run this program using following commands:

  • To compile, execute the following command (set the variable JAVA_HOME and WAS_HOME to the appropriate locations):
    %JAVA_HOME%\bin\javac -classpath 
    %WAS_HOME%\lib\runtime.jar;
    %WAS_HOME%\lib\admin.jar;
    %WAS_HOME%\lib\wsexception.jar;
    %WAS_HOME%\lib\jmxc.jar CustomServiceThreadHungMBean.java

    Package the class file into a JAR file; for example:

    %JAVA_HOME%\bin\jar.exe cvf hung.jar
  • To install the custom service in a WebSphere Application Server cell:
    1. Select the server to which you want to add the custom service (dmgr, nodeagent or application server).
    2. In the Additional Properties section, select Custom Services.
    3. To create a new custom service, select New.
    4. Enter the appropriate information about the MBean; for example:
      • Select Startup so the custom MBean starts every time the server starts.
      • Supply the Classname of this custom service: com.ibm.support.CustomServiceThreadHungMBean.
      • Display Name is user defined and can be any value.
      • Classpath specifies the location of the JAR file; this value can be either an absolute path or symbolic.
Figure 1. Define custom service
Figure 1. Define custom service window

If the WebSphere Application Server install is a Base install, then the custom service must be configured on each server to monitor for hung threads. If the Base installs are federated to a Network Deployment cell, then only the dmgr process requires the custom service. The custom service running in the dmgr process will monitor all managed processes (dmgr, nodeagent, application server, and JMS server). The custom service is not designed to work with Java 2 security.


Conclusion

A Java client Thread_Hung event notification program can help administrators automatically monitor possibly hung threads in an entire cell, as well as execute thread dumps. This tool can provide the user with the necessary information for debugging specific code problems, and can also provide the initial information IBM Support needs to begin helping to fix the potential issue. Included with this article is a client program with a skeleton JMX administrative client, and a custom service for WebSphere Application Server. These program skeletons can easily be redesigned to listen for any event notification, based on filter design and query string, and to react as required, based on handleNotification method implementation.


Download

DescriptionNameSize
Java code sampleshung_thread_java.zip  ( HTTP | FTP )4 KB

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=32191
ArticleTitle=Developing a client to determine a hung thread problem
publish-date=12152004