Java diagnostics, IBM style, Part 1: Introducing the IBM Diagnostic and Monitoring Tools for Java - Dump Analyzer

Cut through endless dump files to get to the heart of the problem

Java applications have become increasingly complex; as a result, diagnosing problems in these applications is a non-trivial task and may require extensive work with an external service organization. A helpful pointer in the right direction could save both time and expense. The IBM Diagnostic and Monitoring Tools for Java - Dump Analyzer is a tool that performs basic analysis against a formatted system dump and produces a concise report indicating what it thinks your next course of action should be.

This article introduces the IBM Diagnostic and Monitoring Tools for Java - Dump Analyzer and provides some background information about the types of problems that the tool can diagnose. It explains the architecture on which the IBM Diagnostic and Monitoring Tools for Java - Dump Analyzer is built and provides some thoughts about the future direction of the tool.

18 July 2013 - Added new resource item for the article, "Java Technology Community," to Resources).


Helen Beeken, Software Engineer, IBM

Helen BeekenDr. Helen Beeken is a software engineer developing RAS tooling at the IBM Java Technology Centre. Before joining the team, she worked on the open source AspectJ and AJDT Eclipse projects and helped to stress-test large software systems.

Daniel Julin, Software Engineer, IBM

Author photoDaniel Julin has 20 years of experience developing and troubleshooting complex online systems. As technical area lead for the WebSphere Serviceability Team, he currently focuses on helping the team define and implement a collection of tools and techniques to assist in problem determination for WebSphere Application Server and to maximize the efficiency of IBM support. He also occasionally assists directly in various critical customer support situations.

developerWorks Contributing author

Julie Stalley, Software Engineer, IBM

Julie StalleyJulie Stalley is a software engineer currently working as part of the Java RAS Tooling team in Hursley. She joined IBM in 1996 and spent five years developing a large client/server application before moving to Hursley in 2001. She has subsequently worked on the Java class libraries, specializing in I/O, networking, and XML.

Martin Trotter, Software Engineer, IBM

Martin Trotter photoMartin Trotter has been working with the Java platform since its very early days and has worked extensively on the Java VM and the garbage collector in particular. He is currently working on improving the tooling available for diagnosing problems in the Java VM.

18 July 2013 (First published 02 October 2007)

Also available in Chinese Russian Japanese Vietnamese

Develop skills on this topic

This content is part of a progressive knowledge path for advancing your skills. See Monitor and diagnose Java applications

The Java language has come to be predominant in software development, and thus the reliability of the Java virtual machine (VM) has become a very important issue. The VM is typically a reliable piece of software, but of course failures do occur during execution for a variety of reasons. A small number of these problems are caused by errors in the VM itself; however, in the majority of cases, they are caused by errors or misconfigurations in the software stack above the VM (in IBM® WebSphere® Application Server, for instance) or in the application itself.

The software stack for a typical project has increased in complexity as information technology has matured, which has led to increasing difficulties for developers trying to determine the causes of problems. In such a complex environment, you may be faced with an overwhelming excess of information with which to diagnose a fault. In a production environment, there may well be many gigabytes of heap, hundreds of threads, thousands of classloaders, tens of thousands of classes, and a huge number of objects.

The IBM Diagnostic and Monitoring Tools for Java - Dump Analyzer (referred to hereafter as the Dump Analyzer) is an extensible framework that seeks a way out of this dilemma. The Dump Analyzer is available to all internal IBM users and external customers to investigate problems using the IBM Developer Kit for the Java Platform (IBM SDK). It uses analyzers to interrogate a formatted system dump (each analyzer asking the dump a specific question) and links the results together with a script to produce a concise report of the analysis. In its first two releases, the Dump Analyzer reports one of the four following outcomes:

About this series

Java diagnostics, IBM style explores new tooling from IBM that can help resolve problems with Java applications and improve their performance. You can expect to come away from every article with new knowledge that you can put immediately to use.

Each of the authors contributing to the series is part of a new team that creates tools to help you resolve problems with your Java applications. The authors have a variety of backgrounds and bring different skills and areas of specialization to the team.

Contact the authors individually with comments or questions about their articles.

  • Out of memory
  • Deadlock detected
  • VM terminated due to signal (a result of internal or middleware/Java application error)
  • Further investigation is required

Each of the first three can be mapped to a VM problem type described in the next section of this article.

No background knowledge is required to read the rest of this article. You'll get step-by-step instructions for analyzing a system dump using the Dump Analyzer, along with a high-level overview of the background to the tool and its architecture. After reading this article, you should have a good understanding of the circumstances under which you may want to use the Dump Analyzer, as well as some understanding of its infrastructure.

A high-level view of VM problem types

There are many ways in which a VM can fail while executing, and each type of failure requires a different approach to diagnosis. Before you look at the workings of the Dump Analyzer in detail, it is worth examining these different types of problems and the analysis required to solve them.

Out of memory problems

A VM can fail because it has run out of memory — either the Java heap memory or the native memory that is used by the VM to hold thread stacks, class information, JIT'ed code, graphic elements, and other artifacts for interfacing with the underlying operating system.

It can be extremely difficult to diagnose such a problem because the memory allocation that caused the failure is unlikely to itself be the ultimate culprit; some large collection had probably been growing inexorably until the VM finally reached the limit of available heap space. It is normally necessary to examine the contents of the heap and to compare snapshots of the heap taken at various times so as to identify collections that have grown rapidly.


A deadlock is a condition in which two or more processes are waiting for another process to release a resource. A thread owning a resource (such as a monitor) is unable to take ownership of another resource because a second thread owns it and is simultaneously attempting to gain ownership of the resource owned by the first thread. These faults often manifest as performance problems. They are relatively easy to diagnose by examining the states of the threads and the resources that they own.

Internal errors

Internal errors can be caused by a variety of problems:

  • Native code may attempt to access an object that has invalid input (such as a stale local reference) or that is coded incorrectly.
  • The garbage collector may incorrectly reclaim some storage that, when referenced, appears to contain pointers to uninitialized memory.
  • The JIT compiler may produce incorrect code that either references or attempts to branch to an invalid location.

Errors in the Java application or middleware

The Dump Analyzer currently deals with errors that occur or are detected at the level of the VM itself, but eventually the same set of tools will be able to diagnose various errors and incorrect behavior in the Java application or middleware that operates in the VM. There are various potential causes of these errors; they typically result from defects in the application or middleware code or misuse or misconfiguration of JVM options. They are generally diagnosed by examining the state of various data structures in the application or middleware to determine if some aspect of that state is incorrect.

Problem diagnosis today

Without a tool like the Dump Analyzer, you would usually begin the process of problem diagnosis by examining the artifacts produced by the VM at the point of failure. Typically, these are:

  • A dump of the process space (a system dump or core file)
  • A dump of the Java heap (a heapdump)
  • A snapshot of the Java process (a Javacore file)
  • A trace file showing some of the execution history

Generally, you would examine these artifacts individually, using specialized programs for each of the specific formats. The process of problem determination then essentially consists of a manual examination of the information available. As the volume of data increases, this process becomes more and more time consuming, and an increasingly specialized job. As a result, customers are often unwilling to undertake the analysis themselves and rely on their VM or middleware vendor to do it, but the majority of problems reported are eventually diagnosed as application, configuration, or environment problems that do not require any changes to the code in the VM or middleware itself. In an ideal world, the diagnostic capability available to the customer would ensure that only genuine defects that require such code changes are reported to the VM or middleware vendor. Other problems would be diagnosed by an automated process using the appropriate artifacts produced by the VM.

Overview of the Dump Analyzer

The Dump Analyzer is a tool based on the Diagnostic Tooling Framework for Java (DTFJ; you'll learn more about this later in this article) designed to analyze system dumps and look for various kinds of problems. The tool is made up of small analysis modules that look at specific dump data and determine if a particular problem (a deadlock, for instance) is present. This design can easily accommodate the addition of new capabilities and can be tailored to find specific problems.

The tool operates at two levels:

  • Each specialized analysis module attempts to diagnose one particular type of problem and produces a concise description of the problem that has been found.
  • When a diagnosis cannot be made, each analysis module can produce a more lengthy report about one particular aspect of the state of the system. This report can be used by a troubleshooting expert, possibly in conjunction with other information, to diagnose the problem.

To add extra flexibility, a simple scripting language is used to control the flow of analysis. Our team intends to develop this facility further over time by providing many different scripts.

Here's the tool's analysis flow:

  1. The tool loads the dump data selected by the user to create an image for further analysis.
  2. The user chooses one or more analysis modules to run against the image; if no specific analyzer is chosen by user, the default script is run.
  3. The analysis modules run.
  4. Each module either returns information that controls the flow of further analysis or generates information for a report.
  5. Once all of the modules have completed their runs, the report is formatted into an HTML or text document.

As noted, if the user does not request specific analysis modules, the tool runs the default script (general.sml), which runs a set of analyzers that check for several common types of problems. If none of these problems are detected, the script invokes a default report that outlines some general information about the state of the VM when the dump was produced.

Later in this article, you'll see an example of the Dump Analyzer in use and get an overview of some of the nondefault analysis modules that could be selected.

Setup required to use the Dump Analyzer

All the Dump Analyzer needs to run is a formatted system dump. A system dump is generated by default when the VM crashes; however, the VM can be configured to create such a dump under other failing circumstances or at the user's request (see the Diagnostic Guides links in Resources for more information).

To format a system dump, you need to run the jextract tool against it. Using the same VM on the same machine that produced the dump, simply run the following command line:

jextract "corefilename"

With a 1.4.2-level VM, this command produces a .sdff file; on VMs at Version 5.0 or above, it produces a file. Note also that on different platforms, there may be various options that control the format of the dump produced at the operating system level. In particular, these can cause the system dump to be truncated, which may prevent the Dump Analyzer from producing a useful diagnosis. The most common error (on UNIX® systems) is to forget to set the ulimit setting to unlimited, but there are other important options on other platforms. To avoid this type of problem, refer to the information in the IBM Diagnostics Guide or search the IBM Software Support Web site (see Resources for links to both) for platform-specific technotes with keywords such as "truncated core."

Using the Dump Analyzer within the IBM Support Assistant

About the IBM Support Assistant

The IBM Support Assistant is a free software serviceability workbench that helps you to resolve problems with IBM software products. ISA has a search facility that spans the bulk of IBM documentation and categorizes the results for review.

It also provides a product information feature that has links to product support and home pages, troubleshooting guides, and various forums and newsgroups. The Service feature of ISA can gather information from your desktop and can easily create a problem report for IBM.

ISA's Tool workbench provides problem determination tools that help resolve issues with IBM products. These tools are constantly updated and enable you to run troubleshooting and diagnostic tools on your desktop. See Resources for an ISA download link.

The main release vehicle for the Dump Analyzer is the IBM Support Assistant (referred to hereafter as ISA). ISA is available to all internal IBM users and external customers (see the Resources section for a download link).

You can install the Dump Analyzer with ISA as follows:

  1. Ensure that Version 3 of ISA is installed.
  2. To install the Dump Analyzer, you must install a product plug-in for which it is relevant — the IBM Developer Kit for Java, for example. (See the Resources section for instructions.)
  3. Restart the ISA client. Now you can install a tool plug-in.
  4. Go to the Updater service. There are two ways to do this:
    • Click the Updater icon on the Welcome page.
    • Click the Updater link on the menu bar.
  5. Select the New Plug-ins tab and wait for ISA to build the catalog of plug-ins that are available for installation.
  6. Open the Common Component Tools folder.
  7. Select IBM Diagnostic and Monitoring Tools for Java - Dump Analyzer and install.

Once you've installed the Dump Analyzer, you can launch it from within ISA:

  1. Restart ISA.
  2. Select Tools.
  3. Select a product for which the Dump Analyzer is available — the IBM Developer Kit for Java, for example.
  4. Click IBM Diagnostic and Monitoring Tools for Java - Dump Analyzer to launch the tool. Your screen should look like Figure 1:
Figure 1. The Dump Analyzer within ISA
The Dump Analyzer within ISA

Here's how you would analyze a formatted system dump:

  1. Enter the fully qualified name of a formatted system dump to be analyzed.
  2. Click Estimate Time to receive a rough estimate of how long the analysis will take.
  3. Click Analyze. The results will appear in the window when complete.

Figure 2 shows an example of the kind of observation summary that the Dump Analyzer produces:

Figure 2. Example observation summary
Example observation summary

Clicking Analyze Another returns to the screen shown in Figure 1, with the previously entered dump file name still entered in the first text box.

Selecting an analyzer module

The field labeled Optional Parameters in the invocation screens in Figures 1 and 2 controls the set of analyzers that are to be executed, as well as other runtime options. Normally, you should leave that field blank; this will cause the default analysis script, general.sml, to be executed. This script checks for the most common types of problems. However, if you already know the particular type of problem that you are investigating or if you need to work on a problem that is not integrated into the default script, you can explicitly specify one or more analyzers to be invoked. These analyzers can be invoked with the name of specific script files or the class name of specific analyzer modules. Typing -help in the Optional Parameters field will list further runtime options.

In the first release of the tool, only a very small number of experimental analyzers have been provided beyond the default script. These include:

  • DefaultDumpReport (class name: This analyzer produces a fairly detailed report on all the main aspects of the state of the VM, somewhat similar to what can be found in a Javacore file (but with some additional DTFJ-specific information).

  • ListZipJars (class name: This experimental analyzer attempts to discover all the zip and JAR files that are currently open within the VM, which may provide insight into any custom libraries being used by the application or the middleware.

  • SystemProperties (class name: This experimental analyzer scans the VM and prints the current value of every Java system property defined in that VM.

  • WASBasicInfo (class name: This is a very preliminary and experimental version of an analyzer that demonstrates the use of this tool to examine the state of a WebSphere Application Server runtime executing inside the VM.

These other analyzers are currently provided mostly to illustrate the flexibility of the tool. Future releases will many additional specialized analyzers along with documentation. In addition, in the fourth article in this series, you will learn how to write your own analyzers to supplement those shipped with the tool itself.

Using the Dump Analyzer from the command line

In some circumstances, you may find it useful to be able to run the Dump Analyzer from the command line (if the analysis needs to be incorporated into some existing problem-handling workflow, for instance). The easiest way to use the Dump Analyzer is through ISA, and you've already seen how to download the Dump Analyzer within ISA.

To run the Dump Analyzer alone, you need four JAR files and a script file. These are:

  • dumpAnalyzer.jar (found in installDir/plugins/ number)/WEB-INF/lib)
  • dtfj-interface.jar (found in installDir/plugins/ number)/WEB-INF/lib/j9)
  • dtfj.jar for Java 5.0 and above (found in installDir/plugins/ number)/WEB-INF/lib/j9)
  • dtfj.jar for Java 1.4.2 (found in installDir/plugins/ number)/WEB-INF/lib/sov)
  • general.sml (found in installDir/plugins/ number)

In all of these file paths, installDir denotes the ISA installation directory; by default, this is C:\Program Files\IBM\IBM Support Assistant v3 on Microsoft Windows or /opt/IBM/IBM Support Assistant v3 on Linux™. You can copy these files elsewhere, or you can run the Dump Analyzer directly from the installDir/plugins/ number) directory. Although ISA is only available on Windows and Linux, you can run the Dump Analyzer from the command line on any platform.

Here are the command-line steps to run Dump Analyzer from the default directory on Windows:

  1. set CP=WEB-INF/lib/dumpAnalyzer.jar
  2. set BCP=WEB-INF/lib/j9/dtfj.jar;WEB-INF/lib/j9/dtfj-interface.jar;WEB-INF/lib/sov/dtfj.jar
  3. java -cp %CP% -Xbootclasspath/p:%BCP% (dumpName) (options)

And here are the steps for Linux:

  1. export CP=WEB-INF/lib/dumpAnalyzer.jar
  2. export BCP=WEB-INF/lib/j9/dtfj.jar:WEB-INF/lib/j9/dtfj-interface.jar:WEB-INF/lib/sov/dtfj.jar
  3. java -cp $CP -Xbootclasspath/p:$BCP (dumpName) (options)

Here, dumpName is the fully qualified dump name to be analyzed and options are the runtime parameters that can be used to configure the Dump Analyzer. Running with the -help option prints a list of all the available parameters.

Figure 3 shows a snapshot of some output from the Dump Analyzer running on the command line:

Figure 3. Example Dump Analyzer command-line output
Example Dump Analyzer command-line output

For more information about DTFJ, see the Resources section.

Future plans

At the time of writing, the initial release of the Dump Analyzer is currently available. Our team intends to continue to make enhancements and updates on a regular basis. In particular, we will focus on two areas:

DTFJ: The architecture behind the Dump Analyzer

The Dump Analyzer uses the DTFJ to interrogate formatted system dumps. DTFJ is an API that supports the building of Java diagnostic tools that can examine a system dump taken from a VM. Before the system dump can be examined, it must first be processed by jextract to add Java runtime-specific information. It is essential that jextract be run against the system dump (jextract core.dmp, on the command line) using the same version of the Java platform and run on the same machine that produced the system dump. Once this is done, the resulting file (.sdff for V1.4 of the Java platform, or for V5.0 and above) can be examined on any system.

The first stage in analysis using DTFJ is to build a DTFJ image using an appropriate image factory. This factory is specific to a given VM level and has internal understanding of the platform-specific dump format. This image represents the entire operating system image; it provides methods for accessing information about the architecture of the machine that produced the system dump, but its major feature is its ability to enable examination of the address spaces contained within the image. Using the DTFJ API, it is possible to navigate from the image to details of runtime parameters, Java threads, the native environment, and objects on the heap. The interface allows tooling to analyze problems and produce report information.

  • We will continue to enhance the user interface of the tool itself, adding panels to control the dumps and analyzers that run, improving the output format, possibly adding an interactive mode, and more.
  • We will increase the number of analyzers and scripts to cover a wider variety of problems.

The area of building new analyzers is particularly exciting. With this DTFJ dump analysis technology, you now have a fairly accessible mechanism for examining low-level VM entities such as threads and monitors so as to diagnose out-of-memory errors, crashes, deadlocks, and the like; in addition, you can also examine the content of any data structure present in the VM. In particular, you can examine the contents of various data structures that are part of the implementation of the application or middleware running inside the VM. We intend to start building a collection of analyzers that exploit this information to help diagnose a variety of problems in WebSphere Application Server and possibly other stack products.

Our aim is to make the tool as useful as possible, so feedback — both about the tool itself and about new analyzers you'd like to see added — is extremely welcome. Feedback can be provided through this article or through ISA.

Coming up in this series

The next article in this series introduces the IBM Diagnostic and Monitoring Tools for Java - Garbage Collection and Memory Visualizer. This tool can help you investigate memory-based Java performance problems by analyzing verbose garbage collection logs. You can use the tool to look at memory usage patterns, determine if there is a memory leak, or tune the garbage collection configuration to improve performance.

You'll revisit the Dump Analyzer in the fourth article in this series. In that article, you'll get a much more in-depth look to the tool's extensibility and learn how to build your own analysis modules for it.



Get products and technologies


  • IBM Java Runtimes and SDKs: Visit this discussion forum for questions related to the IBM Developer Kits for the Java Platform.
  • Java Technology Community: Interact with industry experts as J2EE architects, developers and programmers share their knowledge and experiences on the technology.


developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into Java technology on developerWorks

Zone=Java technology
ArticleTitle=Java diagnostics, IBM style, Part 1: Introducing the IBM Diagnostic and Monitoring Tools for Java - Dump Analyzer