In each column, The Support Authority discusses resources, tools, and other elements of IBM® Technical Support that are available for WebSphere® products, plus techniques and new ideas that can further enhance your IBM support experience.

This just in...

As always, we begin with some new items of interest for the WebSphere community at large:

• Are you ready for Impact 2011? Join us for Impact 2011, April 10-15, 2011 in Las Vegas Nevada, at The Venetian and The Palazzo Hotels. Check out these top 5 reasons to attend Impact (PDF, 115KB), the one conference where business and IT leaders can explore together how to achieve greater business agility.
• Keep checking the IBM Conferences and Events page for a list of upcoming conferences.
• Learn, share, and network at the IBM Electronic Support Community blog on developerWorks.
• Find out about the new Global WebSphere Community (GWC) at websphereusergroup.org. Customize the content on your personalized GWC page and connect to other "WebSpherians" with the same interests. The GWC is also offering a no charge education course on IBM WebSphere CloudBurst™ Appliance. Learn how to configure resources managed in the cloud, create virtual systems for deployment to the cloud, administer resources deployed to the cloud, and create a multiple image pattern and manage dynamic machines.
• Several exciting webcasts are planned for the upcoming months at the WebSphere Technical Exchange. Check the site for details and become a fan on Facebook!
• IBM Support Assistant 4.1.2 is now available. IBM Support Assistant 4.1.2 delivers several defect fixes and a new version of its quick data collection tool, ISA Lite. New features now available in ISA Lite include:
  • Inventory collected in ISA Lite can be easily viewed in a browser.
  • Leverages Ant 1.8.
  • Numbered menu options.
  • You can pause the processing of a response file in different scenarios.
  • A more extensible format that uses name-value pairs for the response file enables the addition of comments inside the response file, name-value pairs in the response file that cover all the relevant questions, and easy response file editing for system customization.
  • Version information is automatically written to the console and log at startup.
  • A -help option for viewing information on how to use the tool.
  • Alternate file transfer options.
  • A visual indicator that shows if the collection ended successfully.
  • A -useHome option to prevent files being written to the ISA Lite installation directory.
  • Support for Windows® 7 and Linux® RedHat 64-bit.
  • Inventory collection support for Solaris.

Continue to monitor the various support-related Web sites, as well as this column, for news about other tools as we encounter them.

And now, on to our main topic...

Back to top

Debugging from dumps

In a companion IBM developerWorks article on Debugging from dumps, we introduced the concept of using the IBM Monitoring and Diagnostics for Java - Memory Analyzer tooling (hereafter referred to as Memory Analyzer) in conjunction with an HPROF binary dump from a HotSpot-based Java runtime (Solaris™ and HP-UX) or a system dump from an IBM Java runtime to debug problems other than memory leaks. That article covered the importance of using those dump types (as opposed to the IBM Portable Heap Dump), how to obtain the dumps, and how to use them to diagnose a number of problem types, including footprint and memory efficiency issues, class loader hierarchy issues, thread data usage, and exception analysis.

This article covers how to enable and generate the HPROF and IBM system dumps in IBM WebSphere Application Server, and introduces how to use Memory Analyzer with the newly available IBM Extensions for Memory Analyzer plug-in to help you understand the state of your WebSphere Application Server environment and diagnose some WebSphere Application Server specific issues.

Back to top

Generating dumps in WebSphere Application Server

In order to utilise the full power of Memory Analyzer, it is important to generate either HPROF format dumps (if using a HotSpot-based Java runtime) or an IBM system dump (if using an IBM-based Java runtime). This is because those dump formats contain additional data stored inside the Java objects on the heap that is vital for diagnosing problems beyond memory leaks and creating reports that show the status of the application server.

Whilst the HPROF and IBM system dump formats contain important additional data, this comes at the cost of additional disk space requirements. Both types of system dumps are larger than the size of the Java heap that they represent, with the IBM system dump being equivalent to the overall memory size of the Java process. This means it is important to ensure that there is sufficient disk space to prevent the dumps from being truncated. By default, the dumps are written to the current working directory of the Java process. In WebSphere Application Server, this is the path of the server's profile directory; for example: WebSphere_Application_Server_Install_Path/profiles/node1. For IBM Java runtimes, you can change this location with the -Xdump file command line option. For HotSpot Java runtimes, you can change the location using the -XX:HeapDumpPath command line option.

When system dumps are being generated, it is also important to ensure that the necessary operating system level settings are in place to enable full dumps to be written. On UNIX®-based systems, this means ensuring that the correct ulimits are set. (See Resources for information on obtaining system dumps from AIX®, Linux®, z/OS®, and Solaris.)

Back to top

Generating dumps from HotSpot Java runtimes

HotSpot-based Java runtimes generate the HPROF format dump only, and can do so when an OutOfMemoryError occurs, or interactively using a variety of different methods:

• Interactive methods
  • Using a Ctrl+Break: If the -XX:+HeapDumpOnCtrlBreak command line option is set for the running application, an HPROF format dump is generated along with a thread dump when a Ctrl+Break event, or SIGQUIT (usually generated using kill -3), is sent via the console. This option might not be available on some versions; if this is the case for you, try:

    -Xrunhprof:format=b,file=heapdump.hprof

  • Using the jmap tool: The jmap utility tool (see Resources), delivered in the bin directory of the JDK, provides an option to request an HPROF dump from the running process. With Java 5, use:

    jmap -dump:format=b pid

    With Java 6, use the command below, where "live" is optional and results in only the "live" objects being written to the dump-file process ID (PID):

    jmap -dump[live,]format=b,file=filename pid

  • Using the operating system: Use the nondestructive gcore command or the destructive kill -6 or kill -11 commands to produce a core file. Then, extract a heap dump from the core file using jmap by pointing to the core dump:

    jmap -dump:format=b,file=heap.hprof path

  • Using the JConsole tool: A dumpHeap operation is provided under the HotSpotDiagnostic MBean in JConsole. This operation requests that a HPROF dump be generated.
• Event-based method

  If the -XX:+HeapDumpOnOutOfMemoryError command line option is set for the running application, an HPROF format dump is generated when an OutOfMemoryError occurs. Having this in place is ideal for production systems because it is nearly always required to diagnose memory issues, and it incurs no ongoing performance overhead. There is no limit to how many heap dumps are produced on this event per JVM run; in newer releases, a maximum of one heap dump is produced on this event per JVM run.

It is a good idea to set the -XX:+HeapDumpOnOutOfMemoryError command line option for any WebSphere Application Server deployment you have in production.

Back to top

Generating dumps from WebSphere Application Server

IBM Java runtimes provide dump and trace engines that can generate either PHD-format or system dumps in a large number of interactive and event-based scenarios. You can also generate interactive dumps using the Health Center tool or programmatically using a Java API:

• Interactive methods
  • Using a SIGQUIT or Ctrl+Break: When a Ctrl+Break or SIGQUIT (usually generated using kill -3) is sent to the IBM runtime, a user event is generated in the IBM dump engine. By default, this event only generates a thread dump file (javacore.txt). You can use the -Xdump:heap:events=user option to generate a PHD-format dump, or the -Xdump:system:events=user option to generate a system dump of the Java application.
  • Using the operating system to produce a system dump:
    • AIX: gencore (or the destructive kill -6 or kill -11)
    • Linux/Solaris: gcore (or the destructive kill -6 or kill -11)
    • Windows: userdump.exe
    • z/OS: SVCDUMP or console dump
  • Using IBM Monitoring and Diagnostics Tools for Java - Health Center: The Health Center tool provides a menu option to request either a PHD or system dump from a running Java process (see Resources).
  • Using the wsadmin scripting client: The following commands use Jython with the -lang jython option (see Resources for details on connecting wsadmin and referencing the target process):
    • Thread dump:

      AdminControl.invoke(AdminControl.completeObjectName("type=JVM,process=server1,*"), "dumpThreads")

    • System dump:

      AdminControl.invoke(AdminControl.completeObjectName("type=JVM,process=server1,*"), "generateSystemDump")

    • Portable heap dump (PHD):

      AdminControl.invoke(AdminControl.completeObjectName("type=JVM,process=server1,*"), "generateHeapDump")

      After 10 heapdumps in a single JVM run, the command will throw HeapDumpOnDiskLimitReachedException (this only applies to this JMX command). Increase this limit with the following command (this setting does not survive a restart):

      AdminControl.invoke(AdminControl.completeObjectName("type=JVM,process=server1,*"), "setMaxDumpsOnDisk", "20")

• Event-based method

  The IBM dump and trace engines provide a flexible set of capabilities for generating PHD and system dumps on a large number of events, from exceptions being thrown to methods being executed. Using these, you should be able to generate dumps for most problem scenarios you want to diagnose:

  • Using the IBM dump engine: The dump engine provides many events on which you can produce a PHD or system dump, and also the ability to filter on the types of those events to provide fine-grained control over when to generate dumps. This is the most common setting on an OutOfMemoryError (set in the generic JVM arguments of the application server):

    -Xdump:heap:none -Xdump:java+system:events=systhrow,filter=java/lang/OutOfMemoryError,range=1..4,request=exclusive+compact+prepwalk

    This command produces an IBM system dump (and a thread dump) rather than a portable heap dump when an OutOfMemoryError occurs. The following -Xdump command line option should be set for any production WebSphere Application Server deployment:

    -Xdump:heap:none -Xdump:java+system:events=systhrow,filter=java/lang/OutOfMemoryError,range=1..4,request=exclusive+compact+prepwalk

Back to top

Postprocessing requirements for IBM system dumps

An IBM system dump must be postprocessed using the jextract tool shipped with the JDK:

WebSphere_Application_Server_Install_Path/java/jre/bin/jextract core

Ideally, jextract is run on the same physical machine that produced the dump, using jextract from the same JDK installation that produced the dump, and with read access to the same libraries with which the Java process was running. Given that jextract can consume significant CPU cycles processing the dump, this might be unacceptable in some production systems. In that case, the dump should be processed on the closest matching system, such as a preproduction test system. The service refresh (SR) and fix pack (FP) versions of the Java runtimes should match.

The jextract tool produces a ZIP file that includes the original core dump, a processed representation of the dump, the Java executable, and the libraries used by the Java process. You can delete the original core dump because it is within the ZIP. The ZIP file is what should be loaded into the Memory Analyzer tool.

Back to top

Setting Java command line options in the WebSphere Application Server admin console

Java command line options can be set in the WebSphere Application Server administrative console using the generic JVM arguments option in the Java Virtual Machine panel of the process definition. To find the Java Virtual Machine panel:

1. Navigate to the admin console and select Servers > Server Types > WebSphere application servers from the left panel.
2. Select your application server from the main panel (Figure 1).
   
   Figure 1. Select application server
   
   
   
3. Expand the Java and Process Management option to the right of the main panel and select Process definition (Figure 2).
   
   Figure 2. Select Process definition
   
   
   
4. Select the Java Virtual Machine option on the right (Figure 3).
   
   Figure 3. Select Java Virtual Machine
   
   
   
5. The Generic JVM arguments text box displays toward the bottom of the main panel.

When the options are added, you will need to save and synchronize the changes before restarting the application server to put the changes in effect.

Back to top

Installing Memory Analyzer and IBM Extensions

Memory Analyzer is installed into the IBM Support Assistant. The IBM Extensions for Memory Analyzer (hereafter referred to as IBM Extensions) are installed using the updater capability in either the IBM Support Assistant or in the Eclipse Memory Analyzer Tool (MAT). Refer to the documentation for IBM Monitoring and Diagnostic Tools for Java - Memory Analyzer, IBM Support Assistant, and the open source Eclipse Memory Analyzer Tool for specific instructions on installing these tools. The general installation steps are summarized here:

1. Install and then open IBM Support Assistant.
2. Select Update > Find New... > Tools Add-ons.
3. Expand the JVM-based Tools section.
4. Select the plug-in called IBM Monitoring and Diagnostic Tools for Java – Memory Analyzer [Tech Preview].
5. After Memory Analyzer is installed, restart IBM Support Assistant.
6. Launch Memory Analyzer by clicking Analyze Problem and then IBM Monitoring and Diagnostic Tools for Java - Memory Analyzer [Tech Preview]. Click the Launch button.

The IBM Extensions are a collection of plug-ins for both the Memory Analyzer tool in the IBM Support Assistant and the Eclipse MAT. The extensions offer additional capabilities for debugging both generic Java applications and specific IBM software products by building knowledge of those products into the extensions. The IBM Extensions currently offers extensions for:

• Java SE runtime
• WebSphere Application Server
• IBM WebSphere eXtreme Scale
• IBM CICS® Transaction Gateway

The steps below summarize how to install the IBM Extensions into IBM Support Assistant; the process for MAT is very similar:

1. From the IBM Support Assistant menu, select File > Preferences.
2. Select Updater Preferences.
3. Check Automatically include this location when searching for new or updated plug-ins.
4. Select Use this specified update site in addition to the IBM Support Assistant update site.
5. Select http from the pull down box, and enter a Location of http://dl.alphaworks.ibm.com/ettktechnologies/updates.
6. You have now enabled IBM Support Assistant to search the IBM Extensions for Memory Analyzer update site when it searches for new tools and updates. To ask IBM Support Assistant to search for new tools, select Update > Find New... > Tools Add-ons from the menu inside IBM Support Assistant.
7. Expand the JVM-based Tools section.
8. Select the plug-ins with names beginning with "IBM Extensions for Memory Analyzer - ". The IBM Extensions for Memory Analyzer - Utilities (required) plug-in must be installed. The others are optional, depending on which products you want extensions for.
9. Accept the licence terms and click Finish.
10. After you restart the IBM Support Assistant workbench, the new extensions should be installed and ready for use inside Memory Analyzer.

Back to top

Using the IBM Extensions inside Memory Analyzer

Always on extensions

Some IBM extensions are "always on," meaning that various views show additional information about an object next to its address. In the case shown in Figure 4, for example, the selected object now describes how many sessions are in the MemoryStore, and what Web application those sessions are for. The number of sessions and the application name are added by the IBM Extensions to immediately describe the origin of the memory use.

In the Dominator tree view (which is a list of objects that together retain the entire Java heap), the classloader grouping replaces the name of the class loader with a description of what it loads. For example, the selected class loader in Figure 5 now shows that it is the class loader for the PlantsByWebSphere Web application.

Pull down menu extensions

A number of the IBM extensions are executable queries that can be found under the Query Browser pull down menu. The IBM Extensions can be found under the IBM Extensions category and are divided into three sub-categories: Java SE Runtime, CICS Transaction Gateway, and WebSphere Application Server (Figure 6).

These queries are currently available:

• Java SE runtime
  • Java Overview provides a summary of the Java runtime and process, including vendor, version, operating system, Java home directory, class path, and command line.
  • DirectByteBuffers produces two reports: a summary covering the number of DirectByteBuffer instances, their associated memory, and what they are used for, and a table of each individual DirectByteBuffer instance.
  • List All Objects produces a list of every Java object on the heap, initially ordered by the size of the individual objects.
  • List All GC Roots produces a list of all of the garbage collection roots; the objects that are referenced from outside of the Java heap, from areas such as threads or JNI code, that act as the starting points used by garbage collection to determine which Java objects are still in use by the Java application.
  • Export Object enables an object graph to be written as a textual representation to a text file.
  • Calculate Naive Retained Heap Size calculates all of the objects retained (referenced) by a given object, ignoring the fact that some of those objects might also be referenced by other parts of the application.
• CICS Transaction Gateway
  • Overview provides a basic overview of CICS Transaction Gateway, including version and build information, the configuration that it is running with, and a list of all the environment variables set.
  • Active Conversations provides all the active IPIC conversations in the CICS Transaction Gateway, the session they currently have and their states.
  • Requests provides the ECI request objects in the CICS Transaction Gateway that have been processed or are being processed.
  • Statistics provides a selection of the statistics recorded by the CICS Transaction Gateway.
• WebSphere Application Server
  • WAS Overview provides a summary of the WebSphere Application Server runtime, including product and version, server name, node, cell, and the status of the deployed applications.
  • WAS Cache Analysis provides a summary of the application server provided caches, showing their size, occupancy, and retained memory.
  • Web Container Analysis provides a breakdown of the deployed Web applications, their locations, and the number of HTTP sessions associated with them.
  • Thread Pool Analysis provides a table for the application server provided threads pools covering: thread pool configuration, current size and presence of hung threads, plus a table for the Java level thread groups, covering thread count and amount of associated memory.
  • HTTP Sessions produces a list of every HTTP session in memory, summarising the application name, session ID, user name, timeout, plus creation and last access times. In addition, it is possible to expand any HTTP session to look at the keys and values associated with the session itself.
  • EJB Container provides a simple breakdown of the number of each type of EJB deployed in the application server.
  • Application Class Loader Leaks lists all of the application class loaders, indicating whether any of them are stopped and therefore leaking. This can then be used to determine which application is problematic.
  • Alarm Manager provides a list of the deferrable, non-deferrable, deferred, and unqueued alarms.
• WebSphere eXtreme Scale
  • Priority Queue Runnables finds the issue described in APAR PM30264.

Back to top

Extensions in action

WAS Overview query

The WAS Overview query is a simple query that shows the version and fix pack level of the WebSphere Application Server currently running, along with the server, node, and cell name, the inferred start up time of the process, and a list of all applications and their running state (Figure 7). This query can be accessed by navigating to Open Query Browser > IBM Extensions > WebSphere Application Server > WAS Overview.

HTTP Sessions query

One common class of problems is related to applications' usage of HTTP sessions. In addition to OutOfMemoryErrors, other issues include finding the contents of sessions, timeout values, session identifiers, and the applications to which the sessions belong. There are many implementation details that make finding sessions and their values non-trivial. These details include different class names across different versions of the product and multiple maps per session (one for persisted sessions and the other for non-persisted sessions, both of which might be active in a single session).

The HTTP Sessions query will extract all of this information into a tree view (Figure 8). This can be accessed through Open Query Browser > IBM Extensions > WebSphere Application Server > HTTP Sessions. The columns below the dotted line in the figure are a continuation of the columns above.

Each session object can be expanded to reveal its key/value pairs. In Figure 9, there are five attributes in the session. Each key is of the form sessN and the value is a custom object. If the value is something other than a String, then the row can be expanded to reveal the object.

This plug-in also works with PHD heapdumps, but most of the metadata will be missing and only the retained heap sizes of the attributes will be known.

Application Class Loader Leaks query

An application class loader leak occurs when applications are restarted but their class loaders are not cleaned up. This is often caused by a reference to the application from somewhere else in the server, such as from ThreadLocals. It might also be caused by defects within WebSphere Application Server itself. If an application class loader leak exists and an application is restarted enough times, this can cause Java or native OutOfMemoryErrors.

The Application ClassLoader Leaks query will find all stopped application class loaders and interrogate suspects that cause the class loader to be ineligible for garbage collection. This plug-in works equally well on a PHD heap dump (Figure 10). This can be accessed through Open Query Browser > IBM Extensions > WebSphere Application Server > Application ClassLoader Leaks.

Back to top

Conclusion

Memory Analyzer and the IBM Extensions for Memory Analyzer give you more insight into what is happening inside WebSphere Application Server and related products. We encourage you to try the techniques and plug-ins outlined in this article, particularly with system dumps instead of PHD dumps for the IBM runtime. Let us know what else you'd like to see exposed through system dumps by participating in the alphaWorks forum.

Resources

Learn

• IBM Java Diagnostics Guide 6: Using the -Xdump option
  
  
• Java HotSpot VM Options
  
  
• Obtaining system dumps:
  • Enabling full AIX core files
  • Setting up and checking your Linux environment
  • Obtaining SVC dumps (z/OS)
  • Setting up dumps (z/OS)
  • Troubleshooting Guide for Java SE 6 with HotSpot VM (PDF)
  • Starting the wsadmin scripting client using wsadmin scripting
  • jmap Memory Map documentation
  • Using dump agents
  • Using the dump viewer
  • How to run on 64bit VM while the native SWT are 32bit
  
  
• The Support Authority: If you need help with WebSphere products, there are many ways to get it
  
  
• IBM Software product Information Centers
  
  
• IBM Software Support Web site
  
  
• IBM Education Assistant
  
  
• IBM developerWorks
  
  
• IBM Redbooks
  
  
• WebSphere Software Accelerated Value Program
  
  

Get products and technologies

• IBM Extensions for Memory Analyzer
  
  
• Tech Preview: IBM Monitoring and Diagnostic Tools for Java - Memory Analyzer Version 0.6
  
  
• IBM Monitoring and Diagnostic Tools for Java - Health Center Version 1.3
  
  
• Eclipse Memory Analyzer Tool
  
  
• IBM Diagnostic Tool Framework for Java Version 1.5
  
  
• IBM Software Support Toolbar
  
  
• IBM Support Assistant
  
  

Discuss

• Memory Analyzer News
  
  
• Troubleshooting Java Applications
  
  
• Forums and newsgroups
  
  
• Java technology Forums
  
  
• WebSphere Support Technical Exchange on Facebook
  
  
• Global WebSphere Community on WebSphere.org
  
  
• Follow IBM Support on Twitter!
  • WebSphere Electronic Support
  • WebSphere Application Server information
  • WebSphere Process Server
  • WebSphere MQ
  • WebSphere Business Process Management
  • WebSphere Business Modeler
  • WebSphere Adapters
  • WebSphere DataPower Appliances
  • WebSphere Commerce
  • IBM Support Assistant Tools
  
  

About the authors

Chris Bailey is a member of the IBM Java Technology Center team at the Hursley Park Development Lab in the United Kingdom. As the technical architect for the IBM Java service and support organization, he is responsible for enabling users of the IBM SDK for Java to deliver successful application deployments. Chris is also involved in gathering and assessing new requirements, delivering new debugging capabilities and tools, making improvements in documentation, and improving the quality of the IBM SDK for Java.

Kevin Grigorenko is a software engineer on the WebSphere Application Server SWAT team, which provides worldwide, on-site, and remote supplemental product-defect support, particularly in critical customer-support situations. He focuses on problem determination for WebSphere Application Server and related stack products and operating systems, including the IBM and Oracle JVMs, AIX, Linux, Solaris, Windows, z/OS, HP/UX, and i5/OS.

Dr. Mahesh Rathi has been involved with WebSphere Application Server product since its inception. He led the security development team before joining the L2 Support team, and joined the SWAT team in 2005. He thoroughly enjoys working with demanding customers, on hot issues, and thrives in pressure situations. He received his PhD in Computer Sciences from Purdue University and taught Software Engineering at Wichita State University before joining IBM.

Rate this article

Comments

Back to top

Table of contents

• This just in...
• Debugging from dumps
• Generating dumps in WebSphere Application Server
• Generating dumps from HotSpot Java runtimes
• Generating dumps from WebSphere Application Server
• Postprocessing requirements for IBM system dumps
• Setting Java command line options in the WebSphere Application Server admin console
• Installing Memory Analyzer and IBM Extensions
• Using the IBM Extensions inside Memory Analyzer
• Extensions in action
• Conclusion
• Resources
• About the authors
• Comments

Next steps from IBM

• Try: WebSphere Application Server
• Briefing: Building SOA solutions and managing the service lifecycle
• Buy: WebSphere Application Server

Dig deeper into WebSphere on developerWorks

IBM SmartCloud trial. No charge.

Unleash the power of hybrid cloud computing today!

---

Public and private solutions in one trial.

Special offers