The Memory Analyzer Tool (MAT) is the recommended tool for analyzing heapdumps and system dumps, whether to find the cause of an OutOfMemoryError (OOM) or memory leak, to size a JVM, or other types of debugging . The tool is available for free with the IBM Support Assistant Workbench (ISA) or it can be downloaded from eclipse.org and extended with updates to support IBM dump types.
One common memory issue is a Java memory leak. This is not like a traditional memory leak in C/C++ where memory is mis-managed, but rather that the application... [More]
Information Centers are one place where IBM publishes official product documentation. A list of all information centers can be found here: http://www.ibm.com/support/publications/us/library/ . The WAS InfoCenters are: WAS 8: http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/index.jsp WAS 7: http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp WAS 6.1: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp WAS 6.0: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp Each WAS InfoCenter is split up by product... [More]
If you often have to create new application servers, you should consider application server templates: http://www14.software.ibm.com/webapp/wsbroker/redirect?version=compass&product=was-nd-mp&topic=trun_create_templates The process is simple: create one application server, add your customizations (JVM arguments, data sources, etc.), then click Templates... > New, and select that server. Starting in WAS 7, you can even edit the template after it has been created. Then, when you create a new application server, either through the... [More]
I was recently at a customer who believed that they had a Java memory leak. They compared heapdumps and couldn't find anything. They had experienced production OutOfMemoryErrors (OOMs) before (for a different reason), and they were so worried about what they perceived, that they increased the maximum heap size to 4GB so that the JVM could handle a day's worth of work, and then they put in a process to restart the JVMs every night.
At a first glance of verbose garbage collection, I agreed with them (loaded in the wonderful Garbage... [More]
Garbage Collection and Memory Visualizer (GCMV) is a great tool to visualize verbosegc and it is available for free in the IBM Support Assistant . One of its lesser well known features is the ability to compare verbosegc from two different JVM runs. This is particularly useful if you changed something and you want to see the effect. First, load the baseline verbosegc as normal. Next, right click anywhere in the plot area and click 'Compare File...': Next, ensure that the X-axis uses a relative format such as hours, instead of date. Otherwise,... [More]
IBM Research recently made its WAIT tool (Whole-system Analysis of Idle Time) available to the public: https://wait.researchlabs.ibm.com/
The WAIT tool takes javacores and operating system statistics snapshots for a period of time as input, and produces a rich webpage as output that visualizes this data. Since WAIT only uses javacores and operating system scripts, you don't need to install anything or even restart the Java processes to use WAIT. For administrators familiar with the performance, hang, and high CPU MustGathers , the... [More]
The IBM Extensions for Memory Analyzer (IEMA) are a set of free plugins that extend the powerful Memory Analyzer Tool with product-specific knowledge when analyzing heapdumps and system dumps. For example, IEMA can list out all HTTP sessions, attributes such as their JSESSIONIDs and user names, and session attributes in a nice tree view for WAS dumps (note that much of the additional information is only available with system dumps on IBM JVMs). For more information on WAS-related extensions in IEMA, see... [More]
The IBM Java HealthCenter is a low overhead agent shipped with the JVM that can provide deep insight into JVM activity for IBM Java 5 and above: http://www.ibm.com/developerworks/java/jdk/tools/healthcenter/ . HealthCenter is similar to VisualVM . HealthCenter can provide information on classloading, JVM arguments, garbage collection, file I/O, locking, native memory, threads, and profiling. The last item, profiling, is my favorite feature of HealthCenter -- it is a very low overhead, sampling profiler that can pinpoint high CPU issues or... [More]
Update : ActiveCount has been made a part of the PMI Basic collection (enabled by default) in WAS 18.104.22.168 and 22.214.171.124 .
WebSphere Application Server comes with the "Basic" level enabled in the Performance Monitoring Infrastructure (PMI) on every server by default . This usually has an overhead of about 2-3% . The Basic level comes with one statistic for all thread pools: PoolSize (see the "Level" column in the link). This is defined as "the average number of threads in pool." This is a bit confusing: it is... [More]
WebSphere Application Server comes with a built-in HTTP(S) server. This post covers different methods of printing the response times of HTTP(S) requests. If all you need are averages, then the built-in Performance Monitoring Infrastructure (PMI) provides average statistics for HTTP(S) response times. However, if you need information on particular requests, then averages may not help . The most robust solution is to use a monitoring product such as ITCAM . This article will cover the basic capabilities that are built-in to WAS. Method #0: Web... [More]
I'm not sure how this applies to modern computer chips and operating systems, but here is interesting research from Liedtke in 1995 and 1997 showing the overhead of system calls:
For measuring the system-call overhead, getpid, the shortest Linux system call, was examined. To measure its cost under ideal circumstances, it was repeatedly invoked in a tight loop. Table 2 shows the consumed cycles and the time per invocation derived from the cycle numbers. The numbers were obtained using the cycle counter register of the Pentium processor. Linux... [More]
Update: For WAS 8, see https://www.ibm.com/developerworks/mydeveloperworks/blogs/kevgrig/entry/using_visual_configuration_explorer_with_was_8
Visual Configuration Explorer (VCE) is a free tool available in the IBM Support Assistant . The major features I like about it are: 1) it can visualize parts of a WAS configuration such as servers, core groups, etc., 2) it can run the configuration against a database of common issues, and 3) it can compare parts of a configuration to each other (e.g. compare two nodes or servers), or compare two or... [More]
This has nothing to do with WebSphere, but I recently had some local fan failures and did some research: The maximum operating temperatures for an IDE or ATA spinning disk hard drive are 0 to +60°C (+32 to +140°F). Hard drives have internal temperature sensors (which computer programs can also read and report) and some may shut themselves off after hitting +65°C: Warnings are issued when the temperature exceeds the customer set threshold (or the default value 60°C). While this threshold can be disabled by the customer, the shutdown threshold,... [More]
The DynaCache MBean is a scriptable interface to interact with DynaCache caches at runtime: http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/index.jsp?topic=%2Fcom.ibm.websphere.javadoc.doc%2Fweb%2FmbeanDocs%2FDynaCache.html For example, to clear a DynaCache cache instance on a server at runtime, use the following Jython wsadmin code: AdminControl.invoke(AdminControl.completeObjectName("type=DynaCache,node=MYNODE,process=MYSERVER,*"), "clearCache", "MYCACHEINSTANCE") The documentation for clearCache says... [More]
Update: I discussed this issue with Nigel Griffiths (the 'n' in nmon) and he has posted information on mpstat -d which is much easier to use and gives basically the same (and in some senses more) information about process and memory affinity: https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/mpstat_d_and_the_undocumented_stats133?lang=en . You should also read his multi-part series on local, near & far memory: https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/tags/entitlement?lang=en .
When using... [More]