The Memory Analyzer Tool (MAT) is the recommended tool for analyzing heapdumps and system dumps, whether to find the cause of an OutOfMemoryError (OOM) or memory leak, to size a JVM, or other types of debugging . The tool is available for free with the IBM Support Assistant Workbench (ISA) or it can be downloaded from eclipse.org and extended with updates to support IBM dump types.
One common memory issue is a Java memory leak. This is not like a traditional memory leak in C/C++ where memory is mis-managed, but rather that the application... [More]
Information Centers are one place where IBM publishes official product documentation. A list of all information centers can be found here: http://www.ibm.com/support/publications/us/library/ . The WAS InfoCenters are: WAS 8: http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/index.jsp WAS 7: http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp WAS 6.1: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp WAS 6.0: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp Each WAS InfoCenter is split up by product... [More]
If you often have to create new application servers, you should consider application server templates: http://www14.software.ibm.com/webapp/wsbroker/redirect?version=compass&product=was-nd-mp&topic=trun_create_templates The process is simple: create one application server, add your customizations (JVM arguments, data sources, etc.), then click Templates... > New, and select that server. Starting in WAS 7, you can even edit the template after it has been created. Then, when you create a new application server, either through the... [More]
I was recently at a customer who believed that they had a Java memory leak. They compared heapdumps and couldn't find anything. They had experienced production OutOfMemoryErrors (OOMs) before (for a different reason), and they were so worried about what they perceived, that they increased the maximum heap size to 4GB so that the JVM could handle a day's worth of work, and then they put in a process to restart the JVMs every night.
At a first glance of verbose garbage collection, I agreed with them (loaded in the wonderful Garbage... [More]
Garbage Collection and Memory Visualizer (GCMV) is a great tool to visualize verbosegc and it is available for free in the IBM Support Assistant . One of its lesser well known features is the ability to compare verbosegc from two different JVM runs. This is particularly useful if you changed something and you want to see the effect. First, load the baseline verbosegc as normal. Next, right click anywhere in the plot area and click 'Compare File...': Next, ensure that the X-axis uses a relative format such as hours, instead of date. Otherwise,... [More]
IBM Research recently made its WAIT tool (Whole-system Analysis of Idle Time) available to the public: https://wait.researchlabs.ibm.com/
The WAIT tool takes javacores and operating system statistics snapshots for a period of time as input, and produces a rich webpage as output that visualizes this data. Since WAIT only uses javacores and operating system scripts, you don't need to install anything or even restart the Java processes to use WAIT. For administrators familiar with the performance, hang, and high CPU MustGathers , the... [More]
The IBM Extensions for Memory Analyzer (IEMA) are a set of free plugins that extend the powerful Memory Analyzer Tool with product-specific knowledge when analyzing heapdumps and system dumps. For example, IEMA can list out all HTTP sessions, attributes such as their JSESSIONIDs and user names, and session attributes in a nice tree view for WAS dumps (note that much of the additional information is only available with system dumps on IBM JVMs). For more information on WAS-related extensions in IEMA, see... [More]
The IBM Java HealthCenter is a low overhead agent shipped with the JVM that can provide deep insight into JVM activity for IBM Java 5 and above: http://www.ibm.com/developerworks/java/jdk/tools/healthcenter/ . HealthCenter is similar to VisualVM . HealthCenter can provide information on classloading, JVM arguments, garbage collection, file I/O, locking, native memory, threads, and profiling. The last item, profiling, is my favorite feature of HealthCenter -- it is a very low overhead, sampling profiler that can pinpoint high CPU issues or... [More]
Update : ActiveCount has been made a part of the PMI Basic collection (enabled by default) in WAS 22.214.171.124 and 126.96.36.199 .
WebSphere Application Server comes with the "Basic" level enabled in the Performance Monitoring Infrastructure (PMI) on every server by default . This usually has an overhead of about 2-3% . The Basic level comes with one statistic for all thread pools: PoolSize (see the "Level" column in the link). This is defined as "the average number of threads in pool." This is a bit confusing: it is... [More]
WebSphere Application Server comes with a built-in HTTP(S) server. This post covers different methods of printing the response times of HTTP(S) requests. If all you need are averages, then the built-in Performance Monitoring Infrastructure (PMI) provides average statistics for HTTP(S) response times. However, if you need information on particular requests, then averages may not help . The most robust solution is to use a monitoring product such as ITCAM . This article will cover the basic capabilities that are built-in to WAS. Method #0: Web... [More]
I'm not sure how this applies to modern computer chips and operating systems, but here is interesting research from Liedtke in 1995 and 1997 showing the overhead of system calls:
For measuring the system-call overhead, getpid, the shortest Linux system call, was examined. To measure its cost under ideal circumstances, it was repeatedly invoked in a tight loop. Table 2 shows the consumed cycles and the time per invocation derived from the cycle numbers. The numbers were obtained using the cycle counter register of the Pentium processor. Linux... [More]
Update: For WAS 8, see https://www.ibm.com/developerworks/mydeveloperworks/blogs/kevgrig/entry/using_visual_configuration_explorer_with_was_8
Visual Configuration Explorer (VCE) is a free tool available in the IBM Support Assistant . The major features I like about it are: 1) it can visualize parts of a WAS configuration such as servers, core groups, etc., 2) it can run the configuration against a database of common issues, and 3) it can compare parts of a configuration to each other (e.g. compare two nodes or servers), or compare two or... [More]
This has nothing to do with WebSphere, but I recently had some local fan failures and did some research: The maximum operating temperatures for an IDE or ATA spinning disk hard drive are 0 to +60°C (+32 to +140°F). Hard drives have internal temperature sensors (which computer programs can also read and report) and some may shut themselves off after hitting +65°C: Warnings are issued when the temperature exceeds the customer set threshold (or the default value 60°C). While this threshold can be disabled by the customer, the shutdown threshold,... [More]
The DynaCache MBean is a scriptable interface to interact with DynaCache caches at runtime: http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/index.jsp?topic=%2Fcom.ibm.websphere.javadoc.doc%2Fweb%2FmbeanDocs%2FDynaCache.html For example, to clear a DynaCache cache instance on a server at runtime, use the following Jython wsadmin code: AdminControl.invoke(AdminControl.completeObjectName("type=DynaCache,node=MYNODE,process=MYSERVER,*"), "clearCache", "MYCACHEINSTANCE") The documentation for clearCache says... [More]
Update: I discussed this issue with Nigel Griffiths (the 'n' in nmon) and he has posted information on mpstat -d which is much easier to use and gives basically the same (and in some senses more) information about process and memory affinity: https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/mpstat_d_and_the_undocumented_stats133?lang=en . You should also read his multi-part series on local, near & far memory: https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/tags/entitlement?lang=en .
When using... [More]
The WAS Data Replication Service (DRS) (  ,  ) provides a mechanism to share data across JVMs. For example, this can be used by Dynacache for servlet caching , or by applications in an object cache using the DistributedMap API . There are a lot of knobs including security, invalidations, disk offload, different mechanisms and topologies for sharing, dependencies, timeouts, ring buffers, tuning such as batching, and more. DRS works on top of the High Availability Manager and DCS.
The sharing mechanisms are: Both Push and Pull... [More]
Wireshark (formerly called Ethereal) is a great, free, open source tool to analyze network packet captures. To gather packet captures on various operating systems, see http://www-01.ibm.com/support/docview.wss?uid=swg21175744 . Wireshark supports decrypting some types of SSL traffic: http://wiki.wireshark.org/SSL . The following developerWorks article covers the basics of using this feature: http://www.ibm.com/developerworks/web/tutorials/wa-tomcat/section4.html . In this post, I'll cover some WAS specific tips.
By default , WAS uses... [More]
Wireshark supports automation through a scripting language called Lua . I've created a simple Lua script which extracts all HTTP requests and responses to a file. For example:
["Dec 8, 2011 10:33:30.267266000 PST" src:127.0.0.1:37204 dst:127.0.0.1:9094 stream:0] HTTP Request: POST localhost:9094/ibm/console/j_security_check
["Dec 8, 2011 10:33:30.292939000 PST" src:127.0.0.1:9094 dst:127.0.0.1:37204 stream:0] HTTP Response: 302 Found
Here's how to run the script:
tshark -r $FILE.pcap -X lua_script:printhttp.lua... [More]
The IBM HTTP Server (IHS) ships with a cool little command line utility called Apache Bench (ab) because IHS is based on httpd . At its simplest, you pass the number of requests you want to send (-n), at what concurrency (-c) and the URL to benchmark. ab will return various statistics on the responses (mean, median, max, standard deviation, etc.). This is really useful when you want to "spot check" backend server performance or compare two different environments, because you do not need to install complex load testing software, and... [More]
The Memory Analyzer Tool (MAT) creates index files after parsing a dump the first time so that reloading a dump is much faster. These index files are saved to the same directory as the dump file. In fact, you can load a dump on a beefier machine and then copy over the index files to your slower machine and get the benefits of loading a dump faster.
One issue I recently hit was that every time I would reload a dump, I would get the message "Reparsing heap dump file due to out of date index file," and the dump would be completely reloaded.... [More]