For platforms that run the IBM JVM, the strategic direction for memory analysis is to move away from PHDs and towards system dumps because they have so much more information . System dumps are the operating system core dumps usually produced with crashes (AIX/Linux=core, z/OS=SVCDUMP, Windows=minidump). In recent versions of the IBM JVM, a system dump can be loaded directly into the Memory Analyzer Tool without running jexract on it. The versions are: Java 5 >= SR12, Java 6 >= SR9, Java 626, and Java 7. The matching WAS versions are WAS >=... [More]
Excessive native memory usage by java.nio.DirectByteBuffers is a classic problem with any generational garbage collector such as gencon (which is the default starting in IBM Java 6.26/WAS 8), particularly on 64-bit. DirectByteBuffers (DBBs) are Java objects that allocate and free native memory. DBBs use a PhantomReference which is essentially a more flexible finalizer and they allow the native memory of the DBB to be freed once there are no longer any live Java references. Finalizers and their ilk are generally not recommended because... [More]
When building a custom Fedora Linux kernel , after you've created a kernel patch and add PatchN/ApplyPatch lines to kernel.spec and run rpmbuild, you may get a prompt that asks 'File to patch:'. In my case, this is because I added the ApplyPatch line after the last ApplyPatch line in kernel.spec, but this turned out to be in the wrong directory. Instead, search for "# END OF PATCH APPLICATIONS" and add the ApplyPatch line before this comment (confusingly, the comment does not mark the last ApplyPatch line!).
This post covers how to graph arbitrary data from the command line. My requirements are: 1) command line input (e.g. piping straight from file), 2) generate PNG graph, 3) generate ASCII art graph, and 4) UNIX friendly. There are a few popular libraries to do this including gnuplot ( http://www.gnuplot.info/ ) and rrdtool ( http://oss.oetiker.ch/rrdtool/ ); however, I've chosen the R project ( http://www.r-project.org/ ) because I think it's more general purpose.
As an example, I will show how to graph IBM HTTP Server mpmstats... [More]
This has nothing to do with WebSphere, but I recently had some local fan failures and did some research: The maximum operating temperatures for an IDE or ATA spinning disk hard drive are 0 to +60°C (+32 to +140°F). Hard drives have internal temperature sensors (which computer programs can also read and report) and some may shut themselves off after hitting +65°C: Warnings are issued when the temperature exceeds the customer set threshold (or the default value 60°C). While this threshold can be disabled by the customer, the shutdown threshold,... [More]
WAS exposes a JVM MBean for each process that has methods to create thread dumps, heap dumps, and system dumps. For example, to produce a thread dump on server1, use this wsadmin command (-lang jython):
The dumpThreads functionality is different depending on the operating system:
POSIX (AIX, Linux, Solaris, etc.): kill(pid, SIGQUIT)
z/OS: In recent versions, produces a javacore,... [More]
Garbage Collection and Memory Visualizer (GCMV) is a great tool to visualize verbosegc and it is available for free in the IBM Support Assistant . One of its lesser well known features is the ability to compare verbosegc from two different JVM runs. This is particularly useful if you changed something and you want to see the effect. First, load the baseline verbosegc as normal. Next, right click anywhere in the plot area and click 'Compare File...': Next, ensure that the X-axis uses a relative format such as hours, instead of date. Otherwise,... [More]
There are two broad categories of profilers: statistical/sampling profilers which sample call stacks and tracing profilers which record method entry/exit. IBM Health Center is an example of a sampling profiler and it is usually the best way to profile. In some cases, a tracing profiler is useful, such as analyzing low volume background work when there is no application activity (for example, to investigate the base footprint of a product like WAS). Even high frequency sampling may not capture enough data.
In our case, we were looking for... [More]
The Memory Analyzer Tool (MAT) is the recommended tool for analyzing heapdumps and system dumps, whether to find the cause of an OutOfMemoryError (OOM) or memory leak, to size a JVM, or other types of debugging . The tool is available for free with the IBM Support Assistant Workbench (ISA) or it can be downloaded from eclipse.org and extended with updates to support IBM dump types.
One common memory issue is a Java memory leak. This is not like a traditional memory leak in C/C++ where memory is mis-managed, but rather that the application... [More]
Surprisingly, even the latest version of HP-UX does not provide a simple tool (such as "ps") to print the full command line of a running program (no equivalent of Solaris /usr/ucb/ps). The -x parameter of ps only prints the first 1024 characters , which is often insufficient for Java programs: Only a subset of the command line is saved by the kernel; as much of the command line will be displayed as is available... The value of DEFAULT_CMD_LINE_WIDTH should be between 64 and 1020 (... [More]
Information Centers are one place where IBM publishes official product documentation. A list of all information centers can be found here: http://www.ibm.com/support/publications/us/library/ . The WAS InfoCenters are: WAS 8: http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/index.jsp WAS 7: http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp WAS 6.1: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp WAS 6.0: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp Each WAS InfoCenter is split up by product... [More]
This is part four in the series on understanding total virtual memory usage from a core dump on Linux ( part 1 , part 2 , part 3 ).
Previously, I recommended using a script to add a timestamp to a core file name when using gcore so that MAT could know when the dump was taken. Now, I'll make the case for a stronger recommendation to completely avoid using gcore and use IBM Java's built-in capabilities of taking a core dump instead.
Gcore is simply a shell script available on some flavors of Linux:
$ whereis gcore
When using IBM Java in 64-bit mode and with a maximum heap size less than 25GB, then Compressed References technology (-Xcompressedrefs) is enabled by default (defaults may be different on older versions of Java on some operating systems): http://pic.dhe.ibm.com/infocenter/java7sdk/v7r0/topic/com.ibm.java.zos.71.doc/diag/appendixes/cmdline/Xcompressedrefs.html
This option will "decrease the size of Java objects and make more effective use of the available space. The result is less frequent garbage collection and improved memory... [More]
IBM Support Assistant Version version 5 (Team Server) has officially launched . Unlike IBM Support Assistant version 4 which was a heavy, desktop application, ISA5 is fully web-based and either runs from a local application server or you can install its EARs into an existing application server.
In addition to a platform for installing and launching tools ( full tools list ), ISA5 comes with a very strong log analysis engine . For example, you can upload a set of WAS logs, click Scan this Case , and it will search for warnings, errors, and... [More]
In recent versions of IHS and httpd, the time stamp in the access is log is the time the request arrived in IHS, not when the response completed. Originally, httpd used the time when the request completed . The code is in mod_log_config in the log_request_time function: httpd 1.3 ; httpd 2.0 . This is the primary reason why you may see "out of order" timestamps in the access log. To figure out when the response completed, you'll need %D or %T in the LogFormat and add them to the timestamp. Here's a comment in the code that explains... [More]