Andrew Hall wrote a wonderful little Perl script named get_memory_use.pl that takes an IBM javacore.txt file and prints the number of reserved bytes for each IBM Java native memory segment (the many pages of hex addresses towards the top of the javacore). Native memory segments are all of the (pooled) native memory allocations that IBM Java makes, such as the Java heap itself, classes, JIT, etc. Since Java 626 (WAS 8), there is a new NATIVEMEMINFO section of the javacore which summarizes this information (actually, it includes this... [More]
It's common wisdom that one should always change one variable at a time when investigating problems, performance testing, etc. The idea is that if you change more than one variable at a time, and the problem goes away, then you don't know which one solved it. For example, let's say one changes the garbage collection policy, maximum heap size, and some of the application code, and performance improves, then one doesn't know what helped. The premise underlying this wisdom is that all variables are independent, which is... [More]
Request Metrics is a feature of traditional WAS that allows simple response time measurement. I've covered this before in Method #1 of HTTP(S) Response Times . This is a quick post on how to do the same for JMS/MDB response times: Ensure "Prepare Servers for Request metrics collection" is checked Select "Custom" for "Components to be instrumented" and select "JMS" and any other relevant components Set "Trace level" to "Hops" Check "Standard Logs"... [More]
I've wanted to write this for a long time and I finally had a situation that called for it. The following tshark Lua script searches network packet captures for anomalous TCP delays in handshakes (long response time to a SYN, response not a SYN/ACK, missing response to a SYN, duplicate SYN) and delays between packets after a handshake. The latter is disabled by default because connections are often re-used, so there may be legitimate delays between the end of one part of a conversation and the beginning of another, so that requires more... [More]
This post covers how to graph arbitrary data from the command line. My requirements are: 1) command line input (e.g. piping straight from file), 2) generate PNG graph, 3) generate ASCII art graph, and 4) UNIX friendly. There are a few popular libraries to do this including gnuplot ( http://www.gnuplot.info/ ) and rrdtool ( http://oss.oetiker.ch/rrdtool/ ); however, I've chosen the R project ( http://www.r-project.org/ ) because I think it's more general purpose. As an example, I will show how to graph IBM HTTP Server mpmstats data (... [More]
When using IBM Java in 64-bit mode and with a maximum heap size less than 25GB, then Compressed References technology (-Xcompressedrefs) is enabled by default (defaults may be different on older versions of Java on some operating systems): http://pic.dhe.ibm.com/infocenter/java7sdk/v7r0/topic/com.ibm.java.zos.71.doc/diag/appendixes/cmdline/Xcompressedrefs.html This option will "decrease the size of Java objects and make more effective use of the available space. The result is less frequent garbage collection and improved memory cache... [More]
Additional trace points for tracking DirectByteBuffer allocations and frees, and the stack traces of who is calling for those have been added in IBM Java 6 SR11 (WAS 188.8.131.52), IBM Java 626 SR3 (WAS 184.108.40.206/220.127.116.11/18.104.22.168), and IBM Java 7 SR3 (WAS 22.214.171.124/126.96.36.199). Note: The overhead of -Xtrace with tpnid should not be significant (unless the rate of DBB allocates and frees is very large); but, as always, you should test the impact in a test environment first. 1. Add the following generic JVM argument... [More]
Before WAS 8.5, the "thread ID" printed in WAS logs (the hexadecimal number after the timestamp) comes from the java/util/logging/LogRecord.getThreadID method. This number was not in javacores, so there was no easy way to correlate javacores with log and trace messages. Moreover, this thread ID was different from java/lang/Thread.getID which might be printed in other components, and that thread ID also wasn't in javacores. (Note: see Mapping Underlying Java Thread Identifiers to those in Logging and Trace for a method to map these values using... [More]
IBM Support Assistant Version version 5 (Team Server) has officially launched . Unlike IBM Support Assistant version 4 which was a heavy, desktop application, ISA5 is fully web-based and either runs from a local application server or you can install its EARs into an existing application server. In addition to a platform for installing and launching tools ( full tools list ), ISA5 comes with a very strong log analysis engine . For example, you can upload a set of WAS logs, click Scan this Case , and it will search for warnings, errors, and other... [More]
Someone asked me how to find who is sending a SIGABRT to a process and below is one technique. If there are better techniques, please leave a comment! Attach to the process using gdb and immediately "continue." When the SIGABRT hits the process, gdb will break execution and leave you at a prompt. Then, simply handle the particular signal you want and print $_siginfo._sifields._kill.si_pid and detach: $ java HelloWorld
Hello World. Waiting indefinitely...
$ ps -elf | grep HelloWorld | grep -v grep
0 S kevin 23947 ...
$ gdb... [More]