I've wanted to write this for a long time and I finally had a situation that called for it. The following tshark Lua script searches network packet captures for anomalous TCP delays in handshakes (long response time to a SYN, response not a SYN/ACK, missing response to a SYN, duplicate SYN) and delays between packets after a handshake. The latter is disabled by default because connections are often re-used, so there may be legitimate delays between the end of one part of a conversation and the beginning of another, so that requires more... [More]
This post covers how to graph arbitrary data from the command line. My requirements are: 1) command line input (e.g. piping straight from file), 2) generate PNG graph, 3) generate ASCII art graph, and 4) UNIX friendly. There are a few popular libraries to do this including gnuplot ( http://www.gnuplot.info/ ) and rrdtool ( http://oss.oetiker.ch/rrdtool/ ); however, I've chosen the R project ( http://www.r-project.org/ ) because I think it's more general purpose.
As an example, I will show how to graph IBM HTTP Server mpmstats... [More]
When using IBM Java in 64-bit mode and with a maximum heap size less than 25GB, then Compressed References technology (-Xcompressedrefs) is enabled by default (defaults may be different on older versions of Java on some operating systems): http://pic.dhe.ibm.com/infocenter/java7sdk/v7r0/topic/com.ibm.java.zos.71.doc/diag/appendixes/cmdline/Xcompressedrefs.html
This option will "decrease the size of Java objects and make more effective use of the available space. The result is less frequent garbage collection and improved memory... [More]
Additional trace points for tracking DirectByteBuffer allocations and frees, and the stack traces of who is calling for those have been added in IBM Java 6 SR11 (WAS 184.108.40.206), IBM Java 626 SR3 (WAS 220.127.116.11/18.104.22.168/22.214.171.124), and IBM Java 7 SR3 (WAS 126.96.36.199/188.8.131.52).
Important Note: Until IBM Java 7.1, using -Xtrace may have a significant performance overhead because the JIT behavior is impacted for all methods. Before using this option in production, test its impact in a test environment.
1. Add the following generic JVM argument... [More]
Before WAS 8.5, the "thread ID" printed in WAS logs (the hexadecimal number after the timestamp) comes from the java/util/logging/LogRecord.getThreadID method. This number was not in javacores, so there was no easy way to correlate javacores with log and trace messages. Moreover, this thread ID was different from java/lang/Thread.getID which might be printed in other components, and that thread ID also wasn't in javacores. (Note: see Mapping Underlying Java Thread Identifiers to those in Logging and Trace for a method to map these values using... [More]
IBM Support Assistant Version version 5 (Team Server) has officially launched . Unlike IBM Support Assistant version 4 which was a heavy, desktop application, ISA5 is fully web-based and either runs from a local application server or you can install its EARs into an existing application server.
In addition to a platform for installing and launching tools ( full tools list ), ISA5 comes with a very strong log analysis engine . For example, you can upload a set of WAS logs, click Scan this Case , and it will search for warnings, errors, and... [More]
Someone asked me how to find who is sending a SIGABRT to a process and below is one technique. If there are better techniques, please leave a comment!
Attach to the process using gdb and immediately "continue." When the SIGABRT hits the process, gdb will break execution and leave you at a prompt. Then, simply handle the particular signal you want and print $_siginfo._sifields._kill.si_pid and detach:
$ java HelloWorld
Hello World. Waiting indefinitely...
$ ps -elf | grep HelloWorld | grep -v grep
0 S kevin 23947... [More]
I'll be presenting a WebSphere Technical Exchange on November 12th @ 11AM Eastern . It is free for all to join and includes a remote presentation and a question & answer session. The topic will be Linux Native Memory Problem Determination Techniques and Tools for WebSphere Application Server. Slides are available here: http://www-01.ibm.com/support/docview.wss?uid=swg27039764&aid=1
Our SWAT team's common customer malpractices document has been refreshed and published as an IBM Redpaper: http://www.redbooks.ibm.com/redpieces/abstracts/redp5033.html?Open .
After most of our engagements, we consider what customer practices caused or contributed to the critical situation. The biggest one? No test environment is equal to the production environment: 35.1%.
Please read this paper and pass it on to incentivize management to invest in the right areas to help avoid critical situations.
It may be useful to understand what PID is sending a kill signal to a process on AIX. You can use this kernel trace:
Login as root
# rm -rf /tmp/aixtrace; mkdir /tmp/aixtrace/; cd /tmp/aixtrace/
# trace -C all -a -T 10M -L 20M -n -j 134,139,465,14e,46c -o ./trc
... Reproduce the problem ... e.g. kill -3 7667754
# cp /etc/trcfmt .
# trcnm -a > trace.nm
# LDR_CNTRL=MAXDATA=0x80000000 gensyms > trace.syms
# LDR_CNTRL=MAXDATA=0x80000000 gennames -f > gennames.out
# pstat -i > trace.inode
# ls -al /dev >... [More]
The latest version of the Interactive Diagnostic Data Explorer (IDDE) tool, available in the IBM Support Assistant , supports a subset of the IBM Extensions for Memory Analyzer plugins (if installed) by typing Ctrl+Space a second time after typing the first !, and it also supports the old DumpAnalyzer plugins by typing Ctrl+Space a second time. For example, running the old DumpAnalyzer basic analyzer:
This is part five in the series on understanding total virtual memory usage from a core dump on Linux ( part 1 , part 2 , part 3 , part 4 ).
In part 4, I alluded to the fact that the gcore command in gdb actually walks the memory regions itself when writing the core dump. This got me thinking: what if this is different than how the kernel writes the core? To get the kernel to produce a core, the only way seems to be to destructively kill the process (e.g. kill -6 or kill -11). By default, IBM Java will catch these signals before the process... [More]
This is part four in the series on understanding total virtual memory usage from a core dump on Linux ( part 1 , part 2 , part 3 ).
Previously, I recommended using a script to add a timestamp to a core file name when using gcore so that MAT could know when the dump was taken. Now, I'll make the case for a stronger recommendation to completely avoid using gcore and use IBM Java's built-in capabilities of taking a core dump instead.
Gcore is simply a shell script available on some flavors of Linux:
$ whereis gcore
We are working to add PDB symbols for WAS native libraries and we ran into a subtle issue. As Microsoft states :
PDB files are generated if a project is built by using the /Zi or /ZI (Produce PDB Information) compiler switch... Generating PDB files for release executables does not affect any optimizations, or significantly alter the size of the generated files... For this reason, you should always produce PDB files...
We added /Zi to the compiler (cl.exe) and it produced a PDB file; however, when we grabbed a dump of the process and... [More]