For platforms that run the IBM JVM, the strategic direction for memory analysis is to move away from PHDs and towards system dumps because they have so much more information . System dumps are the operating system core dumps usually produced with crashes (AIX/Linux=core, z/OS=SVCDUMP, Windows=minidump). In recent versions of the IBM JVM, a system dump can be loaded directly into the Memory Analyzer Tool without running jexract on it. The versions are: Java 5 >= SR12, Java 6 >= SR9, Java 626, and Java 7. The matching WAS versions are WAS >=... [More]
On the IBM JVM, the various environment variables used to change dump parameters (e.g. IBM_JAVACOREDIR, etc.) have been deprecated in favor of the -Xdump generic JVM command line argument.
Someone asked how to change the directory where all the dump artifacts go using -Xdump. Here it is for *nix, and just change /tmp/ in each argument to the desired directory (on Windows, use the Windows path syntax):
-Xdump:java:file=/tmp/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:heap:file=/tmp/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd... [More]
In recent versions of IHS and httpd, the time stamp in the access is log is the time the request arrived in IHS, not when the response completed. Originally, httpd used the time when the request completed . The code is in mod_log_config in the log_request_time function: httpd 1.3 ; httpd 2.0 . This is the primary reason why you may see "out of order" timestamps in the access log. To figure out when the response completed, you'll need %D or %T in the LogFormat and add them to the timestamp. Here's a comment in the code that explains... [More]
I couldn't find a better way to select attributes of a static class instance. Using classof() doesn't help because there could be zero instances of a class. The following example checks if the JVM is a z/OS control region. The trailing space character within the double quotes is important for accuracy.
SELECT c.controller FROM INSTANCEOF java.lang.Class c WHERE c.@displayName.contains("class com.ibm.ws.management.util.PlatformHelperImpl ")
WAS exposes a JVM MBean for each process that has methods to create thread dumps, heap dumps, and system dumps. For example, to produce a thread dump on server1, use this wsadmin command (-lang jython):
The dumpThreads functionality is different depending on the operating system:
POSIX (AIX, Linux, Solaris, etc.): kill(pid, SIGQUIT)
z/OS: In recent versions, produces a javacore,... [More]
I'll be presenting a WebSphere Technical Exchange on May 1st @ 11AM Eastern . The topic will be a deep dive on IBM Java Health Center, primarily around its low-overhead, production-ready profiling capabilities to understand CPU issues on IBM JVMs. The slides are already available here: http://www-01.ibm.com/support/docview.wss?uid=swg27024833&aid=1 .
Whether you're using the older listener ports or the newer activation specifications, tuning the relevant thread pools is a key aspect of MDB performance. Thread pools are configured at a server level, while the number and concurrency of the MDBs is configured independently. Therefore, if your maximum thread pool size is too small, messages may queue unnecessarily. Below are equations for the most common setups which define how to setup the relevant thread pool maximums to avoid queuing. The x=1..n items are all the items of that type... [More]
In computing history, there was a famous crusade by Larry Tesler -- a titan of the industry; he worked at Xerox PARC (Smalltalk), Apple, Amazon, and Yahoo -- which he called "no modes." He said, for example, that you shouldn't have to enter a "mode" just to type text. You should be able to click and type. It was a revolutionary idea back then. There are still some popular modal programs today such as vi and Emacs, and the mode combinations make their users look like wizards, but in general, modes are dead.
However, I have... [More]
A customer recently gave me a very large heapdump which I tried to open in the Memory Analyzer Tool. It chugged for a while and then my computer overheated and suddenly shut itself off. I suspect this is a bug in the Linux kernel which improperly ramps the i7 processor and the fan can't keep up (or maybe my fan just needs a cleaning). The workaround for this on Linux was to reduce the maximum speed of my processors before opening the dump:
for i in `ls /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq`; do sudo bash -c "echo 1500000... [More]
A common aspect to a problem is that an application worked and then the environment (WAS, etc.) was upgraded and the application stopped working. Many customers then say, "therefore, the product is the root cause." It is easy to show that this is a fallacy (neither necessary nor sufficient) with a real world example: A recent customer upgraded from WAS 6.1 to WAS 7 without changing the application and it started to throw various exceptions. It turned out that the performance improvements in WAS 7 and Java 6 exposed existing... [More]
A previous post covered an older way of gathering configuration for Visual Configuration Explorer (VCE) using the VCE Headless Runtime, exported from ISA. The newer and preferred approach is to use the IBM Support Assistant Lite data collector, and this will work with WAS 8 (I've also tested this on 7). It does not work on V8.5.
Download the ISA Lite script: https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?source=swg-isalite&S_PKG=wasunixwin
Extract the ISA Lite script into the <WAS> root directory.
A recent customer was comparing performance between WAS and Tomcat. Tomcat was performing much better. The application used temporary files intensively. After investigating thread dumps, we found that the sampled WAS threads showed much more temprorary file I/O activity than Tomcat threads. Next, we discovered that Tomcat changes Java's default temporary directory to the "temp" subdirectory of the Tomcat installation using the -Djava.io.tmpdir system property.
It turned out that Tomcat happened to be installed on a faster disk than... [More]
One simple and very useful indicator of process health and load is its TCP activity. The following script takes a set of ports and summarizes how many TCP sockets are established, opening, and closing for each port. It has been tested on Linux and AIX. Example output:
$ portstats.sh 80 443
PORT ESTABLISHED OPENING CLOSING
80 3 0 0
443 10 0 2
Total 13 0 2
echo "usage:... [More]
It is generally a malpractice for an application to call System.gc() or Runtime.gc() (hereafter referring to both as System.gc(), since the former simply calls the latter). By default, these calls instruct the JVM to perform a full garbage collection, including tenured spaces and a full compaction. These calls may be unnecessary and may increase the proportion of time spent in garbage collection than otherwise would have occurred if the garbage collector was left alone.
The generic JVM arguments -Xdisableexplicitgc (IBM) and... [More]
Below is a simple wsadmin script that calculates the approximate start time of a set of servers using the UpTime PMI statistic . For example:
$ ./wsadmin.sh -lang jython -username wsadmin -password wsadmin -f uptime.py -server server1
WASX7209I: Connected to process "dmgr" on node localhostCellManager11 using SOAP connector; The type of process is: DeploymentManager
WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[-server, server1]"