The WAS Data Replication Service (DRS) (  ,  ) provides a mechanism to share data across JVMs. For example, this can be used by Dynacache for servlet caching , or by applications in an object cache using the DistributedMap API . There are a lot of knobs including security, invalidations, disk offload, different mechanisms and topologies for sharing, dependencies, timeouts, ring buffers, tuning such as batching, and more. DRS works on top of the High Availability Manager and DCS.
The sharing mechanisms are: Both Push and Pull... [More]
Wireshark (formerly called Ethereal) is a great, free, open source tool to analyze network packet captures. To gather packet captures on various operating systems, see http://www-01.ibm.com/support/docview.wss?uid=swg21175744 . Wireshark supports decrypting some types of SSL traffic: http://wiki.wireshark.org/SSL . The following developerWorks article covers the basics of using this feature: http://www.ibm.com/developerworks/web/tutorials/wa-tomcat/section4.html . In this post, I'll cover some WAS specific tips.
By default , WAS uses... [More]
Wireshark supports automation through a scripting language called Lua . I've created a simple Lua script which extracts all HTTP requests and responses to a file. For example:
["Dec 8, 2011 10:33:30.267266000 PST" src:127.0.0.1:37204 dst:127.0.0.1:9094 stream:0] HTTP Request: POST localhost:9094/ibm/console/j_security_check
["Dec 8, 2011 10:33:30.292939000 PST" src:127.0.0.1:9094 dst:127.0.0.1:37204 stream:0] HTTP Response: 302 Found
Here's how to run the script:
tshark -r $FILE.pcap -X lua_script:printhttp.lua... [More]
The IBM HTTP Server (IHS) ships with a cool little command line utility called Apache Bench (ab) because IHS is based on httpd . At its simplest, you pass the number of requests you want to send (-n), at what concurrency (-c) and the URL to benchmark. ab will return various statistics on the responses (mean, median, max, standard deviation, etc.). This is really useful when you want to "spot check" backend server performance or compare two different environments, because you do not need to install complex load testing software, and... [More]
The Memory Analyzer Tool (MAT) creates index files after parsing a dump the first time so that reloading a dump is much faster. These index files are saved to the same directory as the dump file. In fact, you can load a dump on a beefier machine and then copy over the index files to your slower machine and get the benefits of loading a dump faster.
One issue I recently hit was that every time I would reload a dump, I would get the message "Reparsing heap dump file due to out of date index file," and the dump would be completely reloaded.... [More]
Update (May 30, 2013) : Note that it is preferable to use IBM Java's capabilities to produce system dumps instead of gcore. See why .
The Linux gcore command (also part of gdb) is a very useful command to create a coredump of a running process without killing the process. This is particularly useful starting with WAS 220.127.116.11 and 18.104.22.168 as a core dump can be read directly by the Memory Analyzer Tool, without running jextract, so that core dumps become as easy to use as PHD heapdumps, while providing much more information .
In a previous post , I covered some of the ways to log per-HTTP response time information in WAS. Good news -- a new and more standard approach has been added in WAS 22.214.171.124 through PM46717 . WAS has had NCSA logging -- which is most commonly know through the LogFormat that httpd/IHS uses by default in its access/error logs -- but the format in WAS was not customizable, and did not have response times. WAS 126.96.36.199 introduces a customizable format which includes %D: "The elapsed time of the request. Millisecond accuracy, microsecond... [More]
Recent versions of both the IBM and Oracle JVMs support late attach functionality. This allows you to "inject" either a native JVMTI or Java agent into a running JVM which can run arbitrary code without having to restart the JVM. For IBM Java, the versions are Java 5 >= SR10 (disabled by default) [WAS >= 188.8.131.52], Java 6 >= SR6 (enabled by default on non-z/OS platforms) [WAS >= 184.108.40.206], Java 6 R26 [WAS 8], and Java 7.
If you're in a really sticky situation where you can't restart a server, but you need to do something... [More]
Someone asked for a quick reference on native eye catchers, and I had a hard time finding a comprehensive reference through the search engines, so here's a quick rundown.
Eye-catchers are generally used to aid in tracking down native memory leaks or native OutOfMemoryErrors. After you've checked all the obvious culprits, at some point you may have to manually page through a hexdump. An eye-catcher, as its name suggests, is some sequence of bytes that has a low probability of randomly appearing in memory. If you see one of your eye-catchers,... [More]
Starting in WAS 220.127.116.11 running on IBM JVMs, the default artifacts produced on an OutOfMemoryError have changed. In addition to the old defaults (first four OOMs create a PHD, javacore, and snapdump), a system dump will be produced on the first OOM: http://publib.boulder.ibm.com/infocenter/wasinfo/v8r0/topic/com.ibm.websphere.base.doc/info/aes/ae/ctrb_java626.html .
Some customers have become used to system dumps meaning crashes, so don't be worried. System dumps are a richer and alternative artifact to understand OOMs using the Memory... [More]
I recently saw an interesting situation where it turned out that a network packet was truly lost in transmission (root cause yet to be determined, but probably in the operating system or some security software). This happened between IHS and WAS and it caused IHS to mark the WAS server down. The symptom in the logs was a connection reset error. There are two points I thought were interesting that I'll cover: 1) If the customer had only gathered network trace from the IHS side, they might have concluded the wrong thing, and 2) It may be... [More]
When building a custom Fedora Linux kernel , after you've created a kernel patch and add PatchN/ApplyPatch lines to kernel.spec and run rpmbuild, you may get a prompt that asks 'File to patch:'. In my case, this is because I added the ApplyPatch line after the last ApplyPatch line in kernel.spec, but this turned out to be in the wrong directory. Instead, search for "# END OF PATCH APPLICATIONS" and add the ApplyPatch line before this comment (confusingly, the comment does not mark the last ApplyPatch line!).
This error can occur on Windows, particularly around socket operations. The error is translated from the Winsock error code WSAENOBUFS , 10055. The most common cause of this error is that Windows is configured for the default maximum of 5,000 in-use ports. This can be monitored by watching netstat or perfmon and can be changed with the MaxUserPort registry parameter .
A more advanced cause for this error is non-paged pool exhaustion. The paged and nonpaged pools are areas of memory for certain Windows kernel-mode allocations such as the... [More]
This is a simple wsadmin script that can start or stop binary PMI/TPV logs (of infinite duration and up to 200MB rolled over 20 files) across a cell. The two parameters required are -action (start, stop or list) and -userprefs with your user name (I couldn't find a way to query the user name that was passed in). With no other arguments, this will start PMI logging (at whatever level configured for, which by default is PMI Basic) on every application server in the cell. If you pass -node, it will only do it on the application servers on that... [More]
A recent customer on RedHat Enterprise Linux 6 (RHEL6) was running WAS 8, 64-bit. We noticed that the virtual size of the process was over 14GB (ps -p $PID -o vsz,rss). The maximum heap (-Xmx) was 5.5GB, so we were concerned there was a native memory leak. In IBM Java 626 (which ships with WAS 8), javacores have a NATIVEMEMINFO structure which tracks most JVM native allocations, including -Xmx itself, classes, classloaders, JIT, shared classes, and even some SDK native allocations like DirectByteBuffers. These are the most common things to leak... [More]