Running IBM Java HealthCenter in headless mode
kgibm 0600027VAP Visits (7678)
The IBM Java HealthCenter is a low overhead agent shipped with the JVM that can provide deep insight into JVM activity for IBM Java 5 and above: http
You have to restart the JVM with the generic JVM argument -Xhealthcenter to enable it. By default, HealthCenter opens port 1972 (or the next available port incremented from that) and you connect to the agent using the HealthCenter client available in IBM Support Assistant. Of course, this raises a lot of issues. First, many customers have firewalls or don't allow JVMs to open additional ports, particularly on production systems. Second, if you're profiling multiple JVMs, it's difficult to manually monitor all of them real time. Third, it won't catch everything before the first time a client connects. Finally, there is additional overhead in transferring the data over the network.
However, HealthCenter does have a headless mode in which it will write the data to local files on the JVM's system and the resulting "HCD" file can be loaded into the HealthCenter client for post analysis. To run in this mode, use the generic JVM argument -Xhe
The second limitation is that the JVM must be fully stopped for the HCD file to be produced. The healthcenter.hcd file will also be written into either the current working directory or into the com.
If you try to start the JVM in headless mode and it fails to restart, then the agent is probably too old. If you check the native_std*.log files, you'll probably see a message like the following:
SEVERE: Health Center agent failed to start.
In this case, you'll just need to update the HealthCenter agent libraries. The latest libraries can be downloaded for each platform from: http
You can check which version of the HealthCenter agent is installed by running:
$ <WAS>/java/bin/java -version -Xhe
Note: When running level=headless combined with -version, you will have to Ctrl+C to kill the program. This is a known limitation of HealthCenter when running -version.
Some customers are weary about updating the WAS binaries; however, the files updated are only those of HealthCenter. HealthCenter is shipped with the JVM, but just as an optional feature. If you do not use -Xhealthcenter, then the libraries are not loaded. So although you're updating files within the WAS install, it's equivalent to updating something like the doc directory in the JVM -- it's something that's shipped with the JVM, but not normally used, so you're not changing the functionality of the installation or changing anything that had been previously used (obviously, the functionality and dynamics of the JVM change when HealthCenter is enabled).
You may receive errors unzipping the file because the HealthCenter libraries are "in use." This is because you had just tried to run HealthCenter. First, make sure that the JVM is down. If it still doesn't work, you may need to stop all other JVMs on that node and the node agent. If it still doesn't work, and the JVM is run by a non-root user, then try extracting the zip using the root user, and then chown to the non-root user.
Once you've got the HCD file, start the HealthCenter client in ISA. When you first load the client, it may ask you to connect to a JVM, just click Cancel. Then click File > Open File... and select the HCD file (ensure that it has the extension .hcd or .zip). Then click on Profiling, and you'll see something like the following:
You may notice that some methods do not have any names but are just hexadecimal addresses. This may be due to an old version of the JVM or the agent. Also, for headless mode, some of this is unavoidable, particularly for methods that are executed very early during startup (e.g. classloading). For any kind of normal runs where you don't care about startup, then this is usually not an issue. However, if you do see hot methods without method names, you can infer what they are by looking at what the methods call and who calls them. For example, in the following case, there is a hot method without a name, and we can see from the call path that it is related to security, particularly around creating an LTPA key (i.e user login).
The Self% is how often that method was sampled at the top of the stacks, and the Tree% is how often that method was sampled somewhere in the stacks. I usually first start by looking at the highest Self% (the default sort in the client). Next, I'll sort by Tree%, and I'll scroll past methods that usually don't do much (for example, a Servlet doPost, or a servlet filter), and the first method that is after these types of methods is usually what is doing most of the activity. This is more of an art than a science.
You can also zoom in on a particular time period by dragging a rectangle around a particular time area beneath the profiling view. This is great for comparing the activity at two different times in the run.
The sampling profiler frequently samples the stacks of the CPUs. Highlighted methods in the client are not necessarily those that consume CPU. If the agent samples stacks which are constantly making web service calls, this will show up in the client. This is in many cases even better than just a CPU profiler, since it can pinpoint almost any type of performance issue, not just excessive CPU usage.
In summary, IBM Java Health Center is an extremely powerful tool which has a very low-overhead, headless, sampling profiler on recent versions of WAS (>= 220.127.116.11 and >= 18.104.22.168) which can be used to determine the root cause of performance issues.