One of the most common requests we receive in Java L3 service is to include more data in javacore files.
Javacore files are human-readable text files which summarise the state of the JVM. By default, we write one to disk when you send SIGQUIT to the Java process. They are also written in certain other situations, for example when an OutOfMemoryError is thrown.
Over time, we have added lots of useful information to javacores. For example, in recent service refreshes we have added details of environment variable settings, ulimit settings, and native stacks for all threads in the JVM.
But what if you want to capture some information that we don't include?
If the information you want is available from the operating system, then a little-known feature in the JVM called "dump tool support" can help.
Let's say you're interested in gathering some information about virtual memory usage. On AIX, that's gathered using the "svmon" command.
By adding a JVM command-line parameter, you can change the behaviour of the JVM so that when SIGQUIT is sent to the JVM it runs svmon as well as writing a javacore. Here's how that would look:
-Xdump:tool:events=user,exec="svmon"
Here's the output that gives:
JVMDUMP006I Processing Dump Event "user", detail "#00000000" - Please Wait.
JVMDUMP007I JVM Requesting Tool Dump using 'svmon'
JVMDUMP011I Tool Dump spawned process 254444
size inuse free pin virtual
memory 16777216 5987079 10790137 809510 1092820
pg space 131072 3630
work pers clnt other
pin 311948 0 0 497562
in use 1092820 0 4894259
PageSize PoolSize inuse pgsp pin virtual
s 4 KB - 5584935 3630 639990 690676
m 64 KB - 25134 0 10595 25134
JVMDUMP013I Processed Dump Event "user", detail "#00000000".
If you need it - you can also pass the process ID of Java to the command you're invoking. For example, svmon includes additional information if you use the -P flag. Here's how that looks:
-Xdump:tool:events=user,exec="svmon -P %pid"
Here's the output that gives:
JVMDUMP006I Processing Dump Event "user", detail "#00000000" - Please Wait.
JVMDUMP007I JVM Requesting Tool Dump using 'svmon -P 184790'
JVMDUMP011I Tool Dump spawned process 106974
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
184790 java 20067 6962 0 18353 N Y N
PageSize Inuse Pin Pgsp Virtual
s 4 KB 4979 18 0 3265
m 64 KB 943 434 0 943
Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
16002d d work fork tree m 476 0 0 476
children=4c5f7c, 0
10002 0 work fork tree m 467 434 0 467
children=7f4c40, 0
73c4e5 3 work working storage sm 2967 0 0 2967
c09383 - clnt /dev/hd2:69839 s 622 0 - -
ce139e - clnt /dev/hd2:69934 s 337 0 - -
5b44b4 f work working storage sm 211 0 0 211
c29387 - clnt /dev/hd2:69882 s 177 0 - -
c59389 - clnt /dev/hd2:69891 s 134 0 - -
c51388 - clnt /dev/hd2:69926 s 84 0 - -
c4938b - clnt /dev/hd2:69884 s 70 0 - -
c6938f - clnt /dev/hd2:69888 s 64 0 - -
b2b167 - work s 58 14 0 58
bc937b - clnt /dev/hd2:69933 s 46 0 - -
6644ce 2 work process private sm 29 4 0 29
8f931d - clnt /dev/hd2:69931 s 29 0 - -
c89393 - clnt /dev/hd2:69883 s 28 0 - -
c4138a - clnt /dev/hd2:69889 s 23 0 - -
c7938d - clnt /dev/hd2:69886 s 21 0 - -
c7138c - clnt /dev/hd2:69890 s 21 0 - -
c99391 - clnt /dev/hd2:69887 s 20 0 - -
c39385 - clnt /dev/hd2:69836 s 18 0 - -
229447 1 clnt code,/dev/hd2:69718 s 18 0 - -
c6138e - clnt /dev/hd2:69885 s 2 0 - -
c51b88 e mmap maps 2 source(s) sm 0 0 - -
JVMDUMP013I Processed Dump Event "user", detail "#00000000".
Now you have that information, you can use it to correlate with other data available inside the javacore taken at the same time.
I'm sure you can imagine lots of other ways you could use the "tool dump" support. As well as triggering OS monitoring commands such as "svmon", you could gather networking statistics via "netstat" or process information via "ps". You could even execute your own custom shell script...
Incidentally, the "tool" dump feature of the JVM is actually how Diagnostics Collector is triggered when enabled.
1.225
