IBM Java on Linux: Avoid using gcore
kgibm 0600027VAP Comments (2) Visits (10716)
Previously, I recommended using a script to add a timestamp to a core file name when using gcore so that MAT could know when the dump was taken. Now, I'll make the case for a stronger recommendation to completely avoid using gcore and use IBM Java's built-in capabilities of taking a core dump instead.
Gcore is simply a shell script available on some flavors of Linux:
$ whereis gcore gcore: /usr/bin/gcore $ cat /usr/bin/gcore #!/bin/sh ... gdb -x $tmpfile -batch ... attach $pid gcore $name.$pid detach
Gcore attaches to the process using gdb and runs the gdb gcore command and then detaches. The gcore command manually iterates through the memory sections available through the debugger and writes the ELF core file.
Why is gcore not as good as IBM Java's capabilities? As we found in part 3, even with 0xffff in /pro
When IBM Java creates a core file, it forks itself and kills the child process, thus using the kernel to produce the core unlike gdb:
Linux does not provide an operating system API for generating a system dump from a running process. The JVM produces system dumps on Linux by using the fork() API to start an identical process to the parent JVM process. The JVM then generates a SIGSEGV signal in the child process. The SIGSEGV signal causes Linux to create a system dump for the child process. The parent JVM processes and renames the system dump, as required, by the -Xdump options, and might add additional data into the dump file. The system dump for the child process contains an exact copy of the memory areas used in the parent. The SDK dump viewer can obtain information about the Java threads, classes, and heap from the system dump. However, the dump viewer, and other system dump debuggers show only the single native thread that was running in the child process.
IBM Java then looks at /proc/PID/maps and appends information to the core dump that wouldn't otherwise be there:
The Linux operating system core dump might not contain all the information included in a core dump produced by the JVM dump agents.
Some of these extra sections may not actually be readable, so they won't be in the core dump; however, the ELF file will have LOAD sections for them, so we can actually get an accurate view of the total virtual size of the process with a system dump produced by IBM Java using gdb info files:
$ sudo gdb --batch --command gdbinfofiles.py java core
Note that you may receive a BFD truncation warning (see example above) because of the fact that the ELF LOAD sections exist but don't have any backing data.
Therefore, if IBM Java produces the system dump instead of the kernel or gdb/gcore, then you can get an accurate calculation of the total virtual size of the process using gdb info files or the Interactive Diagnostic Data Explorer (IDDE) command !info mmap. Otherwise, you should gather /proc/PID/smaps at the time of the dump.
Here are some methods to create a core dump using the IBM Java method:
If you cannot use these mechanisms for some reason (or if the JVM is completely hung and they do not work), then gcore or kill can be used, but please also gather the output of 'cat /proc/PID/smaps' right before or after you get the core and save it to a file. I've added this automation to the ibmgcore.sh script which wraps gcore.