In what will probably become a 10 part series before we figure things out, I wanted to share more information I've learned about Linux core dumps. Part 1 successfully found how much was malloc'ed in a core, and part 2 failed to find the total virtual memory usage in a core. A colleague recently pointed me to more information in the most obvious of places: 'man core' !
Since kernel 2.6.23, the Linux-specific /proc/PID/coredump_filter file can be used to control which memory segments are written to the core dump file in the event that a core dump is performed for the process with the corresponding process ID. The value in the file is a bit mask of memory mapping types (see mmap(2)).
When a process is dumped, all anonymous memory is written to a core file as long as the size of the core file isn't limited. But sometimes we don't want to dump some memory segments, for example, huge shared memory. Conversely, sometimes we want to save file-backed memory segments into a core file, not only the individual files. /proc/PID/coredump_filter allows you to customize which memory segments will be dumped when the PID process is dumped. coredump_filter is a bitmask of memory types. If a bit of the bitmask is set, memory segments of the corresponding memory type are dumped, otherwise they are not dumped. The following 7 memory types are supported:
- (bit 0) anonymous private memory
- (bit 1) anonymous shared memory
- (bit 2) file-backed private memory
- (bit 3) file-backed shared memory
- (bit 4) ELF header pages in file-backed private memory areas (it is effective only if the bit 2 is cleared)
- (bit 5) hugetlb private memory
- (bit 6) hugetlb shared memory
Note that MMIO pages such as frame buffer are never dumped and vDSO pages are always dumped regardless of the bitmask status. When a new process is created, the process inherits the bitmask status from its parent. It is useful to set up coredump_filter before the program runs. For example: $ echo 0x7 > /proc/self/coredump_filter
Okay, let's set all the bits and see what we get!
$ uname -a Linux oc2613817758.ibm.com 2.6.32-358.2.1.el6.x86_64 #1 SMP Wed Feb 20 12:17:37 EST 2013 x86_64 x86_64 x86_64 GNU/Linux $ echo 0x7f > /proc/self/coredump_filter $ cat /proc/self/coredump_filter 0000007f # ... run program, get core, etc. ... $ readelf --program-headers core.16787
Running gdbmemory.py as in part 2 still doesn't give the same sum that ps or /proc reports, but that last readelf command was also recommended by my colleague and it's much more interesting:
Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align NOTE 0x00000000000003f8 0x0000000000000000 0x0000000000000000 0x00000000000008ac 0x0000000000000000 R 1 LOAD 0x0000000000000ca4 0x0000000000400000 0x0000000000000000 0x0000000000001000 0x0000000000001000 R E 1 LOAD 0x0000000000001ca4 0x0000000000600000 0x0000000000000000 0x0000000000001000 0x0000000000001000 RW 1 LOAD 0x0000000000002ca4 0x0000000001a9c000 0x0000000000000000 0x0000000000065000 0x0000000000065000 RW 1 LOAD 0x0000000000067ca4 0x0000003e9be00000 0x0000000000000000 0x0000000000020000 0x0000000000020000 R E 1 LOAD 0x0000000000087ca4 0x0000003e9c01f000 0x0000000000000000 0x0000000000001000 0x0000000000001000 R 1 LOAD 0x0000000000088ca4 0x0000003e9c020000 0x0000000000000000 0x0000000000001000 0x0000000000001000 RW 1 LOAD 0x0000000000089ca4 0x0000003e9c021000 0x0000000000000000 0x0000000000001000 0x0000000000001000 RW 1 LOAD 0x000000000008aca4 0x0000003e9c200000 0x0000000000000000 0x000000000018a000 0x000000000018a000 R E 1 LOAD 0x0000000000214ca4 0x0000003e9c589000 0x0000000000000000 0x0000000000004000 0x0000000000004000 R 1 LOAD 0x0000000000218ca4 0x0000003e9c58d000 0x0000000000000000 0x0000000000001000 0x0000000000001000 RW 1 LOAD 0x0000000000219ca4 0x0000003e9c58e000 0x0000000000000000 0x0000000000005000 0x0000000000005000 RW 1 LOAD 0x000000000021eca4 0x00007f3863ab9000 0x0000000000000000 0x0000000000003000 0x0000000000003000 RW 1 LOAD 0x0000000000221ca4 0x00007f3863ad7000 0x0000000000000000 0x0000000000002000 0x0000000000002000 RW 1 LOAD 0x0000000000223ca4 0x00007fffacb72000 0x0000000000000000 0x0000000000015000 0x0000000000015000 RW 1 LOAD 0x0000000000238ca4 0x00007fffacbff000 0x0000000000000000 0x0000000000001000 0x0000000000001000 R E 1 LOAD 0x0000000000239ca4 0xffffffffff600000 0x0000000000000000 0x0000000000001000 0x0000000000001000 R E 1
This happens to line up pretty closely with maps/smaps/pmap output:
Address Kbytes RSS Dirty Mode Mapping 0000000000400000 4 4 0 r-x-- a.out 0000000000600000 4 4 4 rw--- a.out 0000000001a9c000 404 228 228 rw--- [ anon ] 0000003e9be00000 128 104 0 r-x-- ld-2.12.so 0000003e9c01f000 4 4 4 r---- ld-2.12.so 0000003e9c020000 4 4 4 rw--- ld-2.12.so 0000003e9c021000 4 4 4 rw--- [ anon ] 0000003e9c200000 1576 272 0 r-x-- libc-2.12.so 0000003e9c38a000 2044 0 0 ----- libc-2.12.so 0000003e9c589000 16 12 8 r---- libc-2.12.so 0000003e9c58d000 4 4 4 rw--- libc-2.12.so 0000003e9c58e000 20 12 12 rw--- [ anon ] 00007f3863ab9000 12 12 12 rw--- [ anon ] 00007f3863ad7000 8 8 8 rw--- [ anon ] 00007fffacb72000 84 8 8 rw--- [ stack ] 00007fffacbff000 4 4 0 r-x-- [ anon ] ffffffffff600000 4 0 0 r-x-- [ anon ] ---------------- ------ ------ ------ total kB 4324 684 2966
And now it becomes much easier to see what's missing looking at the LOAD sections. The VirtAddr section lines up with the Address column in pmap, and the MemSiz value lines up with Kbytes in pmap. Matching the two, we can see that what's missing in the core is one of the ld-2.12.so lines at address 0x0000003e9c38a000. (Update May 14th: Notice that the missing section doesn't even have a read flag in the Mode column, which probably explains why it wasn't dumped.) This is actually the same information as "info files" in gdb if you just look at the "Local core dump file" section:
Local core dump file: `core.16787', file type elf64-x86-64. 0x0000000000400000 - 0x0000000000401000 is load1 0x0000000000600000 - 0x0000000000601000 is load2 0x0000000001a9c000 - 0x0000000001b01000 is load3 0x0000003e9be00000 - 0x0000003e9be20000 is load4 0x0000003e9c01f000 - 0x0000003e9c020000 is load5 0x0000003e9c020000 - 0x0000003e9c021000 is load6 0x0000003e9c021000 - 0x0000003e9c022000 is load7 0x0000003e9c200000 - 0x0000003e9c38a000 is load8 0x0000003e9c589000 - 0x0000003e9c58d000 is load9 0x0000003e9c58d000 - 0x0000003e9c58e000 is load10 0x0000003e9c58e000 - 0x0000003e9c593000 is load11 0x00007f3863ab9000 - 0x00007f3863abc000 is load12 0x00007f3863ad7000 - 0x00007f3863ad9000 is load13 0x00007fffacb72000 - 0x00007fffacb87000 is load14 0x00007fffacbff000 - 0x00007fffacc00000 is load15 0xffffffffff600000 - 0xffffffffff601000 is load16
Therefore, the "Local core dump file" stanza of gdb "info files" seems like the best place to look to approximate the virtual address size of the process at the time of the dump. This will not account for everything, especially if coredump_filter is the default value. Here is a simplified gdbinfofiles.py which just adds up the load sections of the core file from info files. In fact, you don't even need to pass the executable (e.g. java) since GDB is essentially just looking at the ELF structure of the core file.
# gdb --batch --command gdbinfofiles.py exe core import sys import re import traceback import struct verbose = False #verbose = True output = gdb.execute("info files", False, True) lines = output.split('\n') addrRE = re.compile(r"\s+0x([\da-fA-F]+) - 0x([\da-fA-F]+) is .*") sum = 0 adding = 0 for line in lines: if line.find("Local core dump file") != -1: adding = 1 elif line.find("Local") != -1: adding = 0 if adding == 1: m = addrRE.match(line) if m != None: start = int(m.group(1), 16) end = int(m.group(2), 16) if verbose: print "Start=" + str(start) + ",End=" + str(end) diff = end - start sum += diff print "Sum memory ranges = " + str(sum)