Question & Answer
Question
IBM Java for Linux How To: Reconciling Linux "top" output with NATIVEMEMINFO section of javacore.txt
Answer
This document provides instructions for reconciling memory usage measured with the Linux "Top" tool (or other memory monitoring tools) and the NATIVEMEMINFO section of Java thread dumps (javacore.txt).
Examining process size with Linux "top"
The "top" command shows the amount of virtual memory used by the process. It includes all code, data and shared libraries plus pages that have been swapped out. This is shown in the VIRT column in KBYTES. Use 'top -p
# top -p 23812
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23812 user1 20 0 3097036 994056 13892 S 0.0 25 .0 0:02.54 java
Step B
Process size in NATIVEMEMINFO section of Java thread dump
The NATIVEMEMINFO section of the javacore.txt (thread dump) provides information about the native memory allocated by the Java runtime environment. A thread dump (javacore.txt) can be created using the 'kill -3
NULL ------------------------------------------------------------------------
0SECTION NATIVEMEMINFO subcomponent dump routine
NULL =================================
0MEMUSER
1MEMUSER JRE: 2,421,451,056 bytes / 1471 allocations
1MEMUSER |
2MEMUSER +--VM: 2,408,431,592 bytes / 985 allocations
2MEMUSER | |
3MEMUSER | +--Classes: 3,491,440 bytes / 126 allocations
2MEMUSER | |
3MEMUSER | +--Memory Manager (GC): 2,191,740,752 bytes / 150 allocations
3MEMUSER | | |
4MEMUSER | | +--Java Heap: 2,147,483,648 bytes / 1 allocation
3MEMUSER | | |
4MEMUSER | | +--Other: 44,257,104 bytes / 149 allocations
2MEMUSER | |
3MEMUSER | +--Threads: 3,563,024 bytes / 93 allocations
3MEMUSER | | |
4MEMUSER | | +--Java Stack: 127,688 bytes / 9 allocations
3MEMUSER | | |
4MEMUSER | | +--Native Stack: 3,276,800 bytes / 10 allocations
3MEMUSER | | |
4MEMUSER | | +--Other: 158,536 bytes / 74 allocations
2MEMUSER | |
3MEMUSER | +--Trace: 110,048 bytes / 230 allocations
2MEMUSER | |
3MEMUSER | +--JVMTI: 17,784 bytes / 13 allocations
2MEMUSER | |
3MEMUSER | +--JNI: 32,568 bytes / 58 allocations
2MEMUSER | |
3MEMUSER | +--Port Library: 208,601,104 bytes / 70 allocations
3MEMUSER | | |
4MEMUSER | | +--Unused <32bit allocation regions: 208,590,696 bytes / 1 allocation
3MEMUSER | | |
4MEMUSER | | +--Other: 10,408 bytes / 69 allocations
2MEMUSER | |
3MEMUSER | +--Other: 874,872 bytes / 245 allocations
1MEMUSER |
2MEMUSER +--JIT: 11,786,320 bytes / 146 allocations
2MEMUSER | |
3MEMUSER | +--JIT Code Cache: 8,388,608 bytes / 4 allocations
2MEMUSER | |
3MEMUSER | +--JIT Data Cache: 2,097,216 bytes / 1 allocation
2MEMUSER | |
3MEMUSER | +--Other: 1,300,496 bytes / 141 allocations
1MEMUSER |
2MEMUSER +--Class Libraries: 1,233,144 bytes / 340 allocations
2MEMUSER | |
3MEMUSER | +--Harmony Class Libraries: 2,000 bytes / 1 allocation
2MEMUSER | |
3MEMUSER | +--VM Class Libraries: 1,231,144 bytes / 339 allocations
3MEMUSER | | |
4MEMUSER | | +--sun.misc.Unsafe: 528 bytes / 6 allocations
4MEMUSER | | | |
5MEMUSER | | | +--Direct Byte Buffers: 528 bytes / 6 allocations
3MEMUSER | | |
4MEMUSER | | +--Other: 1,230,616 bytes / 333 allocations
NULL
JVM's real memory usage and MALLOC_ARENA_MAX
When the virtual size of a Java process measured using "top", exceeds the value reported in the NATIVEMEMINFO, the value of MALLOC_ARENA_MAX should be examined. To avoid locking situations, MALLOC_ARENA_MAX, a Linux environment variable, has been introduced to control dynamic memory allocation for thread memory pools. For applications with large numbers of threads (i.e. Java), this may inflate the amount of native memory used.
# top -p 23812
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23812 user1 20 0 3097036 0.981g 13864 S 0.0 25.8 0:02.50 java
Reconciling "top" and NATIVEMEMINFO
Although dynamic memory allocation allocates extra memory on a per thread basis for performance reasons, you can reconcile the "top" output with NATIVEMEMINFO by setting MALLOC_ARENA_MAX to 1. Please note, changing the MALLOC_ARENA_MAX to other than the default (which depends on the number of cores in the machine), is not recommended as it can degrade peformance.
To view the process size without memory pools set MALLOC_ARENA_MAX to 1. For example,
# export MALLOC_ARENA_MAX=1
# java -Xss1024k -Xms2048m -Xmx2048m GenerateOOM
# top -p 23812
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23812 carver 20 0 2448980 994056 13892 S 0.0 25 .0 0:02.54 java
Note: The total native memory usage reported by NATIVEMEMINFO is always slightly less than the total native address space, becasue it does not include overhead introduced by the operating system.
Closer examination of the address space with pmap
If you wish to examine the process size breakdown more closely, you can use 'pmap -x'
23841: java -Xms2048m -Xmx2048m GenerateOOM
Address Kbytes RSS Dirty Mode Mapping
0000000000010000 4 4 4 rw--- [ anon ]
0000000000400000 4 4 0 r-x-- java
0000000000500000 4 4 4 rw--- java
0000000001009000 264 264 264 rw--- [ anon ]
000000000104b000 20 0 0 ----- [ anon ]
0000000001050000 2097088 196444 196444 rw--- [ anon ] <--------- Java Heap
0000000081040000 44 0 0 ----- [ anon ]
000000008104b000 51200 984 984 rw--- [ anon ]
000000008424b000 153600 0 0 ----- [ anon ]
00007f2f08200000 8192 8192 8192 rwx-- [ anon ]
00007f2f0d22a000 88 88 0 r-x-- libgcc_s.so.1 <------- Shared library mapping
00007f2f0d240000 2044 0 0 ----- libgcc_s.so.1
00007f2f0d43f000 4 4 4 rw--- libgcc_s.so.1
00007f2f0d455000 4 0 0 ----- [ anon ]
....................................
....................................
00007f2f10df7000 496 484 0 r-x-- libjclse7b_26.so
00007f2f10e73000 1024 0 0 ----- libjclse7b_26.so
00007f2f10f73000 16 16 16 rw--- libjclse7b_26.so
....................................
....................................
00007f2f15559000 16 4 4 rw--- [ anon ]
00007f2f1555d000 140 140 0 r-x-- ld-2.19.so
00007f2f1558f000 140 140 140 rw--- [ anon ]
00007f2f155b2000 4 0 0 ----- [ anon ]
00007f2f155b3000 624 192 192 rw--- [ anon ]
....................................
....................................
00007f2f1577f000 4 4 4 r---- ld-2.19.so
00007f2f15780000 4 4 4 rw--- ld-2.19.so
00007f2f15781000 4 4 4 rw--- [ anon ]
00007ffd47f43000 132 32 32 rw--- [ stack ]
00007ffd47fbb000 8 0 0 r---- [ anon ]
00007ffd47fbd000 8 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]
---------------- ------- ------- -------
total kB 2446960 228984 214968
Contact IBM Support
If, after reading and following the above instructions, further assistance is required, please complete the following steps:
1. Confirm that you have review and completed all of the above steps.
2. Contact IBM and open a new IBM service request (i.e., a new IBM PMR).
3. Collect and upload data as per the data collection procedures noted in the above sections or package and upload the current data and details by following the instructions on this web page:
IBM Java for AIX MustGather: How to upload diagnostic data and testcases to IBM
| Document Type: | Technical Document |
| Content Type: | General |
| Hardware: | all Power |
| Operating System: | all Linux Versions |
| IBM Java: | all Java Versions |
| Author(s): | John Carver |
| Reviewer(s): | Vidya Makineedi |
Was this topic helpful?
Document Information
More support for:
IBM Java
Software version:
Version Independent
Operating system(s):
Linux
Document number:
632035
Modified date:
17 June 2018
UID
isg3T1025759