Using OMEGAMON Monitoring for JVM to Identify Out of Memory Conditions
pkeargle 270006N3R8 Visits (9872)
New in OMEGAMON for JVM on z/OS v540, a feature for diagnosing potential native memory exhaustion has been delivered. An OutofMemoryError condition can occur for several reasons, one of which is when a JVM has run out of native memory in the address space.
When the JVM starts, memory for the initial Java heap size is allocated (specified by the -Xms Java option). For a 64-bit JVM this is usually above the bar (>2Gb) memory. For a 31-bit JVM this memory is allocated from 31-bit addressable memory (the “extended region”). The JVM itself also requires native memory, and applications that make Java Native interface (JNI) calls will also need native memory. Since the JVM is written in Language Environment (LE)-conforming C/C++, native memory requests (such as malloc() and new()) will be satisfied by the LE heap manager. Not to be confused with the Java heap, LE manages C/C++ memory requests from an LE managed heap. Furthermore, some operating system services also demand memory from the extended region for LSQA (Local System Queue Area), SWA (Scheduler Work Area) and subpools 229/230. The Extended region is represented in Figure 1.
The screenshot below shows the new OMEGAMON for JVM “Native Memory Summary” workspace in the enhanced 3270 user interface. It displays memory utilization at three levels:
This scenario demonstrates how to utilize the “Native Memory Summary” workspace to diagnose OutOfMemory conditions caused when all native memory is exhausted. The “Extended Region Free %” is showing a warning condition. Only 8% of 31-bit addressable memory remains for the address space. Of the maximum extended region size of 1,513 MB the Java Runtime (JRE) occupies 1,137 MB; LE is using 54 MB, and the system areas occupy 148MB (screenshot above). In the Native Memory Category tree, we can see that the largest category is the ”VM Memory Manager (GC)”. Expanding this node shows that the Java heap is occupying the largest part of the region (screenshot below). If the JVM is not experiencing a high rate of garbage collection, then it will be prudent to reduce the size of the initial Java heap (-Xms). This will leave more space for native memory and avoid and unexpected outage.
In this scenario, we will demonstrate how to utilize the “Native Memory Summary” workspace to diagnose a Java class storage leak. In the screenshot below, the “Extended Region Free %” is showing a warning condition with 7% remaining for the address space. Examining the Native Memory Category tree, we see the ”VM Memory Manager (GC)” is the largest category. Expanding this parent and the child “Classes” category, its continual increase is the indicator that something is wrong. Therefore, more native memory will be consumed. As a result, it will lead to decrease in memory for the address space.
It is also important to examine the other values in the subpanels to identify the category of memory that is increasing. If the LE heap (Language Environment Heap Data subpanel) is increasing without either of the former increasing, then there is some application native code at the root of the problem. If the LSQA/SWA/229/230 (z/OS Extended Region Detail subpanel) is increasing, then it is an operating system service causing the problem. In result, will cause the “Extended Region Free %” to decrease. If the critical threshold is met (<5%), then the JVM may need to be shut down and restarted in an orderly manner before an OutofMemory condition occurs (screenshot below).
If free memory is diminishing over time and we see a particular JVM native memory category correspondingly increase (such as JIT cache, JNI or class memory), then that category is the cause of the leak. The “Java Native Memory” subpanel allows you to drill down to get an in-depth analysis of which parent category or sub-category may be causing issues in the JVM. This capability can assist in narrowing down the memory category in question.
If you are interested in learning more about OMEGAMON Monitoring for JVM and the other new features in v540, a great place to start is the IBM Knowledge Center (htt
OMEGAMON for JVM V5.4.0 is available as a standalone offering, or as an integral part of IBM Service Management Suite for z/OS, OMEGAMON Performance Management Suite for z/OS, and OMEGAMON for z/OS Management Suite. For more information on the scale and capability of these offerings, contact your local IBM representative or visit our Operational Excellence solutions webpage.