Troubleshooting
Problem
This document provides information regarding the 32-bit memory address space for the J9 VM.
Resolving The Problem
The IBM Technology for Java (J9) VM is available on IBM i OS starting with V5R4. At V5R4, only the 32-bit JVM is offered. At IBM i 6.1 and later, both the 32- and 64-bit VM environments are available.
To use the J9 JVM, you must set the following environment variables.
Refer to How to determine what Java Development Kits (JDKs) are installed and use them in your environment? for more information.
o IBM Technology for Java 32 bit
ADDENVVAR JAVA_HOME '/QOpenSys/QIBM/ProdData/JavaVM/jdkXX/32bit'
o IBM Technology for Java 64 bit
ADDENVVAR JAVA_HOME '/QOpenSys/QIBM/ProdData/JavaVM/jdkXX/64bit'
Updates for the J9 VM are shipped as Service Releases. To determine what Service Release is applied to your system, you can set up your JDK environment to the specific JDK and then run the command java -version from a Qshell (QSH) or QP2TERM session and look for the srxx in the output as shown below:
New Service Releases are shipped in the Java Group PTFs, so it is very important and recommended that users stay current with their Java Group PTF levels as they move to and run with the J9 VM.
As clients consider moving to the 32-bit JVM, the key question is: "Will our application fit in the 32-bit environment?" Java heap is probably a familiar term and refers to space available for Java objects that are managed by Garbage Collection. 32-bit JVM environments can only support up to a 2GB heap space for Java and Break memory together. As such, the larger the Java heap size grows, the less available Break Memory and vice versa. In some cases, reducing your Java maximum heap size will allow Java applications that require more Break Memory to perform well. If your Java application requires more than 2GB of heap space, it is recommended to use a 64-bit JVM to avoid java.lang.OutOfMemory errors.
Default Scenario:
o Max Heap is set at 2 GB.
- We have a total of 8 segments used for Java heap (2048 / 256 = 8)
o We typically have 6 segments reserved for miscellaneous items to run the JVM.
- Kernel, Application program text, Stack data, Shared Library text, and so on.
o This leaves 2 segments (512 MB) available for break memory.
- (16 total) - (8 java heap) - (6 reserved) = 2
Lower Max heap to 1500 Scenario:
o Max Heap is set to 1500 MB
- We have a total of 6 segments used for Java heap. (1500 / 256 = 5.8)
o 6 segments reserved for miscellaneous items (same as above)
o Leaves 4 segments (1024 MB) available for break memory
Note: Changing the Max heap to 1500 MB leaves a little room in the last segment. The IBM WebSphere Application Server software application will often mmap a couple MB of space for files. If you leave a little room in that last segment, it will use that space. If you set the max heap right on a segment boundary, the IBM WAS software application will mmap from the next segment, which takes that entire segment away from being available for break memory.
How much break memory does your application need?
It is impossible for anyone to tell you what the break memory requirements for any particular environment will be, as it depends on many things. However, we do have estimates regarding the "best case scenario" of how many threads you can safely run, depending on the maximum heap size:
How to monitor the amount of Break Memory being used by your JVM
IBM i V5R4Mx ONLY
At 5.4, to watch the size of internal memory, you have to go through Advanced Analysis and use the JVMINFO -memory <taskid> option. From the display, page down to find the section on internal memory:
Refer to IBM Technology for Java (J9) Serviceability - JVMINFO and How to Gather System, Java, Heap, and GcCycle Dumps in IBM i 5.4 and Later for more information.
IBM i 6.1 and later
At 6.1, you can execute the WRKJVMJOB CL Command and then enter option 9 next to the JVM to Display Garbage Collection Information and locate the "Internal memory size" value. This displays the current amount of internal (break) memory in use.
Symptoms of running out of Break Memory
Break memory exhaustion issues can be surfaced in many forms. Sometimes the issue can be corruption, sometimes an immediate segmentation fault, sometimes something completely unpredictable. At 5.4 we fought with this issue typically surfacing as a crash of the JVM. At 6.1, a change was made where we attempt to understand if the allocation of Break memory is successful and, if not, return and OutOfMemory exception.
At 6.1, there are two common indicators as to why you are running out of native memory:
1. 0200 (function check) vlog, downstream from a call to "<SYSCALL64>:AS_SETH"
2. java/lang/OutOfMemoryError instances with the text, "unable to fork native thread"
To use the J9 JVM, you must set the following environment variables.
Refer to How to determine what Java Development Kits (JDKs) are installed and use them in your environment? for more information.
o IBM Technology for Java 32 bit
ADDENVVAR JAVA_HOME '/QOpenSys/QIBM/ProdData/JavaVM/jdkXX/32bit'
o IBM Technology for Java 64 bit
ADDENVVAR JAVA_HOME '/QOpenSys/QIBM/ProdData/JavaVM/jdkXX/64bit'
Updates for the J9 VM are shipped as Service Releases. To determine what Service Release is applied to your system, you can set up your JDK environment to the specific JDK and then run the command java -version from a Qshell (QSH) or QP2TERM session and look for the srxx in the output as shown below:
| > export JAVA_HOME=/QOpenSys/QIBM/ProdData/JavaVM/jdk80/32bit $ > java -version |
| Java(TM) SE Runtime Environment (build 8.0.8.40 - pap3280sr8fp40-20250123_01(SR8 FP40)) IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc-32-Bit 20241212_83587 (JIT enabled, AOT enabled) OpenJ9 - 32d7c6c9a17 OMR - ab5c961de IBM - dd9cff8) JCL - 20250122_01 based on Oracle jdk8u441-b07 $ |
New Service Releases are shipped in the Java Group PTFs, so it is very important and recommended that users stay current with their Java Group PTF levels as they move to and run with the J9 VM.
As clients consider moving to the 32-bit JVM, the key question is: "Will our application fit in the 32-bit environment?" Java heap is probably a familiar term and refers to space available for Java objects that are managed by Garbage Collection. 32-bit JVM environments can only support up to a 2GB heap space for Java and Break memory together. As such, the larger the Java heap size grows, the less available Break Memory and vice versa. In some cases, reducing your Java maximum heap size will allow Java applications that require more Break Memory to perform well. If your Java application requires more than 2GB of heap space, it is recommended to use a 64-bit JVM to avoid java.lang.OutOfMemory errors.
Break Memory is a new interest with the 32-bit environment and will be the focus for the rest of this document. To understand Break Memory, you need to have a good understanding of the 32-bit address space and consider how memory is used by the JVM.
What is Break Memory (also called Private Memory or Internal Memory)?
Break memory is allocated in the JVM using malloc. It has several uses, which include the following:
1. Thread Stacks
2. JIT
3. GC
4. JVM C/C++ code
5. Native Methods including customer written PASE native methods
To better understand how the memory fits together, consider the following diagram of the 32-bit address space:
What is Break Memory (also called Private Memory or Internal Memory)?
Break memory is allocated in the JVM using malloc. It has several uses, which include the following:
1. Thread Stacks
2. JIT
3. GC
4. JVM C/C++ code
5. Native Methods including customer written PASE native methods
To better understand how the memory fits together, consider the following diagram of the 32-bit address space:
- There are 16 segments of 256 MB each for a hard limit of 4 GB .
- Several segments have a predefined purpose and are not available for Java heap or Break memory.
- Once any part of a segment is used for a particular purpose, the entire segment is reserved for that purpose.
- You reserve space for the Java heap with the max heap setting (-Xmx) at JVM startup. 2 GB is the default setting for max heap.
- The more memory you take for Java heap (mmap), the less memory you have for Break memory (malloc).
Default Scenario:
o Max Heap is set at 2 GB.
- We have a total of 8 segments used for Java heap (2048 / 256 = 8)
o We typically have 6 segments reserved for miscellaneous items to run the JVM.
- Kernel, Application program text, Stack data, Shared Library text, and so on.
o This leaves 2 segments (512 MB) available for break memory.
- (16 total) - (8 java heap) - (6 reserved) = 2
Lower Max heap to 1500 Scenario:
o Max Heap is set to 1500 MB
- We have a total of 6 segments used for Java heap. (1500 / 256 = 5.8)
o 6 segments reserved for miscellaneous items (same as above)
o Leaves 4 segments (1024 MB) available for break memory
Note: Changing the Max heap to 1500 MB leaves a little room in the last segment. The IBM WebSphere Application Server software application will often mmap a couple MB of space for files. If you leave a little room in that last segment, it will use that space. If you set the max heap right on a segment boundary, the IBM WAS software application will mmap from the next segment, which takes that entire segment away from being available for break memory.
How much break memory does your application need?
It is impossible for anyone to tell you what the break memory requirements for any particular environment will be, as it depends on many things. However, we do have estimates regarding the "best case scenario" of how many threads you can safely run, depending on the maximum heap size:
|
Heap size (MB)
|
Max number of threads (APPROXIMATE)
|
|
100
|
> 5,000
|
|
1,024
|
> 5,000
|
|
2,048 (default maximum)
|
1560
|
|
3,072
|
576
|
How to monitor the amount of Break Memory being used by your JVM
IBM i V5R4Mx ONLY
At 5.4, to watch the size of internal memory, you have to go through Advanced Analysis and use the JVMINFO -memory <taskid> option. From the display, page down to find the section on internal memory:
Refer to IBM Technology for Java (J9) Serviceability - JVMINFO and How to Gather System, Java, Heap, and GcCycle Dumps in IBM i 5.4 and Later for more information.
IBM i 6.1 and later
At 6.1, you can execute the WRKJVMJOB CL Command and then enter option 9 next to the JVM to Display Garbage Collection Information and locate the "Internal memory size" value. This displays the current amount of internal (break) memory in use.
Symptoms of running out of Break Memory
Break memory exhaustion issues can be surfaced in many forms. Sometimes the issue can be corruption, sometimes an immediate segmentation fault, sometimes something completely unpredictable. At 5.4 we fought with this issue typically surfacing as a crash of the JVM. At 6.1, a change was made where we attempt to understand if the allocation of Break memory is successful and, if not, return and OutOfMemory exception.
At 6.1, there are two common indicators as to why you are running out of native memory:
1. 0200 (function check) vlog, downstream from a call to "<SYSCALL64>:AS_SETH"
2. java/lang/OutOfMemoryError instances with the text, "unable to fork native thread"
[{"Type":"MASTER","Line of Business":{"code":"LOB68","label":"Power HW"},"Business Unit":{"code":"BU070","label":"IBM Infrastructure"},"Product":{"code":"SWG60","label":"IBM i"},"ARM Category":[{"code":"a8m0z0000000CH5AAM","label":"Java Development Kit"}],"ARM Case Number":"","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"All Versions"}]
Historical Number
535123575
Was this topic helpful?
Document Information
More support for:
IBM i
Component:
Java Development Kit
Software version:
All Versions
Operating system(s):
IBM i
Document number:
634291
Modified date:
02 May 2025
UID
nas8N1012753
Manage My Notification Subscriptions