Resolving Java memory problems

A java.lang.OutOfMemoryError occurs when a Java™ virtual machine cannot allocate an object because it is out of memory and more memory cannot be made available by the garbage collector.

Symptoms include failure to create a thread and failure to load Java classes. This situation can be caused by, for example; exhaustion of the Java heap, exhaustion of the Java stack, exhaustion of the native stack, or the JIT compiler ran out of space. To determine the cause, you must determine why the object cannot be created. The z/OS® Connect job log records that an OutOfMemoryError occurred. Examine the job log to find the cause.

Identifying the cause

THREADS

OutOfMemoryError reports an errno 112. For example,

 java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 112 (0x70), errno2 189858397 (0xb51025d) received

It is likely that a thread limit was reached. The number of threads that can be created by the server is limited by the OMVS BPXPARM MAXTHREADS and MAXTHREADTASKS, whichever is the least. If the MAXTHREADS or MAXTHREADTASKS limit is reached, any attempt to create another thread fails and the OutOfMemoryError occurs.

MEMORY

OutOfMemoryError reports an errno 132. For example,

 java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 132 (0x84), errno2 -1055784930 (0xffffffffc112001e)“ received

Either the MEMLIMIT or HEAP sizes were reached:

MEMLIMIT - The memory available in the JVM can be controlled by setting the MEMLIMIT parameter in the z/OS Connect started task procedure. The sample BAQSTRT specifies 8G. Java documents that each Java thread created in the JVM requires at least 3 MB more for its stack storage and control blocks.

HEAP - The OutOfMemoryError reports Java heap space. For example,

 JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".  java/lang/OutOfMemoryError: Java heap space

Heap is exhausted. Increase the specified Xmx.

Monitoring Usage

THREADS

Monitor the OMVS thread usage of the z/OS Connect Server by locating the process ID for the server in its messages.log file.

******************************************************************************** 
product = WAS FOR Z/OS 19.0.0.3, z/OS Connect 03.00.22 (wlp-1.0.26.cl190320190321-1636) 
wlp.install.dir = /u/zosconnect/v3r0/wlp/ 
server.config.dir = /var/zosconnect/servers/V3Server/ 
java.home = /java/java80_bit64_sr5_fp36/J8.0_64 
java.version = 1.8.0_211 
java.runtime = Java(TM) SE Runtime Environment (8.0.5.36 - pmz6480sr5fp36-20190510_01(SR5 FP36)) 
os = z/OS (02.02.00; s390x) (en_US) 
process = 50528665@MYMACHINE 
********************************************************************************

Enter the SDSF DISPLAY OMVS command to show the status of the process:


/D OMVS,LIMITS,PID=50528665
 RESPONSE=MV01 
  BPXO051I 16.31.06 DISPLAY OMVS 699 
  OMVS 0010 ACTIVE OMVS=(P1,X3,DB,JV,00,RE,22,TG) 
  USER JOBNAME ASID PID PPID STATE START CT_SECS 
  TESTUSR BAQ30JOB 003E 50528665 33751390 HK------ 12.05.15 3601.7 
    LATCHWAITPID= 0 CMD=/java/java80_64/J8.0_64/bin/java -javaagent 
 PROCESS LIMITS: LIMMSG=NONE 
                CURRENT  HIGHWATER  PROCESS 
                USAGE    USAGE      LIMIT 
 MAXFILEPROC      220      225      65535 
 MAXFILESIZE      ---      ---    NOLIMIT 
 MAXPROCUSER        5       11        512 
 MAXQUEUEDSIGS      1        1     100000 
 MAXTHREADS       300      360       1000 
 MAXTHREADTASKS   302      360        365  See text  
 IPCSHMNSEGS        0        0       1000 
 MAXCORESIZE      ---      ---    4194304 
 MAXMEMLIMIT     908M     976M      4096M

The display shows that the thread usage reached the maximum value as set in MAXTHREADTASKS and would benefit from an increased maximum.

MEMORY

Monitor memory usage with IBM OMEGAMONIBM®OMEGAMON® for JVM on z/OS and the IBM Monitoring and Diagnostic Tools - Health Center Eclipse plug-in.

Configuration considerations

THREADS

Set the value of OMVS BPXPARM MAXTHREADS higher than the expected number of Java threads created in the z/OS Connect Server.

In the WebSphere® Liberty Profile environment, the number of Java threads is managed dynamically depending on workload so it is difficult to determine the number of threads created. To limit the number of Java threads that can be generated, configure the maximum number of default executor threads allowed in the z/OS Connect Server. In the following server.xml definition example, the maximum number of default executor threads, which run most of the workload in the server, is limited to no more than 300:

 <executor maxThreads="300" name="Default Executor"/>

Note: There are other types of threads in the JVM, so there are more than the limit of default executor threads present.

MEMLIMIT

The suggested initial setting for MEMLIMIT in the z/OS Connect Server's job is 8 GB.

Every thread running AMODE64 requires a minimum of 3 MB of above-the-bar virtual storage for each thread's stack storage. If you specify a maximum stack size of less than 1 MB, the value is rounded up so that 3 MB of storage is used. For example, for a large application with 1000 threads, the MEMLIMIT would need to be increased by 3 GB (1000 * 3 MB) for thread stack virtual storage. So, increasing the MAXTHREADS to a high value might cause excessive usage of memory higher than the bar in a heavily used server.

HEAP

The default Xmx value for a z/OS Connect Server is 512 MB.
Every thread occupies 1.6 KB of heap storage.
Every IPIC session uses 300 KB of the heap for its send and receive buffers. The default number of sessions per IPIC connection is 100. So when an IPIC connection is established, with 100 sessions, 30 MB of Java heap is allocated.
The IPIC payload data for in-flight requests is allocated from the heap.

Note:

The allocation for the Java heap is allocated out of the above the bar memory on JVM startup.
Increasing the heap reduces the available memory from the MEMLIMIT region. If the Xmx specification is increased, the MEMLIMIT specification should be increased by a similar amount.
If the problem is not related to heap exhaustion, increasing the heap can make the problem worse.

Java Health Center

Java Health Center data can be used to determine various JVM runtime characteristics including heap usage, method profiling, and network usage with a relatively low-performance impact.

The Java Health Center agent is built into the IBM Java on z/OS, so the only configuration required is to activate the agent. To run the Java Health Center in headless (unattended) mode, the following can be added to the z/OS Connect Server JVM options.

-Xhealthcenter:level=headless

The following is written to the STDERR of the server's job log on startup.

[Tue Aug 11 12:35:05 2020] com.ibm.diagnostics.healthcenter.headless INFO: 4.0.1                                                                           [Tue Aug 11 12:35:05 2020] com.ibm.diagnostics.healthcenter.headless INFO: Headless data collection has started                                            [Tue Aug 11 12:35:05 2020] com.ibm.diagnostics.healthcenter.headless INFO: Agent will keep last 5 hcd files                                                [Tue Aug 11 12:35:05 2020] com.ibm.diagnostics.healthcenter.headless INFO: Headless collection output directory is <path>

This specification causes one or more .hdc files to be written when the server is shut down, as reported in the server's job log STDERR.

[Tue Aug 11 12:40:11 2020] com.ibm.diagnostics.healthcenter.headless INFO: Creating hcd import file /u/<user_ID>/healthcenter110820_123505_16974374_1.hcd    [Tue Aug 11 12:40:12 2020] com.ibm.diagnostics.healthcenter.headless INFO: hcd import file /<path>/healthcenter110820_123505_16974374_1.hcd created

Once created, the .hcd files can be downloaded and loaded into the IBM Monitoring and Diagnostic Tools - Health Center, which can be a stand-alone Eclipse or an Eclipse plug-in. For more information, see A launch icon to indicate a link opens a new tab or window. IBM Monitoring and Diagnostic Tools - Health Center in the IBM Monitoring and Diagnostics Tools documentation.

Known issues

The problem that is described by WebSphere Liberty Profile APAR PH03409 can cause erratic thread usage, which can contribute to out of memory errors. This problem is resolved in z/OS Connect 3.0.17.0 and later.
The problem that is described by WebSphere Liberty Profile APAR PH15280 can result in ACEE control blocks not being deleted which can contribute to out of memory errors. This problem is resolved in z/OS Connect 3.0.26.0 and later.
The problem that is described by z/OS Connect APAR PH18448 can result in API requester IMS and z/OS applications reporting.
```
CEE3500S Not enough storage was available to load module
```
The number of threads that can be created by the JVM is limited by the OMVS MAXTHREADS parameter. DOC APAR PH07475 documents that for AMODE64 stack storage, there is always a minimum allocation of 3 M per thread.

zosConnect-3.0 containers java.lang.OutOfMemoryError

zosConnect-3.0 Applies to zosConnect-3.0.

Containers Applies to z/OS Connect container deployments.

z/OS Connect might report this error if the Java heap size is insufficient. This is not a failure in z/OS Connect, and you need to set the environment variables that affect a JVM running inside a container.

To resolve the issue, specify _JAVA_OPTIONS in the docker-compose.yaml file to increase the Java heap size. For example,

services:
zosConnect:
image: icr.io/zosconnect/ibm-zcon-designer:3.0.91
environment:
- CICS_USER=xxxx
- CICS_PASSWORD=xxx
- CICS_HOST=xxxx
- CICS_PORT=nnnn
- HTTP_PORT=9080
- _JAVA_OPTIONS=-verbose:gc -Xverbosegclog:/logs/gc.%Y%m%d.%H%M%S.txt -Xms1024m -Xmx1024m

This change sets the environment variable in the docker-compose.yaml file and the change is then picked up when the JVM starts in the container.

Checking the memory and CPU usage of a Docker container

zosConnect-3.0 Applies to zosConnect-3.0.

Containers Applies to z/OS Connect container deployments.

The following procedure uses Docker as an example. Other OCI-compliant container platform runtimes are supported.

Run the Docker stats command.

docker stats --no-stream containerId

Where containerId specifies the name or ID of the Docker container.

The command generates information about container memory and CPU usage. For example,

docker stats --no-stream 194e93d45332
    CONTAINER ID   NAME                  CPU %     MEM USAGE / LIMIT     MEM %     NET I/O        BLOCK I/O       PIDS
    194e93d45332   docker_zosConnect_1   6.85%     1.094GiB / 2.922GiB   37.45%    227kB / 20MB   600MB / 172MB   90

Collecting IBM z/OS Connect Designer container in Docker Memory and CPU Information

zosConnect-3.0 Applies to zosConnect-3.0.

Containers Applies to z/OS Connect container deployments.

The following command provides detailed information on the IBM z/OS Connect Designer container in Docker memory and CPU information. The command must be run with the same privileges as other Docker commands on the host system, for example as superuser or by using sudo.

To check the memory and CPU usage of the container, run the following command with Docker stats command.

docker stats --no-stream <containerId>

where containerID specifies the name of the Docker container or the container ID.

The command generates information about container memory and CPU usage.

  
    docker stats --no-stream 194e93d45332
    CONTAINER ID   NAME                  CPU %     MEM USAGE / LIMIT     MEM %     NET I/O        BLOCK I/O       PIDS
    194e93d45332   docker_zosConnect_1   6.85%     1.094GiB / 2.922GiB   37.45%    227kB / 20MB   600MB / 172MB   90

Collecting IBM z/OS Connect Designer in Docker JVM memory dumps

zosConnect-3.0 Applies to zosConnect-3.0.

Containers Applies to z/OS Connect container deployments.

The JVM memory dumps are useful for diagnosing problems at the JVM level, such as hung threads, deadlocks, excessive processing, excessive memory consumption, memory leaks, and defects in the virtual machine.

Run the following command with Docker exec. The command must be run with the same privileges as other Docker commands on the host system, for example as superuser or by using sudo.

docker exec -it <containerId> /opt/ibm/wlp/bin/server javadump defaultServer --include=thread,heap,system

Where containerID specifies the name of the Docker container or the container ID.

The command generates javacore, heapdump, core memory dumps in the container.

  
        docker exec -it 0d090ff1dd4c  /opt/ibm/wlp/bin/server javadump defaultServer --include=thread,heap,system 
        Dumping server defaultServer.
        Server defaultServer dump complete in /opt/ibm/wlp/output/defaultServer/javacore.20220208.095910.1.0002.txt.
        Server defaultServer dump complete in /opt/ibm/wlp/output/defaultServer/heapdump.20220208.095910.1.0003.phd.
        Server defaultServer dump complete in /opt/ibm/wlp/output/defaultServer/core.20220208.095911.1.0004.dmp.

You can run the following Docker command to copy the generated files to the host system.

 docker cp <containerName>:<fileToCopy> <directoryPath>

where

containerName specifies the name of the Docker container.
fileToCopy specifies the name of the generated file to copy.
directoryPath specifies the local directory to copy the file.

For example,

  
docker cp docker_zosConnect_1:/opt/ibm/wlp/output/defaultServer/javacore.20220208.095910.1.0002.txt /logsToIBM/
docker cp docker_zosConnect_1:/opt/ibm/wlp/output/defaultServer/heapdump.20220208.095910.1.0003.phd /logsToIBM/
docker cp docker_zosConnect_1:/opt/ibm/wlp/output/defaultServer/core.20220208.095911.1.0004.dmp /logsToIBM/

Note: Delete the collected memory dumps in the container after copying to the host system to avoid space issues.

Collecting (MustGather) data for Out Of Memory problems

For details on how to get MustGather information that you need to collect when you are diagnosing Out of Memory problems, see MustGather data for Out Of Memory problems.