Identifying Language Environment storage needs for JVM servers
After identifying the actual storage needs, it is possible to determine whether the supplied DFHAXRO options need to be modified or not. This allows values to be chosen that either avoid the need for incremental storage allocations, or reduce the number to an acceptable level.
About this task
The HEAP64 runtime option in DFHAXRO controls the heap size of the Language Environment® enclave for a JVM server. This option includes settings for 64-bit, 31-bit, and 24-bit storage. You can use your own program instead of DFHAXRO if preferred. The program must be specified on the JVMSERVER resource.
Procedure
- Set the RPTO(ON) and RPTS(ON) options
in DFHAXRO. These options are in comments in the supplied source of DFHAXRO. Specifying these options causes Language Environment to report on the storage options and to write a storage report showing the actual storage used.
- Disable the JVMSERVER resource. The JVM server shuts down and the Language Environment enclave is removed.
- Enable the JVMSERVER resource. CICS® uses the Language Environment runtime options in DFHAXRO to create the enclave for the JVM server. The JVM also starts up.
- Run your Java™ workloads in the JVM server to collect data about the storage that is used by the Language Environment enclave.
- Remove the RPTO(ON) and RPTS(ON) options from DFHAXRO.
- Disable the JVMSERVER resource to generate the storage
reports. The storage reports include a suggestion for the initial Language Environment enclave heap storage. The entry “Suggested initial size” in the 64-bit user heap statistics contains the suggested value and is equal to the total amount of Language Environment enclave heap storage that was used by the JVM server.
Results
The storage reports are saved in an stderr file in z/OS® UNIX, or can also go to your CICS JES output if you are using the JOBLOG or DD:// routing syntax. The directory depends on whether you have redirected output for the JVM in the JVM profile. If no redirection exists, the file is saved in the working directory for the JVM. If no value is set for WORK_DIR in the profile, the file is saved in the /tmp directory.
Use the information in the storage reports to select a suitable value for the Language Environment enclave heap storage in the DFHAXRO HEAP64 option. Storage requirements might change from one CICS execution to the next, and are typically not the same for different CICS systems that share the one DFHAXRO, thus requiring a compromise.
The normal aim is to set the HEAP64 initial allocations to the suggested sizes to avoid or reduce the number of increments. The more increments that are used, the more likely that the ratio of z/OS storage compared to actively used Language Environment storage increases. Many increments can also cause an increase in the amount of CPU time that is used by Language Environment to manage the HEAP64 storage requests. Java allocates the JVM Heap as a Memory Object via IARV64 and not through a Language Environment storage request. If a Java migration is performed with an initial allocation that includes -Xmx, it normally doubles the storage that is used for the Java Heap, and might result in MEMLIMIT being too small.
Allocating many increments might produce the effect of a Storage Leak, which manifests as a continual increase in z/OS storage over time. In practice, this is more likely to be Storage Creep, which is characterized by an increase in both z/OS allocated storage and Language Environment free storage. A Storage Leak shows a continual increase in both z/OS and Language Environment used storage. 31-bit HEAP64 storage is allocated in z/OS subpool 1 whereas JIT storage is allocated in z/OS subpool 2 in 2MB increments.
HEAPPOOLS and HEAPPOOLS64 are active in the default DFHAXRO and can be effective when configured, but the correct values are dependent on the workload and hence precise tuning might be difficult.
STACK64 should be checked to ensure that the maximum storage used is not close to the defined limit, which is typically 16 MB. Exceeding the limit will results in runtime errors.
What is not obvious from LE RPTSTG output is that, while using STACK64(1M,1M,16M) provides a safe value for JVM thread stack expansion, it can result in a large MEMLIMIT being required to avoid CICS SOS Above the Bar during GDSA expansion. With the 16M maximum, 20 MB is allocated per JVM thread in three Memory Objects - one of 16+1 MB, one of 2 MB and one of 1 MB. Only 3 MB is initially usable, and of out this 1MB is allocated for the native stack, 1MB for the LE control block and 1MB for the reserve stack, leaving 16MB as guarded stack storage and another 1MB as guarded reserve stack storage. Only the 3MB of usable allocated storage is counted towards the z/OS IARV64 MEMLIMIT check. However, CICS counts all 20 MB to decide whether it can expand the GDSA by a multiple of GB without exceeding MEMLIMIT. A single JVM server can legitimately use more than 200 threads, and 200 threads equates to 4,000MB towards the CICS MEMLIMIT check. Therefore, reducing the STACK64 maximum to a lower value that still permits some expansion can help towards reducing the MEMLIMIT size and the possibility of SOS Above the Bar.
Example
HEAPPOOLS(ALIGN,8,10,32,10,128,10,256,10,1024,10,2048,10,0,10,0,10,0,10,0,10,0,10,0,10)
HEAPPOOLS64(ALIGN,8,4000,32,2000,128,700,256,350,1024,100,2048,50,3072,50,4096,50,8192,
25,16384,10,32768,5,65536,5)
HEAP64(256M,4M,KEEP,4194304,1048576,KEEP,1024,1024,KEEP)
LIBHEAP64(3M,3M,FREE,16384,8192,FREE,8192,4096,FREE)
STACK64(1M,1M,16M)
THREADSTACK64(OFF,1M,1M,128M) STACK64 statistics: Initial size: 1M Increment size: 1M Maximum used by all concurrent threads: 1M Largest used by any thread: 1M - no change required Number of increments allocated: 0 THREADSTACK64 statistics: Initial size: 1M Increment size: 1M Maximum used by all concurrent threads: 0M Largest used by any thread: 0M - not used Number of increments allocated: 0 64bit User HEAP statistics: Initial size: 256M Increment size: 4M Total heap storage used: 730857472 Suggested initial size: 697M - use this Successful Get Heap requests: 783546 Successful Free Heap requests: 780785 Number of segments allocated: 135 - too many increments Number of segments freed: 0 31bit User HEAP statistics: Initial size: 4194304 Increment size: 1048576 Totalheap storage used (suggested initial size): 137165672 - use this Successful Get Heap requests: 1345332 Successful Free Heap requests: 1345260 Number of segments allocated: 125 - too many increments Number of segments freed: 0 64bit Library HEAP statistics: Initial size: 3M Increment size: 3M Total heap storage used: 4640032 Suggested initial size: 5M Successful Get Heap requests: 113381 Successful Free Heap requests: 112860 Number of segments allocated: 1 - low, so no change required Number of segments freed: 0 31bit Library HEAP statistics: Initial size: 16384 Increment size: 8192 Total heap storage used (suggested initial size): 520 Successful Get Heap requests: 33725 Successful Free Heap requests: 33725 Number of segments allocated: 1 - low, so no change required Number of segments freed: 0 Suggested Percentages for current CellSizes: HEAPP(ALIGN,8,1,32,1,128,1,256,1,1024,1,2048,1,0)
When reviewing RPTSTG output, remember that the HEAP64 increment sizes are for the minimum amount of storage that Language Environment allocates, and any increment could be substantially bigger than that value. Hence it is not possible to accurately determine how much z/OS storage was used when 1 or more increments have been allocated. The actual number of increments is reported for 64bit HEAP (that is, 135), for 31bit HEAP the actual number of increments is one less than is shown (that is, 124 not 125).
Because of the way that Language Environment's storage management works when increments are used, the amount of 31-bit and 64-bit z/OS storage allocated may be significantly higher than shown in RPTSTG "maximum used".
The suggested DFHAXRO changes are:
* Heap storage
DC C'HEAP64(700M,' Initial 64bit heap - change (Note 1)
DC C'4M,' 64bit Heap increment
DC C'KEEP,' 64bit Increments kept
DC C'128M,' Initial 31bit heap - change (Note 2)
DC C'2M,' 31bit Heap increment - change (Note 3)
DC C'FREE,' 31bit Increments freed - change (Note 4)
DC C'1K,' Initial 24bit heap
DC C'1K,' 24bit Heap increment
DC C'KEEP) ' 24bit Increments kept
* Heap pools
DC C'HP64(ALIGN) '
DC C'HEAPP(ALIGN,8,1,32,1,128,1,256,1,1024,1,2048,1,0) ' - change (Note 5)
* Library Heap storage
DC C'LIBHEAP64(3M,3M) ' Initial 64bit heap - do not change (Note 6)
* 64bit stack storage
DC C'STACK64(1M,1M,16M) ' - consider a change (Note 7)
- As shown by RPTSTG output 64bit "Suggested initial size" plus a small increase.
- As shown by RPTSTG output 31bit "Suggested initial size" but with a small reduction as we are using FREE.
- The 31-bit HEAP increment may be better as a value of 2M instead of 1M.
- Optionally, using 31-bit HEAP FREE may result in less z/OS storage being allocated to map the "Total heap storage used" than with KEEP.
- As recommended by RPTSTG output after the HEAPPOOLS statistics, but may benefit from further optimization. The default of 10% of the 31-bit Heap initial size of 128MB is likely to result in an excessive amount of storage being allocated. A minimum of 6 pools each of 10% of the initial heap size of 128MB causes 77MB to be allocated. This will be included in the "Total heap storage used" value (because the HEAPPOOLS storage extents are allocated there), irrespective of what percentage of the pool s is productively used. Using HEAPPOOLS cell sizes greater than 256 bytes might result in inefficient use of Language Environment HEAP storage.
- Only one increment was required, which is not a problem.
- The largest used was 1MB. Reducing the maximum of 16M to a value such as 8 MB or even lower would significantly reduce the amount of STACK64 storage that CICS counts towards MEMLIMIT when checking to see whether it can allocate a new GDSA extent. STACK64 changes should be tested thoroughly before migrating them into a production environment.
This is an example of using 31-bit HEAP FREE on another run of the same JVM server. The "Number of segments" shows the number of GETMAINs and FREEMAINs performed, which was low for the time that the JVM server was active. The difference of 2 shows that the enclave terminated with only the initial allocation plus one increment, which is likely to be less than the "Total heap storage" and shows the effectiveness of FREE. "Total heap storage used" was higher, but any total often changes from one run of a JVM server to another, hence basing changes on only one set of RPTSTG may not provide the best possible settings.
31bit User HEAP statistics: Initial size: 134217728 Increment size: 2097152 Total heap storage used (suggested initial size): 154056664 Successful Get Heap requests: 3253239 Successful Free Heap requests: 3253176 Number of segments allocated: 149 Number of segments freed: 147
It is important to read the Language Environment Debugging Guide in order to correctly interpret RPTSTG output.