Threads, Java heap, MEMLIMIT, REGION
To ensure your IBM z/OS Connect server performs optimally, you need to carefully configure the Java™ heap, MEMLIMIT, and REGION values, and continually monitor your server while your workload is running.
- How to measure your workload
- Native threads
- Java threads: Overview
- Java threads: Executor threads
- Calculating and monitoring the Java heap size
- Pause-less garbage collection
- Calculating the MEMLIMIT
- Setting the REGION size
- Working example
How to measure your workload
Before you can calculate your Java heap size and MEMLIMIT, you need to understand your workload, and how to obtain performance measurements.
Some workloads are steady and consistent, whereas others might have peaks and troughs throughout a period.
When you prepare your test workload, have realistic targets, such as the minimum number of transactions per second (TPS), or an average response time per request. You should include a think time in your workload driver to simulate a realistic incoming workload. The think time value represents the delay between a simulated client receiving a response from IBM z/OS Connect and sending it the next request. A large think time reduces the transaction rate, whereas a small think time increases the transaction rate. Do not use a think time of zero as this is not only unrealistic, but risks overwhelming the IBM z/OS Connect server causing threads to queue and potentially time out.
- Run a steady and consistent workload. If the workload is not steady and consistent, measure the workload at its peak to ensure sufficient resources are available.
- Ensure that other workloads are not running on the LPAR when you take your performance measurement.
- Use CPUs that are dedicated to the LPAR where IBM z/OS Connect is running.
- Use a short SMF interval, for example, 1 minute, or 5 minutes, to minimize variation in the results.
Warm upthe IBM z/OS Connect server.
Warming upthe IBM z/OS Connect server
- Your IBM z/OS Connect server runs in a JVM. A running JVM must perform several critical functions. The most significant of these functions are just-in-time (JIT) compilation and garbage collection (GC). JIT and GC can use significant amounts of CPU in the JVM. Therefore, before you take any performance measurements, it is important to warm up the JIT so it is fully optimized by running several hundred thousand requests through your IBM z/OS Connect server. The first time a transaction is run in Java, the z/Architecture® instructions that are produced by the JIT compiler are at a low optimization level, which results in a relatively high CPU cost to run the Java methods. As more transactions are run, the Java method invocation counts are increased. Therefore, the JIT re-compiles a Java method to a more aggressive level of optimization. This greater level of optimization results in a Java method that requires less CPU to run than before the recompilation took place. As a result, the CPU that is required to run the transaction reduces. This process is repeated several times during the lifetime of the JVM. The warm-up period completes at the point at which the CPU cost per transaction ceases to show any improvements. After the workload is running in a steady-state for the warm-up period, it is assumed that the JIT compiler will not optimize the IBM z/OS Connect server further, and CPU measurements can be taken.
- Shutting down a IBM z/OS Connect server (and thus the JVM it runs in) discards the JIT-compiled native code. Therefore, the iterative process of optimization begins again when the JVM is restarted.
- When your workload is running and your IBM z/OS Connect server is
warmed up, you can start monitoring thread usage, the Java heap, and native storage usage.
IBM z/OS Connect uses native threads and Java threads to process requests. For each Java thread, the JVM asks the operating system to create a native thread in the LE native stack storage that the operating system controls.
- Native threads
- Each native thread requires a minimum of 3M of LE native stack storage (1M for the JVM and 2M
for LE). This storage is allocated above-the-bar. Therefore, the value set for MEMLIMIT must allow
enough storage for these threads. The supplied JCL for IBM z/OS Connect sets this value to 8G. For more
information, see Calculating the MEMLIMIT.
To observe the number of native threads for the IBM z/OS Connect address space, you can issue MVS commands from SDSF. You need the process ID (PID) number for the IBM z/OS Connect server. You can find the PID from the messages.log file, or by issuing the following MVS command from SDSF: /D OMVS,ASID=<hexadecimal address space id for your server>.
For example, /D OMVS,ASID=B4.
When you have the PID number, you can issue the following MVS command from SDSF: /D OMVS,LIMITS,PID=<server pid>. This command displays the current usage and high water usage values for MAXTHREADS and MAXTHREADTASKS, and the PROCESS LIMIT for your IBM z/OS Connect server. The limit for MAXTHREADS must be greater than or equal to MAXTHREADTASKS to ensure that enough threads are available to handle the tasks currently being processed, as shown in the following output.
- Java threads: Overview
- Java threads are used for many different purposes, such as
JIT compilation, GC, and handling IBM z/OS Connect
requests. Most Java threads that are used for handling IBM z/OS Connect workloads are executor threads.
Each Java thread requires a minimum of 1.6K of Java heap storage. You can use a z/OS monitor, such as the Java Health Center, to show the number of active Java threads that are used by IBM z/OS Connect .
- Java threads: Executor threads
- Executor threads are Java threads created by
Liberty and used to handle IBM z/OS Connect requests.
You can limit the requests that come into IBM z/OS Connect by configuring the number of executor
threads in server.xml. For most users, setting the number of executor threads
is not required, but there may be times when a user wants to prevent too many requests coming into
the server at any one time. You can see how many executor threads are active in the Java Health Center and the state of each. For example,
RUNNABLE, etc. On the Threads tab, filter on
Exec*. This filter uses case-sensitive values.
When all executor threads are busy, incoming requests are queued. The request will only timeout if an associated timeout value is exceeded, for example, asyncRequestTimout, or waiting for a socket, session, or connection.As each executor thread requires a minimum of 1.6 KB of Java heap, you can avoid your IBM z/OS Connect server running out of memory (that is, reaching the MEMLIMIT) by setting maxThreads on the executor element in server.xml. For example,
By default, maxThreads is unlimited so consider adding this element to your server.xml and setting a value by monitoring your incoming workload.
Calculating and monitoring the Java heap size
The Java heap is a runtime data area from which memory is allocated for all class instances and arrays. IBM z/OS Connect loads all the objects it needs into the Java heap, including features, APIs, services, API requesters, policies, and security certificates. The Java heap is managed in 64-bit storage by the JVM and uses some of the storage allocated by the MEMLIMIT setting.
Every workload that runs through a IBM z/OS Connect server is different, so it is very important to understand the peaks and troughs of your workload over time, and continually monitor the Java heap to ensure optimum performance.
When a IBM z/OS Connect server starts up, it loads all the required objects into the Java heap. When the server is running normally, further Java objects are required to manage the incoming requests, connections, and session pools. The Java heap memory is also used for parsing and transforming API requests and responses. This heap is application-specific and the amount required for transformation depends on the size and complexity of the payload. This payload might include nested arrays, long field names, and so on. For more information, see How payload size and transformation affect performance for the API provider.
As requests start and finish at various times, it can be difficult to establish how much heap is required at any one time. Adjustments to the Java heap size need to be made for inconsistent workloads where peaks and troughs might occur throughout the day. Further allowances need to be made for requests that take longer than anticipated due to delays caused by other products in the workflow. It is important not to overwhelm the IBM z/OS Connect server with work and to configure enough heap for these situations.
- Workloads running slower, possibly due to a badly designed API, or contention for connections.
- API callers unexpectedly sending larger than normal request payloads.
- API requester endpoints unexpectedly sending larger than normal response payloads.
- Delays in SORs (CICS®, IMS, Db2®) while more requests are coming into the IBM z/OS Connect server.
- Trace enabled to diagnose a problem.
Therefore, it is important to constantly monitor your Java heap to ensure that there are enough resources to allow for such circumstances.
- Maximum heap size (-Xmx)
- The maximum heap size is set by the JVM property
-Xmx. By default, the IBM z/OS Connect server has a maximum Java heap size of 512M. However, this value might not be large enough for your expected workloads. Java properties cannot be changed for an active server, so unless you use the default -Xmx, you must set this before starting your server.
Calculating the amount of heap your server requires is not straightforward and might require running several iterations of your workload to obtain the optimum heap size.As a rough guide, when you set your maximum heap size you should allow a minimum of the following values:
- 60M for a IBM z/OS Connect server to start.
- An extra 8M for every 100 archive files that are to be loaded, either API, service, or API requester. For example, if you have 200 APIs and 200 services, add 32M. Do not waste heap space by installing archive files that are no longer required, such as old versions.
- For a single connection to CICS over IPIC, allow 300K per
- The number of sessions on an IPIC connection between IBM z/OS Connect and a CICS region is negotiated during the establishment of the connection. The lower of the two values, sendSessions on the zosconnect_cicsIpicConnection definition in server.xml and Receivecount on the IPCONN CICS definition (if configured), is used. The default value is 100, taken from the default sendSessions value. For more information about configuring your IPIC connection, see IP interconnectivity (IPIC) overview.
- As soon as the connection is established, heap storage is allocated for all the sessions. For example, if the negotiated number of sessions is 100, then 100 sessions would require 30M (100 x 300K per session).
- Use CICS statistics to monitor the usage of each IPIC connection while running a workload and adjust the value of sendSessions accordingly to avoid wasting Java heap.
- For each connection to IMS, allow 1.6K.
- Each IMS service request obtains a connection from the resource adapter connection pool. The default value for maxPoolSize on the connection factory element is 50. This value is the maximum number of physical connections for a pool and is usually sufficient for most installations. Therefore, if a workload requires 50 simultaneous requests to IMS, then 80K of heap storage would be required at that point in time, and also while the connections are held in the connection pool. For more information, see Connections from IBM z/OS Connect to SORs for API provider.
- To aid performance, unused connections remain in the pool ready to be reused, and are only removed when agedTimeout expires. The agedTimeout attribute is set on the connectionManager element for a connection factory. The default is -1, which means that there is no timeout and the connections remain in the pool.
- Allow room for your Java heap to grow. You can calculate how much extra heap is required only by monitoring your heap over a long period (hours or days) as this very much depends on the type and speed of your workload.
As a suggestion, start with the default of 512M and then use a Java monitor such as Omegamon for JVM or the Java Health Center to see how much heap is being used. The Working example describes a suggested approach.If the maximum heap size set for your IBM z/OS Connect server is too small, the JVM might be unable to reclaim enough storage during a garbage collection cycle. This causes the JVM and IBM z/OS Connect to terminate, typically with one of the following errors:
errno 112: suggests a thread limit was reached.
errno 132: suggests MEMLIMIT or maximum heap size was reached.
- An abend, SIGSEGV, or General Protection Fault
However, if the maximum heap size is set too large, your IBM z/OS Connect might exhibit inconsistent performance. A large heap can increase the length of time required for the GC to pause the processing of requests while it manages the objects in memory by freeing up unused memory objects and compacting areas of the heap to reduce wasted space. For workloads that perform consistently with few peaks and troughs, it is easier to tune your Java heap, but for erratic workloads it is safer to specify enough memory to cater for the workload peaks and possibly compromise performance.
- Initial heap size (-Xms)
- In addition to setting the maximum heap size (-Xmx), you can optionally set
the initial heap size (-Xms). For workloads that run through IBM z/OS Connect , the main priority is to avoid long GC
pauses. Therefore, the default setting of 8M is used to allow the GC to run frequently and
efficiently with minimal GC pauses.
Setting the minimum and maximum heap size to the same value is typically not a good idea for IBM z/OS Connect workloads because garbage collection is delayed until the heap is full and then a long GC pause occurs, which might impact response times. This configuration also uses a large amount of RAM, which leaves less RAM available for other applications.
- GC policy
- Java offers various GC policies that can be set for your
IBM z/OS Connect server. For most users, the default
gencon, tends to provide the best and consistent performance for IBM z/OS Connect .When you use the
genconGC policy, the Java heap uses new and old (nursery and tenure) areas within the Java heap to provide smoother GCs. The garbage collection algorithms are very good at automatically tuning the sizes of different areas in the Java heap, so typically you do not need to set Xmns, Xmnx, Xmos, or Xmox. In fact, poor setting of these parameters can have a detrimental impact to your server's performance. For most users, the following default settings of the
genconGC policy are sufficient.
- -Xmns - initial size of new (nursery) area is 25% of
- -Xmnx - maximum size of new (nursery) area is 25% of
- -Xmos - initial size of old (tenure) area is 75% of
- -Xmox - maximum size of old (tenure) area is approximately
- -Xmns - initial size of new (nursery) area is 25% of
- Monitoring the Java heap
- To observe the Java heap values used by your IBM z/OS Connect server at initialization, specify the
-verbose:sizes JVM option for your server. For more information, see Specifying JVM options. The STDERR for your server shows the
heap values, as shown in the following example where -Xmx was set to 512M, and
-Xms set to 64M.
- The Java heap used by your IBM z/OS Connect server can be monitored by using tools
such as IBM Omegamon for JVM or the Java Health Centre. The latter is one of the IBM Monitoring and Diagnostic Tools.The Garbage Collector (GC) adapts the heap size to keep occupancy between 40% and 70% for the following reasons:
- A heap occupancy greater than 70% causes more frequent GC cycles, which can reduce performance.
- A heap occupancy less than 40% means infrequent GC cycles. However, these cycles are longer than necessary, causing longer pause times, which can reduce performance.
- Therefore, to optimize application performance and keep within the 40-70% range, the maximum
heap size setting should be at least 43% larger than the maximum occupancy of the IBM z/OS Connect server. For example, if your IBM z/OS Connect server has a maximum occupancy of 600M,
the maximum heap size is calculated as follows:600 + (600 * 43/100) = 858M
If the heap is fully expanded and the occupancy level is greater than 70%, increase the -Xmx value so that the heap is not more than 70% occupied. Accordingly, you might then need to increase the MEMLIMIT size.For the best performance, the maximum heap size should, if possible, be contained in your machine's physical memory to avoid paging. The physical memory of your machine can be seen at the top of the verbose GC log. For example,
<system> <attribute name="physicalMemory" value="33617920000" />
This value is in bytes, so the RAM for the LPAR in this example is approximately 32G. The heap for your IBM z/OS Connect servers share the LPAR's RAM with other applications. Therefore, if you notice on the SDSF DA panels your IBM z/OS Connect servers paging, allocate more RAM to the LPAR or reduce the number of processes that compete for the RAM in the LPAR.
The following screen capture is taken from the Java Health Center while a IBM z/OS Connect ran a consistent workload. You can see how the GCs fall within the ideal 40-70% range of the heap size currently used. When the new (nursery) area of the heap is filled up, the JVM pauses to clear the old (tenure) area, and moves the objects from the new area to the old area. During this JVM pause, no other work is processed. Therefore, the JVM pauses should be kept to a minimum.In the following Summary screen capture for the GC, also taken from the Java Health Center, the two key metrics are:
- The "Proportion of time spent in garbage collection pauses (%)". A value of 2% or less indicates good performance.
- The "Proportion of time spent unpaused (%)". A value of 98% or more indicates good performance.
- Pause-less garbage collection
- The Java option, concurrentScavenge, is
supported by all generations of IBM Z® hardware to enable
pause-less GC with two modes of operation: hardware-based and software-based operations. IBM z13®™ and earlier hardware operates in software-based
pause-less GC mode; IBM z14®™ and later hardware (with
supported software) operates in hardware-based mode. It is only supported when using the
genconGC policy and is enabled by specifying -Xgc:concurrentScavenge as a Java option.
Despite the name of this feature, GCs are not completely free of pauses, but pause-times might be shorter for some workloads that throttle peak throughput to meet response time Service Level Agreements (SLAs). For more information, see concurrentScavenge in the IBM SDK, Java Technology Edition 8 documentation.
Specifying optional environment variables
- Specify the STDENV variables in line, as shown in the sample started task procedure provided in <hlq>.SBAQSAMP(BAQSTRT).
- Use a z/OS
UNIX file. For example, place the file
<name>.env in the
<WLP_USER_DIR>/servers directory, enter each of your
environment variables on separate lines, and add the following to the started task
where <name>.env is the name of the file containing the environment variables in the <WLP_USER_DIR>/servers directory.
//STDENV DD PATH='<WLP_USER_DIR>/servers/<name>.env'
Specifying JVM options
JVM_OPTIONS STDENVvariable. If you need to specify multiple JVM options which would exceed the line length of your started task procedure, you can either store all the STDENV variables in a z/OS UNIX file as described above, or store only your JVM options in a z/OS UNIX file. For example, create a z/OS UNIX EBCDIC file called <name>.options in the <WLP_USER_DIR>/servers directory, enter each of your JVM options on separate lines and add the following to the STDENV statement of the started task procedure:
where <name>.options is the name of the file that contains the JVM options in the <WLP_USER_DIR>/servers directory. For more information, see https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.zos.80.doc/diag/appendixes/cmdline/Xoptionsfile.html?cp=SS7K4U_liberty in the WebSphere Application Server for z/OS Liberty documentation.
Calculating the MEMLIMIT
MEMLIMIT is a fixed memory size that you can set in the JCL for IBM z/OS Connect . It controls the amount of virtual
storage above-the-bar that IBM z/OS Connect can access
in its address space. However, it is a
soft cap and not pre-allocated as the storage is only
used if required. Sample JCL supplied with IBM z/OS Connect sets this value to 8G, which should be
ample for most users. However, if the maximum Java heap is set
to 5.5G or more, then MEMLIMIT should be increased to be 50% larger than Java heap. For example, if maximum Java heap
size is 6G, then MEMLIMIT = 6G + (50% of 6G) which is 9G. The
extra virtual storage is to allow for native thread stacks, the JIT data cache, and other JVM
If you choose to set the maximum number of executor threads (see Java threads: Executor threads) you can alternatively calculate MEMLIMIT use the following formula
To check the MEMLIMIT value configured for your IBM z/OS Connect server, you can either issue the following MVS command from SDSF /D OMVS,LIMITS,PID=<server pid> and observe the MAXMEMLIMIT value in the PROCESS LIMIT column, or look in the messages.log file for message CWWKB0126I. The value for MEMLIMIT in the messages.log file is shown as a hexadecimal value. For example, CWWKB0126I: MEMLIMIT=2000, which equates to 8G.
From SDSF, you can issue the MVS command /D OMVS,LIMITS,PID=<server pid> to see how much virtual storage is currently being used by your IBM z/OS Connect server by observing the CURRENT USAGE and HIGHWATER USAGE values. For example, in Figure 7, the current usage is 1870M. The amount of virtual storage peaked at 1990M, well within the 8192M configured.
Setting the REGION size
You can set the REGION value in the JCL for IBM z/OS Connect . z/OS uses the region size to determine the amount of 31-bit storage available to running programs (that is, above-the-line and below-the-bar). Do not restrict the region size, but allow the Java runtime environment to use what is necessary. Restricting the region size might cause failures with storage-related error messages or abends such as 878-10.
Therefore, leave the default setting of REGION=0M, so the JVM uses what it needs.
Handling out-of-memory errors
-XX:+CrashOnOutOfMemoryError option is not supported by the IBM JVM. When an
OutOfMemoryError occurs in the IBM z/OS Connect JVM, the server continues running, but
stops processing work.
Instead, use the Java option
<command_string> is a command or list of commands to run when a
java.lang.OutOfMemoryError occurs. Multiple commands must be separated by a
-XX:OnOutOfMemoryError='"/bin/date; /bin/echo MSG1234 Restart;kill -9 %p
java.lang.OutOfMemoryErrorcondition occurs, the message MSG1234 Restart is issued and the server is stopped.
- Maximum heap size of 1G (Xmx)
- Minimum heap size of 8M (Xms)
- MEMLIMIT = 8G
- REGION size = 0M
- One IPIC connection with 100 sessions.
Before a workload was started, the Java heap for the started server was observed to use approximately 80M-90M.
A workload was then started with 100 clients sending 100 byte JSON requests to the server, with each client receiving a 16K response. After several hundred thousand requests had run to allow the JIT to warm up, the Java heap size was observed by using the Java Health Center. As the number of clients was gradually increased to 500 clients (500 being the maximum number of clients for this scenario), it was clear from the Java Health Center that the Java heap was outside the optimum 40-70% range at 34%. This indicated that the maximum heap size was too large for this scenario and could be reduced to the heap size observed, plus 43%. Not only does this reduction save on RAM but makes the GC more efficient and fewer pause times are observed.
- Maximum heap size of 328M (Xmx)
- Minimum heap size of 64M (Xms)
- MEMLIMIT = 8G
- REGION size = 0M
- One IPIC connection with 100 sessions.
The two key metrics, Proportion of time spent in garbage collection pauses (%) and Proportion of time spent unpaused (%) were checked in the Summary pane of the Java Health Center and indicated good performance.
|Workload #clients||TPS||Java heap size||#Java threads||Native threads (MaxThreads)||MAXMEMLIMIT Highwater Usage|
- The initial measurement was taken immediately after the server started and showed how much heap was used and the number of Java and native threads. See table row where "Workload #clients = 0".
- A workload of 100 clients required an increase in Java heap and threads to handle the requests. 30M of this heap was for the IPIC connection, which was established with 100 sessions. The amount of virtual storage required from the MEMLIMIT setting of 8G increased to 1289M.
- Increasing the number of simultaneous clients naturally required more Java heap and threads as the server was required to handle and process more work in parallel. However, the extra Java heap and threads did not increase significantly, while the amount of virtual storage required from MEMLIMIT remained static at 1289M.
The output from the MVS command /D OMVS,LIMITS,PID=<server pid> showed that the number of native threads used for the workload was acceptable and did not increase dramatically as the workload increased. The virtual storage required, 1990M, which was well within the MEMLIMIT of 8G.
In addition to monitoring the Java heap and thread usage, check the CPU usage for any contention through collecting and viewing SMF records. For more information, see GCPs and zIIPs.