6. Application Server Tuning
Contents
This section discusses settings that can optimize your supported application server. While these settings provide optimal performance in IBM test environments, your environment may require different settings. You can use these settings as a guideline or as a starting point, and then customize the settings to your specific environment requirements. For the latest supported application server versions and fix packs, always check the IBM TRIRIGA Compatibility Matrix.
6.1 General Recommendations
6.1.1 JVM Heap Size Values
You can use the following minimum heap size values to start tuning:
- Initial Heap Size:
6144M
- Maximum Heap Size:
6144M
When planning for overall system memory consumption, include additional overhead memory to be used by the JVM outside of the heap size as well as available RAM for the operating system.
A general rule-of-thumb is to include an additional 30-40% of memory to account for this overhead. Not having enough additional available RAM can cause severe performance issues.
If you are not utilizing the
Lease Accounting or BIRT reporting features of TRIRIGA, you may be able to start your heap
size tuning with a value less than 6144M
. However, smaller values less than
4096M
are not recommended, unless you have a development or test deployment
of TRIRIGA.
In a production environment, set the initial heap size to be the same value as the maximum heap size. This setting avoids the overhead of Java having to increase the heap size during processing. However, in modern JVMs, only a small amount of time is spent allocating new heap space. So, in a non-production environment, or one where speed is not a concern yet memory may be constrained, it is fine to keep the initial heap size smaller.
Java virtual machine (JVM) heap size parameters directly influence garbage collection (GC) behavior. Increasing the heap size permits more objects to be created before it triggers a garbage collection. However, a larger heap size means a larger amount of time is needed to find and process objects that need to be garbage collected. Consequently, JVM heap size tuning often involves a balance between the interval between garbage collections and the pause time needed to perform the garbage collection.
To tune the JVM heap size, enable verbose GC. When enabled, the JVM prints out useful information at each garbage collection, such as the amount of free and used bytes in the heap, the interval between garbage collections, and the pause time. This information can then be used to analyze the heap usage.
Once tuning is complete, disable verbose GC and delete the log file, since the log file can grow quite large while verbose GC is enabled. To enable verbose GC for WebSphere® Application Server, see the section 9.1.4 Displaying Garbage Collection Statistics on the Server in the 9 Troubleshooting and Monitoring best practices.
In addition, the TRIRIGA Application Platform can log
heap memory utilization in the systemmetrics.log
file. You can analyze the
tab-delimited systemmetrics.log
file to ensure that heap memory utilization is
ideal. To set up Performance Monitor logging (Key Metrics section) in the Administrator Console to
obtain this data, see the Administrator Console User Guide.
By using the guidelines above, you can analyze heap memory usage and garbage collection data to determine the appropriate value for a specific deployment that minimizes garbage collection times while providing enough memory to the application. Your specific deployment may require different settings to produce optimal performance or avoid out-of-memory issues in the JVM.
a. Lease Accounting and BIRT Reporting
Lease Accounting and BIRT reporting can require large amounts of heap space,
depending on the data that is being processed. For Lease Accounting, this is particularly true for
any process server that runs the asynchronous workflows in Lease Accounting. For BIRT, heap
requirements are dependent on the type of report that is being run and the amount of data that a
report can return. Some BIRT reports may require heap sizes much larger than 6144M
.
Heap sizes as large as 12288M
have been seen to process as-shipped Lease Disclosure
reports with a large amount of data.
Be aware that even if you have offloaded BIRT reporting to a separate BIRT process server, there are instances when BIRT reports can run on the application server; namely, when reports are rerun from the BIRT report icon bar. In those cases, heap memory on the application server may also require heap sizes as large as those allocated on the BIRT process server.
6.1.2 Connection Pool
The JDBC data source connection pool properties should be configured to allow for enough connections required by your system. These values are dependent on the number of concurrent users expected, in addition to the number of transactions expected. The following options and values can be configured for connection pools:
- Minimum Connections: Specifies the minimum number of connections to maintain in the pool.
- Typical Default:
1
- Recommended:
10
- Typical Default:
- Maximum Connections: Specifies the maximum number of connection to maintain in the pool.
- Typical Default:
10
- Recommended:
200
- Typical Default:
A setting of 200
maximum connections is common for a JVM that is not expected to
exceed 2500 concurrent users. For example, if you have two JVMs, both with 200
maximum connections, then in theory, you can support up to 5000 concurrent users.
6.1.3 Temporary Files
Over time, on the application or process servers, the temp (temporary) directory might get filled with temporary files. Consequently, you might want to clean up these files on a regular basis. Generally, it is safe to delete temporary files that are at least 2 days old.
You can set up a cronjob
for the Linux or Unix platform, or a scheduled task for
the Windows platform, that deletes temp files that are over 2 days old, and
that are owned by the user who is running the TRIRIGA application. For example, in a Linux
environment:
- Step 1. Add the following
find
command in a shell script file (/path/to/script/cleanTririgaTemp.sh):find -mtime +2 -user tririga -exec rm {} \;
- Step 2. Use the
chmod
command to make it executable:chmod u+x /path/to/script/cleanTririgaTemp.sh
- Step 3. Run
crontab -e
:$ crontab -e
- Step 4. Set the script to run daily at 1
AM:
0 1 * * * /path/to/script/cleanTririgaTemp.sh
6.2 IBM WebSphere Application Server, Liberty Profile
When choosing between IBM WebSphere Application Server (WAS) types to use with TRIRIGA, the best practice is to use IBM WAS Liberty Profile with IBM TRIRIGA Application Platform 3.4.2.x and later. When should you choose traditional WebSphere (tWAS) over Liberty? If you have a large WebSphere infrastructure and dedicated expert-level support in your organization, then tWAS might be a good choice. But you should still evaluate the advantages of Liberty over tWAS for your TRIRIGA installation.
- Liberty is easier to manage
than traditional WebSphere (tWAS) and WebLogic (WL). Liberty has just a few configuration files:
The
server.xml
,server.env
, andjvm.options
. - Liberty takes a more secure and efficient approach: Only install, enable, and configure what is needed. The Liberty paradigm enables the application server to be nimble, dynamic, and fast. The older tWAS paradigm was to have most features installed by default. For TRIRIGA, there is a massive amount of overhead that comes with tWAS.
- Liberty installation and configuration takes literally seconds with the TRIRIGA installer. With Liberty bundled with the TRIRIGA installer, there are no separate downloads, installations, or configuration steps. The time savings alone for Liberty installation can be compared by calculating tWAS installation as follows.
- tWAS installation times:
- Time to download Installation Manager and tWAS: 60 minutes
- Time to install Installation Manager: 10 minutes
- Time to install tWAS: 30 minutes
- Time to apply fix packs to tWAS: 30 minutes
- Time to create a profile: 30 minutes
- Time to change to the correct Java version: 5 minutes
- Additional time to install TRIRIGA: 45 minutes
- Total time for tWAS installation alone: 3.5 to 4 man-hours per server as a conservative estimate. Multiply that by the number of servers you have, and by comparison, Liberty provides a multi-thousand dollar savings in man-hours spent in installation alone.
6.3 IBM WebSphere Application Server, Full Profile
This section discusses the recommended settings that can optimize the IBM WebSphere Application Server (WAS) Full Profile or traditional WAS (tWAS). For more information on tuning traditional WAS, see WebSphere Application Server Performance.
It is important to tune the application server at the operating system level. For more information on tuning operating system parameters for WAS, see the section 4.1 Server Operating System Configuration in the 4 Operating System Configuration best practices, and see Tuning operating systems.
While the following settings provide optimal performance in IBM test environments, your environment may require different settings. You can use these settings as a guideline or as a starting point, and then customize the settings to your specific environment requirements. This information is based on the IBM SDK. If you run an environment that uses the Oracle SDK, see Tuning HotSpot Java virtual machines (Solaris & HP-UX).
6.3.1 Generic JVM Arguments
You can tune the JVM settings in WAS by using the WAS Administrative Console. After logging into the console, select Servers > Server Types > WebSphere application servers > [server_name] > Process definition > Java Virtual Machine.
-
Unloading Kickoff Threshold: The command line option
-Xgc:classUnloadingKickoffThreshold=6500
tells the system that a concurrent tenure collection be started every time 6500 new class loaders have been created since the last tenure collection.-Xgc:classUnloadingKickoffThreshold=6500
6.3.2 Thread Pool Tuning
You can tune the JVM settings in WAS by using the WAS Administrative Console. After logging into the console, select Servers > Server Types > WebSphere application servers > [server_name] > Thread pools. Thread pools enable components of the server to reuse threads, eliminating the need to create new threads at run time to service each new request. The Default thread pool is used when requests come in for a message-driven bean or if a particular transport chain has not been defined for a specific thread pool. The WebContainer thread pool is used when requests come in over HTTP.
The following options and values can be configured for thread pools:
- Minimum Size: Specifies the minimum number of threads to maintain in the pool.
- Default Thread Pool:
20
- WebContainer Thread Pool:
120
- Default Thread Pool:
- Maximum Size: Specifies the maximum number of threads to maintain in the pool.
- Default Thread Pool:
50
- WebContainer Thread Pool:
120
- Default Thread Pool:
- Thread Inactivity Timeout: Specifies the amount of inactivity (in milliseconds) that can
elapse before a thread is reclaimed.
- Default Thread Pool:
30000
- WebContainer Thread Pool:
60000
- Default Thread Pool:
- Allow Thread Allocation Beyond Maximum Thread Size: Specifies whether the number of
threads can increase beyond the maximum size configured for the thread pool.
- Value: Enabled
You can use Tivoli Performance Viewer (TPV) to monitor thread usage to ensure you have set the values appropriately. If TPV shows the Percent Maxed metric remaining consistently in the double digits, consider increasing the maximum size. For more information on configuring thread pool settings, see Thread pool settings.
6.3.3 IBM DB2 JDBC Data Sources
When using IBM DB2® as the back-end server, ensure that the data collection performance settings are configured by adding or verifying the following property in the data source properties:
jdbcCollection=NULLIDR1
In addition, ensure that the REOPT(ONCE)
bind option is set in DB2. For more information on setting the
bind option, see the section 5.3.1 IBM DB2 Database
Server Tuning.
In addition, the isolation level for WebSphere Application Server must be set to 2
(TRANSACTION_READ_COMMITTED) to prevent database locking. Add or verify the following property in
the data source properties:
webSphereDefaultIsolationLevel=2
You can verify these settings in WAS by using the WAS Administrative Console. After logging into the console, navigate to Resources > JDBC > JDBC providers. Select the DB2_JDBC-Driver_for_IBM-TRIRIGA. Under Additional Properties, select Data Sources, then select your TRIRIGA data source (e.g. data). On the configuration screen for the data source, under Additional Properties, select Custom Properties.
6.4 IBM HTTP Server
You can use the settings below to optimize IBM HTTP Server (IHS) for TRIRIGA.
While the following settings provide optimal performance in IBM test environments, your environment may require different settings. You can use these settings as a guideline or as a starting point, and then customize the settings to your specific environment requirements. These settings were optimized and tested for the Microsoft Windows environment. Although the settings were also tested on an AIX® environment, they are not optimized for UNIX or Linux®. For more information on how to tune IBM HTTP Server, see IBM HTTP Server Performance Tuning.
The following options and values can be configured by editing the IHS httpd.conf
configuration file:
- TimeOut: Specifies the amount of time the server waits for certain events before failing
a request.
- Value:
900
- Value:
- KeepAliveTimeOut: Specifies the amount of time the server waits for subsequent requests
on a persistent connection. Set this value to
60
for those environments that may have high network latency, such as when users are located in areas that are geographically different from the location of the servers.- High Network Bandwidth:
10
- Low Network Bandwidth:
60
- High Network Bandwidth:
- MaxKeepAliveRequests: Specifies the number of requests allowed on a persistent
connection. Setting this value to
0
(zero) allows unlimited requests.- Value:
0
- Value:
- MaxRequestsPerChild: Specifies the limit on the number of requests that an individual
child server handles during its life. Once this limit is reached, the child process dies. Setting
this value to
0
(zero) ensures that the process never expires.- Value:
0
- Value:
- ThreadLimit: Specifies the upper limit on the configurable number of threads per child
process.
- Value for Windows environments:
2400
- Value for Windows environments:
- ThreadsPerChild: Specifies the number of threads created by each child process. In a
Windows environment, where there is only one child process, this value must be set to a high
enough value to handle the entire load of the server. In UNIX or Linux environments,
threads and processes are allocated differently than in Windows. You would not normally set the
ThreadLimit to such a high value. You can set the ThreadLimit and
ThreadsPerChild to the same value.
- Value for Windows environments:
2400
- Value for Windows environments:
6.5 Oracle WebLogic Server
If you are running WebLogic instead of WebSphere for TRIRIGA, see the WebLogic Server Performance and Tuning guide that is provided by Oracle for general tuning guidance. Specific settings, such as JVM heap size recommendations, should follow TRIRIGA guidelines, where applicable. However, you will need to refer to the tuning guide for appropriate instructions to set these values for WebLogic.
For HTTP compression on WebLogic, use Apache HTTP
Server 2.x and above as an external
HTTP server. Use mod_deflate
and set DeflateCompressionLevel
to
3
to improve response time in environments that have low bandwidth and high
latency.
6.6 HTTP Compression
HTTP compression is a capability that can be built into web servers, such as IBM HTTP Server (IHS) and Apache HTTP Server (AHS) 2.x and above, and web browsers to make better use of available bandwidth, and to provide faster transmission speeds. For more information, see the section 2.3.1 HTTP Compression in the 2 Network Considerations best practices.
6.7 Load Balancing
Load balancing is the distribution of the task load across multiple instances of an application. User load comes from users who are logged into the system and use the interface to perform tasks. Non-user load comes from items such as scheduled jobs, agents, and integrations. You can distribute user load and non-user load across different application servers. The IBM HTTP Server for WebSphere Application Server plug-in acts as an "agent" that redirects HTTP requests from the web server to the application server using the HTTP protocol.
You can improve the performance of IBM HTTP Server (with the plug-in) by modifying the load balancing option that the plug-in uses when sending requests to the application servers associated with that web server. The default load balance option, Round Robin, provides an even distribution of work across cluster members. However, the Random option yields a more even distribution of work across the cluster. You must test the options to determine which option works better for your deployment of TRIRIGA. You can analyze the security logs for the TRIRIGA Application Platform to identify the number of users per JVM to assist in determining which option provides the best load balancing.