6. Application Server Tuning

This section discusses settings that can optimize your supported application server. While these settings provide optimal performance in IBM test environments, your environment may require different settings. You can use these settings as a guideline or as a starting point, and then customize the settings to your specific environment requirements. For the latest supported application server versions and fix packs, always check the IBM TRIRIGA Compatibility Matrix.

6.1 General Recommendations

6.1.1 JVM Heap Size Values

You can use the following minimum heap size values to start tuning:

  • Initial Heap Size: 6144M
  • Maximum Heap Size: 6144M
Note: Overall System Memory & Heap Sizes

When planning for overall system memory consumption, include additional overhead memory to be used by the JVM outside of the heap size as well as available RAM for the operating system.

A general rule-of-thumb is to include an additional 30-40% of memory to account for this overhead. Not having enough additional available RAM can cause severe performance issues.

If you are not utilizing the Lease Accounting or BIRT reporting features of TRIRIGA, you may be able to start your heap size tuning with a value less than 6144M. However, smaller values less than 4096M are not recommended, unless you have a development or test deployment of TRIRIGA.

In a production environment, set the initial heap size to be the same value as the maximum heap size. This setting avoids the overhead of Java having to increase the heap size during processing. However, in modern JVMs, only a small amount of time is spent allocating new heap space. So, in a non-production environment, or one where speed is not a concern yet memory may be constrained, it is fine to keep the initial heap size smaller.

Java virtual machine (JVM) heap size parameters directly influence garbage collection (GC) behavior. Increasing the heap size permits more objects to be created before it triggers a garbage collection. However, a larger heap size means a larger amount of time is needed to find and process objects that need to be garbage collected. Consequently, JVM heap size tuning often involves a balance between the interval between garbage collections and the pause time needed to perform the garbage collection.

To tune the JVM heap size, enable verbose GC. When enabled, the JVM prints out useful information at each garbage collection, such as the amount of free and used bytes in the heap, the interval between garbage collections, and the pause time. This information can then be used to analyze the heap usage.

Once tuning is complete, disable verbose GC and delete the log file, since the log file can grow quite large while verbose GC is enabled. To enable verbose GC for WebSphere® Application Server, see the section 9.1.4 Displaying Garbage Collection Statistics on the Server in the 9 Troubleshooting and Monitoring best practices.

In addition, the TRIRIGA Application Platform can log heap memory utilization in the systemmetrics.log file. You can analyze the tab-delimited systemmetrics.log file to ensure that heap memory utilization is ideal. To set up Performance Monitor logging (Key Metrics section) in the Administrator Console to obtain this data, see the Administrator Console User Guide.

By using the guidelines above, you can analyze heap memory usage and garbage collection data to determine the appropriate value for a specific deployment that minimizes garbage collection times while providing enough memory to the application. Your specific deployment may require different settings to produce optimal performance or avoid out-of-memory issues in the JVM.

a. Lease Accounting and BIRT Reporting

Lease Accounting and BIRT reporting can require large amounts of heap space, depending on the data that is being processed. For Lease Accounting, this is particularly true for any process server that runs the asynchronous workflows in Lease Accounting. For BIRT, heap requirements are dependent on the type of report that is being run and the amount of data that a report can return. Some BIRT reports may require heap sizes much larger than 6144M. Heap sizes as large as 12288M have been seen to process as-shipped Lease Disclosure reports with a large amount of data.

Be aware that even if you have offloaded BIRT reporting to a separate BIRT process server, there are instances when BIRT reports can run on the application server; namely, when reports are rerun from the BIRT report icon bar. In those cases, heap memory on the application server may also require heap sizes as large as those allocated on the BIRT process server.

6.1.2 Connection Pool

The JDBC data source connection pool properties should be configured to allow for enough connections required by your system. These values are dependent on the number of concurrent users expected, in addition to the number of transactions expected. The following options and values can be configured for connection pools:

  • Minimum Connections: Specifies the minimum number of connections to maintain in the pool.
    • Typical Default: 1
    • Recommended: 10
  • Maximum Connections: Specifies the maximum number of connection to maintain in the pool.
    • Typical Default: 10
    • Recommended: 200

A setting of 200 maximum connections is common for a JVM that is not expected to exceed 2500 concurrent users. For example, if you have two JVMs, both with 200 maximum connections, then in theory, you can support up to 5000 concurrent users.

6.1.3 Temporary Files

Over time, on the application or process servers, the temp (temporary) directory might get filled with temporary files. Consequently, you might want to clean up these files on a regular basis. Generally, it is safe to delete temporary files that are at least 2 days old.

You can set up a cronjob for the Linux or Unix platform, or a scheduled task for the Windows platform, that deletes temp files that are over 2 days old, and that are owned by the user who is running the TRIRIGA application. For example, in a Linux environment:

  • Step 1. Add the following find command in a shell script file (/path/to/script/cleanTririgaTemp.sh):
    find -mtime +2 -user tririga -exec rm {} \;
  • Step 2. Use the chmod command to make it executable:
    chmod u+x /path/to/script/cleanTririgaTemp.sh
  • Step 3. Run crontab -e:
    $ crontab -e
  • Step 4. Set the script to run daily at 1 AM:
    0 1 * * * /path/to/script/cleanTririgaTemp.sh

6.2 IBM WebSphere Application Server, Liberty Profile

When choosing between IBM WebSphere Application Server (WAS) types to use with TRIRIGA, the best practice is to use IBM WAS Liberty Profile with IBM TRIRIGA Application Platform 3.4.2.x and later. When should you choose traditional WebSphere (tWAS) over Liberty? If you have a large WebSphere infrastructure and dedicated expert-level support in your organization, then tWAS might be a good choice. But you should still evaluate the advantages of Liberty over tWAS for your TRIRIGA installation.

  • Liberty is easier to manage than traditional WebSphere (tWAS) and WebLogic (WL). Liberty has just a few configuration files: The server.xml, server.env, and jvm.options.
  • Liberty takes a more secure and efficient approach: Only install, enable, and configure what is needed. The Liberty paradigm enables the application server to be nimble, dynamic, and fast. The older tWAS paradigm was to have most features installed by default. For TRIRIGA, there is a massive amount of overhead that comes with tWAS.
  • Liberty installation and configuration takes literally seconds with the TRIRIGA installer. With Liberty bundled with the TRIRIGA installer, there are no separate downloads, installations, or configuration steps. The time savings alone for Liberty installation can be compared by calculating tWAS installation as follows.
  • tWAS installation times:
    • Time to download Installation Manager and tWAS: 60 minutes
    • Time to install Installation Manager: 10 minutes
    • Time to install tWAS: 30 minutes
    • Time to apply fix packs to tWAS: 30 minutes
    • Time to create a profile: 30 minutes
    • Time to change to the correct Java version: 5 minutes
    • Additional time to install TRIRIGA: 45 minutes
    • Total time for tWAS installation alone: 3.5 to 4 man-hours per server as a conservative estimate. Multiply that by the number of servers you have, and by comparison, Liberty provides a multi-thousand dollar savings in man-hours spent in installation alone.

6.3 IBM WebSphere Application Server, Full Profile

This section discusses the recommended settings that can optimize the IBM WebSphere Application Server (WAS) Full Profile or traditional WAS (tWAS). For more information on tuning traditional WAS, see WebSphere Application Server Performance.

It is important to tune the application server at the operating system level. For more information on tuning operating system parameters for WAS, see the section 4.1 Server Operating System Configuration in the 4 Operating System Configuration best practices, and see Tuning operating systems.

While the following settings provide optimal performance in IBM test environments, your environment may require different settings. You can use these settings as a guideline or as a starting point, and then customize the settings to your specific environment requirements. This information is based on the IBM SDK. If you run an environment that uses the Oracle SDK, see Tuning HotSpot Java virtual machines (Solaris & HP-UX).

6.3.1 Generic JVM Arguments

You can tune the JVM settings in WAS by using the WAS Administrative Console. After logging into the console, select Servers > Server Types > WebSphere application servers > [server_name] > Process definition > Java Virtual Machine.

  • Unloading Kickoff Threshold: The command line option -Xgc:classUnloadingKickoffThreshold=6500 tells the system that a concurrent tenure collection be started every time 6500 new class loaders have been created since the last tenure collection.

    -Xgc:classUnloadingKickoffThreshold=6500
    

6.3.2 Thread Pool Tuning

You can tune the JVM settings in WAS by using the WAS Administrative Console. After logging into the console, select Servers > Server Types > WebSphere application servers > [server_name] > Thread pools. Thread pools enable components of the server to reuse threads, eliminating the need to create new threads at run time to service each new request. The Default thread pool is used when requests come in for a message-driven bean or if a particular transport chain has not been defined for a specific thread pool. The WebContainer thread pool is used when requests come in over HTTP.

The following options and values can be configured for thread pools:

  • Minimum Size: Specifies the minimum number of threads to maintain in the pool.
    • Default Thread Pool: 20
    • WebContainer Thread Pool: 120
  • Maximum Size: Specifies the maximum number of threads to maintain in the pool.
    • Default Thread Pool: 50
    • WebContainer Thread Pool: 120
  • Thread Inactivity Timeout: Specifies the amount of inactivity (in milliseconds) that can elapse before a thread is reclaimed.
    • Default Thread Pool: 30000
    • WebContainer Thread Pool: 60000
  • Allow Thread Allocation Beyond Maximum Thread Size: Specifies whether the number of threads can increase beyond the maximum size configured for the thread pool.
    • Value: Enabled

You can use Tivoli Performance Viewer (TPV) to monitor thread usage to ensure you have set the values appropriately. If TPV shows the Percent Maxed metric remaining consistently in the double digits, consider increasing the maximum size. For more information on configuring thread pool settings, see Thread pool settings.

6.3.3 IBM DB2 JDBC Data Sources

When using IBM DB2® as the back-end server, ensure that the data collection performance settings are configured by adding or verifying the following property in the data source properties:

jdbcCollection=NULLIDR1

In addition, ensure that the REOPT(ONCE) bind option is set in DB2. For more information on setting the bind option, see the section 5.3.1 IBM DB2 Database Server Tuning.

In addition, the isolation level for WebSphere Application Server must be set to 2 (TRANSACTION_READ_COMMITTED) to prevent database locking. Add or verify the following property in the data source properties:

webSphereDefaultIsolationLevel=2

You can verify these settings in WAS by using the WAS Administrative Console. After logging into the console, navigate to Resources > JDBC > JDBC providers. Select the DB2_JDBC-Driver_for_IBM-TRIRIGA. Under Additional Properties, select Data Sources, then select your TRIRIGA data source (e.g. data). On the configuration screen for the data source, under Additional Properties, select Custom Properties.

6.4 IBM HTTP Server

You can use the settings below to optimize IBM HTTP Server (IHS) for TRIRIGA.

While the following settings provide optimal performance in IBM test environments, your environment may require different settings. You can use these settings as a guideline or as a starting point, and then customize the settings to your specific environment requirements. These settings were optimized and tested for the Microsoft Windows environment. Although the settings were also tested on an AIX® environment, they are not optimized for UNIX or Linux®. For more information on how to tune IBM HTTP Server, see IBM HTTP Server Performance Tuning.

The following options and values can be configured by editing the IHS httpd.conf configuration file:

  • TimeOut: Specifies the amount of time the server waits for certain events before failing a request.
    • Value: 900
  • KeepAliveTimeOut: Specifies the amount of time the server waits for subsequent requests on a persistent connection. Set this value to 60 for those environments that may have high network latency, such as when users are located in areas that are geographically different from the location of the servers.
    • High Network Bandwidth: 10
    • Low Network Bandwidth: 60
  • MaxKeepAliveRequests: Specifies the number of requests allowed on a persistent connection. Setting this value to 0 (zero) allows unlimited requests.
    • Value: 0
  • MaxRequestsPerChild: Specifies the limit on the number of requests that an individual child server handles during its life. Once this limit is reached, the child process dies. Setting this value to 0 (zero) ensures that the process never expires.
    • Value: 0
  • ThreadLimit: Specifies the upper limit on the configurable number of threads per child process.
    • Value for Windows environments: 2400
  • ThreadsPerChild: Specifies the number of threads created by each child process. In a Windows environment, where there is only one child process, this value must be set to a high enough value to handle the entire load of the server. In UNIX or Linux environments, threads and processes are allocated differently than in Windows. You would not normally set the ThreadLimit to such a high value. You can set the ThreadLimit and ThreadsPerChild to the same value.
    • Value for Windows environments: 2400

6.5 Oracle WebLogic Server

Note: As of 2020, Oracle WebLogic Server is no longer supported with IBM TRIRIGA Application Platform 3.6.1.2 and later.

If you are running WebLogic instead of WebSphere for TRIRIGA, see the WebLogic Server Performance and Tuning guide that is provided by Oracle for general tuning guidance. Specific settings, such as JVM heap size recommendations, should follow TRIRIGA guidelines, where applicable. However, you will need to refer to the tuning guide for appropriate instructions to set these values for WebLogic.

For HTTP compression on WebLogic, use Apache HTTP Server 2.x and above as an external HTTP server. Use mod_deflate and set DeflateCompressionLevel to 3 to improve response time in environments that have low bandwidth and high latency.

6.6 HTTP Compression

HTTP compression is a capability that can be built into web servers, such as IBM HTTP Server (IHS) and Apache HTTP Server (AHS) 2.x and above, and web browsers to make better use of available bandwidth, and to provide faster transmission speeds. For more information, see the section 2.3.1 HTTP Compression in the 2 Network Considerations best practices.

6.7 Load Balancing

Load balancing is the distribution of the task load across multiple instances of an application. User load comes from users who are logged into the system and use the interface to perform tasks. Non-user load comes from items such as scheduled jobs, agents, and integrations. You can distribute user load and non-user load across different application servers. The IBM HTTP Server for WebSphere Application Server plug-in acts as an "agent" that redirects HTTP requests from the web server to the application server using the HTTP protocol.

You can improve the performance of IBM HTTP Server (with the plug-in) by modifying the load balancing option that the plug-in uses when sending requests to the application servers associated with that web server. The default load balance option, Round Robin, provides an even distribution of work across cluster members. However, the Random option yields a more even distribution of work across the cluster. You must test the options to determine which option works better for your deployment of TRIRIGA. You can analyze the security logs for the TRIRIGA Application Platform to identify the number of users per JVM to assist in determining which option provides the best load balancing.