Identifying performance problems

Use problem determination techniques and tools to identify any performance problems.

If possible, perform load testing during the implementation phase to identify performance problems. You can use load testing after TRIRIGA® is in production to determine whether there is any performance impact from patches or from data growth over time. If you encounter problems, use the following techniques and tools to identify any performance problems.

Problem determination techniques

Application server logs

Review IBM TRIRIGA Application Platform server logs for any errors. For load-balanced implementations, pay attention to the distribution of users across the JVMs. Monitor the JVM memory utilization on the application server by enabling verbose garbage collection.

Log files are in the TRIRIGA installation subfolder named \log\. Log files are appended with the creation date. You can also download and roll over log files from the Error Logs manager in the Administrator Console. Several tools are available that can help you to analyze the following log files:

server.log: Contains standard TRIRIGA logging.
security.log: Contains security-related logging, user logins, and admin user changes.
ObjectMigration.log: Contains Object Migration (OM) tool logging.
performance.log: Contains performance DEBUG level logging from the platform logging mechanism.
systemmetrics.log: Contains performance information from the Administrator Console.

Web server logs

Review the web server logs for errors. View the maximum connections and total connection attempts to see whether your web server can use as many of the connections as it needs. Compare these numbers to memory and processor usage figures to determine whether a connection causes the problem and not some other component.

View the following IBM HTTP Server (IHS) log files:

access.log
admin_access.log
admin_error.log
error.log
http_plug.log

The ThreadsPerChild parameter specifies the number of threads that are created by each child process. You might need to raise the ThreadsPerChild setting in the IHS httpd.conf configuration file. The value forThreadsPerChild in Windows environments is 2400.

Database server

Monitor database server memory and instance memory. Gather database traces and snapshots to assist with tuning issues. For more information on Microsoft SQL Server, see Microsoft SQL Server database. For more information on general database tuning, see Tuning the database server.

Network

Sometimes you need to understand the network speed throughput from different client locations. In the Administrator Console, use the Performance Monitor to test different client locations and compare network speed throughputs. The network Speed Throughput Test link tests the network speed from the server to the current user's computer. In the results, you can see your speed throughput in kilobytes, and then an average comparison with other network structures.

Provide the network Speed Throughput Test link to users without the need for them to log in. Monitor the bytes per second processed by the network interface card to identify a potential network overload. If you need more detailed network information to understand bandwidth requirements, there are bandwidth-monitoring tools that provide the ability to analyze HTTP requests, the number of roundtrips between tiers, and TCP/IP packet information. For more information, see Tuning the network.

CPU

Monitor the CPU to ensure that all processors are being used as expected and that overall CPU utilization remains healthy.

Memory

Monitor total memory usage. JVM heap size is the most important memory metric to monitor on application servers. There are several parts to the TRIRIGA platform that, when unchecked or poorly configured, can contribute to a large memory footprint on the application and process servers, and can cause the server to get into an Out of Memory situation where the TRIRIGA server crashes. For more information, see Tuning the application server and Tuning IBM® TRIRIGA components.

Manage the following memory-related performance issues:

Memory Footprint Contributors: The following processes can increase the heap memory on the application and process servers:
- Workflow Instances: When set to ALWAYS, the WF_INSTANCE_SAVE property consumes a large amount of memory on the application server, and slows down the performance of workflows and actions. Set workflow instance saving to ALWAYS only if you are actively debugging workflows. Do not set to ALWAYS for longer than needed.
- BIRT reporting: The BIRT engine consumes a large amount of heap memory when exporting large datasets.
- DataConnect task: When you build a workflow with the DataConnect task, commit no more than 10 records at a time in the Transaction section. Tune this setting to reflect on the degree of integration for your IBM TRIRIGA implementation.
Diagnosing Out of Memory errors: When an Out of Memory error occurs, restart the application and process servers. Generate a heap dump at the point of the Out of Memory error.
In WebSphere Application Server liberty profile, the heap output file is created in the default directory ${server.output.dir}.
Memory Analyzer: When you get the heap dump, analyze it with the Eclipse Memory Analyzer Tool (MAT) to find the memory leaks and reduce memory consumption.
MAT consumes a lot of memory if your heap dump is larger than 6 GB. Your workstation should have at least 16 GB of RAM, and you should close all other applications and configure the Eclipse config.ini file to set its own max heap size to 15 GB by using -Xmx15G.

The Overview section gives you insight into what the heap contains. Typically, the first- or second-level objects explain what consumed the heap. Figures 1 and 2 are examples of BIRT report and workflow instance Out of Memory heaps. You can identify the main area that consumed the heap in the Problem Suspect section.

Memory analyzer workflow instance 'Out of Memory' heap pie chart — Figure 1. Memory analyzer workflow instance `Out of Memory` heap

Memory analyzer BIRT report Out of Memory heap pie chart — Figure 2. Memory analyzer BIRT report `Out of Memory` heap

Problem determination tools

Use the following tools to determine any performance problems with your IBM TRIRIGA Application Platform.

Application platform tools

Use the following tools to analyze system performance, workflow performance, and other areas of the platform:

IBM TRIRIGA Administrator Console: Provides a collection of administrative tools to analyze and optimize system performance.
Workflow Analysis Utility (WAU): Analyzes workflow performance and process execution. The utility reads IBM TRIRIGA performance logs and displays performance analytics for workflows, including workflow execution time, and process flow, that is, the order in which workflows run and what triggered them to run.
IBM TRIRIGA Performance Analyzer: Analyzes issues in system performance. The Performance Analyzer provides a more streamlined approach to troubleshooting performance issues than the traditional IBM TRIRIGA performance log analysis. The Performance Analyzer helps you to better isolate and analyze the causes of performance issues by generating a log that is more targeted at the problem area.

Heap dump, thread dump, and garbage collection utilities

Use the following tools to debug Java code:

IBM Thread and Monitor Dump Analyzer for Java (TMDA): Analyzes javacore files and diagnoses monitor locks and thread activities to identify the root cause of hangs, deadlocks, and resource contention. TMDA also to monitors bottlenecks.
IBM Pattern Modeling and Analysis Tool for Java Garbage Collector: Parses verbose GC trace, analyzes Java heap usage, and provides key configurations based on pattern modeling of Java heap usage.
IBM Support Assistant: A workbench for problem determination.
Eclipse Memory Analyzer Tool (MAT): Finds memory leaks and reduces memory consumption. MAT consumes a large amount of memory if your heap dump is larger than 6 GB. Your workstation should have at least 16 GB of RAM, and close all other applications and configure the Eclipse config.ini file to set its own max heap size to 15 GB by using -Xmx15G.

Application profiling utilities

Use the following tools to profile and debug Java code:

Health Center: A GUI-based diagnostics tool for monitoring the status of a running Java virtual machine (JVM).
YourKit: A CPU and memory Java Profiler that supports J2EE/J2ME.
OProfile: A system-wide profiler for Linux® systems and can profile any running code with low overhead.

Database utilities

Each of the database platforms contains tools to analyze database health, and SQL queries to assist with any long-running SQL statements. For more information, see your database documentation. Some of the more useful tools include the following:

Oracle Automatic Workload Repository (AWR) snapshots
Oracle Automatic Database Diagnostic Monitor (ADDM) analysis reports
IBM Data Studio
IBM Db2 monitors and snapshots
IBM Db2 Design advisor db2advis and execution plans