Identifying performance problems

Use problem determination techniques and tools to identify any performance problems.

If possible, perform load testing during the implementation phase to identify performance problems. To determine whether there is any performance impact from patches or from data growth over time, you can use load testing after Maximo® Real Estate and Facilities is in production. If you encounter problems, use the following techniques and tools to identify any performance problems.

Determining the source of the problem

Application server logs

Review IBM® Maximo Real Estate and Facilities server logs for any errors. For load-balanced implementations, pay attention to the distribution of users across the JVMs. Monitor the JVM memory utilization on the application server by enabling verbose garbage collection.

Log files are in the Maximo Real Estate and Facilities installation subfolder that is named \log\. Log files are appended with the creation date. You can also download and roll over log files from the error logs in the Administrator Console. Several tools are available that can help you to analyze the following log files:
server.log
Contains standard Maximo Real Estate and Facilities logging.
security.log
Contains security-related logging, user logins, and admin user changes.
ObjectMigration.log
Contains Object Migration (OM) tool logging.
performance.log
Contains performance DEBUG-level logging from the platform logging mechanism.
systemmetrics.log
Contains performance information from the Administrator Console.

Web server logs

Review the web server logs for errors. View the maximum connections and total connection attempts to see whether your web server can use as many connections as it needs. Compare these numbers to memory and processor usage figures to determine whether a connection causes the problem and not some other component.

View the following IBM HTTP Server log files:
  • access.log
  • admin_access.log
  • admin_error.log
  • error.log
  • http_plug.log

The ThreadsPerChild parameter specifies the number of threads that are created by each child process. You might need to increase the ThreadsPerChild setting in the httpd.conf configuration file for IBM HTTP Server . The value forThreadsPerChild in Windows environments is 2400.

Database server

Monitor database server memory and instance memory. To assist with tuning issues, gather database traces and snapshots. For more information about Microsoft SQL Server, see Microsoft SQL Server database. For more information about general database tuning, see Tuning the database server.

Network

Sometimes you need to understand the network speed throughput from different client locations. In the Administrator Console, use the Performance Monitor to test different client locations and compare network speed throughputs. The network Speed Throughput Test link tests the network speed from the server to the current user's computer. In the results, you can see your speed throughput in kilobytes, and then you can see an average comparison with other network structures.

Provide the network Speed Throughput Test link to users without the need for them to log in. To identify a potential network overload, monitor the bytes per second that are processed by the network interface card. If you need more detailed network information to understand bandwidth requirements, there are bandwidth-monitoring tools that provide the ability to analyze HTTP requests, the number of roundtrips between tiers, and TCP/IP packet information. For more information, see Tuning the network.

CPU

Monitor the CPU to ensure that all processors are being used as expected and that overall CPU utilization remains healthy.

Memory

Monitor total memory usage. JVM heap size is the most important memory metric to monitor on application servers. There are several parts to the Maximo Real Estate and Facilities platform that, when unchecked or poorly configured, can contribute to a substantial memory footprint on the application server and can cause the server to encounter an Out of Memory situation in which the Maximo Real Estate and Facilities server crashes. For more information, see Tuning IBM Maximo Real Estate and Facilities components.

Manage the following memory-related performance issues:
Memory footprint contributors
The following processes can increase the heap memory on the application server:
  • Workflow Instances: When set to ALWAYS, the WF_INSTANCE_SAVE property consumes a substantial amount of memory on the application server and slows down the performance of workflows and actions. Set workflow instance saving to ALWAYS only if you are actively debugging workflows. Do not set to ALWAYS for longer than needed.
  • BIRT reporting: The BIRT engine consumes a substantial amount of heap memory when exporting extensive datasets.
  • DataConnect task: When you build a workflow with the DataConnect task, commit no more than 10 records at a time in the Transaction section. Tune this setting to reflect on the degree of integration for your Maximo Real Estate and Facilities implementation.
Diagnosing Out of Memory errors
When an Out of Memory error occurs, restart the application server. Generate a heap dump at the point of the Out of Memory error.

In WebSphere® Application Server Liberty profile, the heap output file is created in the default directory, ${server.output.dir}.

Memory Analyzer
Analyze the heap dump by using the Eclipse Eclipse Memory Analyzer Tool (MAT) to find the memory leaks and reduce memory consumption. If your heap dump is larger than 6 GB, the MAT consumes a lot of memory. Your workstation needs at least 16 GB of RAM. Close all other applications and configure the Eclipse config.ini file to set its own max heap size to 15 GB by using -Xmx15G.

The Overview section gives you insight into what the heap contains. Typically, the first- level or second-level objects explain what consumed the heap. Figures 1 and 2 are examples of BIRT report and workflow instance Out of Memory heaps. You can identify the main area that consumed the heap in the Problem Suspect section.

The Memory analyzer workflow instance Out of Memory heap message is as follows:
1,051,855 instances of "com.tririga.platform.workflow.runtime.WFContext", loaded 
by "war:IBM-TRIRIGA_Build-227459/ibm-tririga.war" occupy 3,637,218,776 (86.70%) bytes.
Figure 1. Memory analyzer workflow instance Out of Memory heap
Memory analyzer workflow instance 'Out of Memory' heap pie chart
The memory analyzer BIRT report Out of Memory heap is as follows:
One instance of "org.eclipse.birt.report.model.core.DesignSession", loaded 
by "<system class loader>" occupies 8,471,008,096 (89.43%) bytes. The memory is accumulated in one instance of 
"Java.Lang.Object[]" loaded by "<system class loader>".
Figure 2. Memory analyzer BIRT report Out of Memory heap
Memory analyzer BIRT report Out of Memory heap pie chart

Problem determination tools

Use the following tools to determine any performance problems with your IBM Maximo Real Estate and Facilities.

Application platform tools

Use the following tools to analyze system performance, workflow performance, and other areas of the platform:
IBM Maximo Real Estate and Facilities Administrator Console
Provides a collection of administrative tools to analyze and optimize system performance.
Workflow Analysis Utility (WAU)
Analyzes workflow performance and process execution. The utility reads Maximo Real Estate and Facilities performance logs and displays performance analytics for workflows, including workflow execution time, and process flow, which is the order in which workflows run and what triggered them to run. The Workflow Analysis Utility is available as a separate utility from the product.
Performance Analyzer
Analyzes issues in system performance. The Performance Analyzer provides a more streamlined approach to troubleshooting performance issues than the traditional Maximo Real Estate and Facilities performance log analysis. The Performance Analyzer helps you to better isolate and analyze the causes of performance issues by generating a log that is more targeted for the problem area.Access the Performance Analyzer in IBM Maximo Real Estate and Facilities under Tools. You can generate a log from the Performance Analyzer that is more targeted at the problem area. When you select the performance timings that you want and start a performance run, the analyzer writes to the performance log while you do the process or actions that cause the slowness.

When you stop the performance run, the analyzer automatically loads and analyzes the performance data to show the most time consuming events of each category type in the performance run. A result summary of the performance run shows the top events that took the longest time to run. You can further analyze the data by reviewing the result details. You can also upload an existing performance log to the analyzer and the analyzer does the same analysis of the data.

Heap dump, thread dump, and garbage collection utilities

Use the following tools to debug Java™ code:

IBM Thread and Monitor Dump Analyzer for Java
Analyzes javacore files and diagnoses monitor locks and thread activities to identify the root cause of hangs, deadlocks, and resource contention. IBM Thread and Monitor Dump Analyzer for Java also monitors bottlenecks.
IBM Pattern Modeling and Analysis Tool for Java Garbage Collector
Parses verbose GC trace, analyzes Java heap usage, and provides key configurations based on pattern modeling of Java heap usage.
Eclipse Memory Analyzer Tool (MAT)
Finds memory leaks and reduces memory consumption. If your heap dump is larger than 6 GB, the MAT consumes a substantial amount of memory. Your workstation needs at least 16 GB of RAM. Close all other applications and configure the Eclipse config.ini file to set its own max heap size to 15 GB by using -Xmx15G.

Application profiling utilities

Use the following tools to profile and debug Java code:

Health Center
A GUI-based diagnostics tool for monitoring the status of a running Java virtual machine (JVM).
YourKit
A CPU and memory Java Profiler that supports J2EE/J2ME.
OProfile
A system-wide profiler for Linux® systems that can profile any running code with low overhead.

Database utilities

Each of the database platforms contains tools to analyze database health and SQL queries to assist with any long-running SQL statements. For more information, see your database documentation. Some of the more useful tools include the following:

  • Oracle Automatic Workload Repository (AWR) snapshots
  • Oracle Automatic Database Diagnostic Monitor (ADDM) analysis reports
  • IBM Data Studio
  • IBM Db2® monitors and snapshots
  • IBM Db2 Design advisor db2advis and execution plans