IBM Support

MustGather: Performance, Hang, or High CPU issues with CLM applications

Troubleshooting


Problem

This document will assist you in collecting the data necessary to diagnose and resolve performance, hang, or high CPU for application server or database server issues with the IBM Rational Collaborative Lifecycle Management (CLM) products (includes Rational Team Concert (RTC), IBM Rational Quality Manager (RQM) and IBM Rational DOORS Next Generation (DNG) formerly Rational Requirements Composer (RRC)).

Symptom

Performance, hang, or high CPU issues with the CLM products may occur when installed in a distributed environment. Each issue may contribute to a variety of symptoms and behavioral deficiencies

Cause

This MustGather will assist you in collecting the data necessary to help you diagnose and resolve the issue. If you are unable to determine the root cause using the information collected, you should open a case with IBM Support for further investigation providing the data collected.

Resolving The Problem


You can use the IBM Support Assistant Lite (ISA Lite) Data Collector tool to quickly collect diagnostic files, such as log files, configuration files or to run traces. This tool is bundled with CLM 4.x and above. ISA Lite collects information about your Jazz Team Server environment and stores the information in a .zip archive file. If you have a need to open a case with IBM Support for further assistance, you can send the archive file with the data collection so that they can help diagnose and fix problems.


The information below should be gathered in addition to the normal information and log gathering done by ISA Lite.

Note: Further information to help troubleshoot performance issues with your CLM products is also available in the Performance troubleshooting section of the Deployment wiki on jazz.net.


Business Impact

Includes:

  • What affect is this having on the business
  • Is this a production or test environment
  • How many users are affected
     

Unexpected Behavior

Details include:

  • Problem description
  • Steps to recreate
  • Were there any recent changes made to the environment
  • Were new users able to log into any CLM applications?
  • What behavior did the logged-in users encounter?
  • Any errors or exceptions logged at the time the incident occurred

Topology

Description of the Topology includes:

  • Is this a stand alone or distributed environment?
  • The version of the CLM applications including all applied iFixes and hot fixes
  • Are the CLM applications deployed on Apache Tomcat or IBM WebSphere

    If WebSphere, provide the output of:

    <WebSphere_Install_Root>\bin\versionInfo -maintenancePackages

    and indicate if WebSphere is installed as 64bit or 32 bit

     
  • The Operating System the ELM applications are installed on
    • Specify 64- or 32- bit
    • Include the number of CPU's
    • How much memory is available
    • Available disk space
    • Size of indices on disk, for example, conf\<app>\indices
  • The Database vendor and version being used.

    If multiple database instances are being used, provide details on which ELM application is using which database instance and indicate if the database is installed as 32bit or 64bit
  • JVM params provided at server startup (heap sizes and config cache tuning) .  If ISA tool is used, this information is already available.
Javacore Log Files and Application Logs


Due to the distributed and interconnected nature of the CLM applications we will need at minimum data from the JTS server and the affected CLM application which are collected at the time the incident occurs. If possible, providing data from all JVMs would be preferred

  • Enable verbose gc and javacore creation
    • Websphere instructions are located here
    • Tomcat instructions:
      • Add the following two lines to the server.startup script or to the Java Options in the Tomcat Control Panel if running as a Windows Service (ensure that the - character is present at the beginning)
        -verbose:gc
        -Xdump:java:events=user
      • Restart the Application Server for the changes to take effect.
  • Gather 4-6 Java Cores in 30s intervals according to How to gather Java cores for different application servers in Engineering Lifecycle Management applications document.
  • Include all the CLM logs including ETL logs located in:

    WebSphere location: <WebSphere_Install_Root>\profiles\(profile_name)\logs directory
    Liberty Profile and Tomcat location: <CLM_Install_Root>\server\logs directory
  • If ELM is running on WebSphere, provide the WebSphere Performance, hang, or high CPU mustgathers for the operating system CLM is installed on (in addition to those listed above)
  • Provide access.log, error.log and http_plugin.log file if you use IHS.
  • Provide the report from monitoring tool like Splunk or APM tool if you have configured them to gather the data from the server.

Database usage logs
Gather Oracle AWR report or corresponding report for different database vendor.
Additional Information
  • Gather a screenshot of the following page from the web UI and save the content as Full HTML. Ensure you gather a screenshot for each application (example: jts, ccm, jazz, and others). For DNG, you may have to add the JVM parameter,  -Dcom.ibm.team.repository.service.internal.vvc.enableCounters=true  and restart the server prior to navigating to this page.  https://<hostname>/<context root>/service/com.ibm.team.repository.service.internal.counters.ICounterContentService
  • Gather the screenshot from list of active services: https://<hostname>/<context root>/admin#action=com.ibm.team.repository.admin.activeServices
  • Enable queryStats (Use Chrome!):
    - Open /rm/admin#action=com.ibm.team.repository.admin.configureAdvanced
    - Find the entry "Enable repodebug service" and set the value to "true"
    - Open /rm/repodebug/database/queryStats and select start
    - Perform one of the operations that is slow
    - Refresh /rm/repodebug/database/queryStats, save the entire page, and submit in the PMR
    - Disable "Enable repodebug service" (set to false)
  • For DOORS Next Generation, it is also useful for support to know the following:  
    • # of projects
    • Configuration Management enabled for projects?
    • Estimated # of artifacts per project (does not need to be precise!)
    • Value for "JDBC Connection Pool Size" (under ConnectionPoolService heading) and "Mediator Pool Size" in RM Advanced Properties
    • Estimated # of concurrent users
    • If Windows, check the value of -x[no]compressedrefs in server startup
    • If Windows, check value for Jena File Access Mode (and Default) in RM Advanced Properties

[{"Business Unit":{"code":"BU055","label":"Cognitive Applications"},"Product":{"code":"SSUVV6","label":"IBM Engineering Test Management"},"Component":"","Platform":[{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB02","label":"AI Applications"}},{"Business Unit":{"code":"BU055","label":"Cognitive Applications"},"Product":{"code":"SSUC3U","label":"IBM Engineering Workflow Management"},"Component":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB02","label":"AI Applications"}},{"Business Unit":{"code":"BU055","label":"Cognitive Applications"},"Product":{"code":"SSUVLZ","label":"IBM Engineering Requirements Management DOORS Next"},"Component":"","Platform":[{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB02","label":"AI Applications"}}]

Product Synonym

Rational DOORS Next Generation;Rational Team Concert;Rational Quality Manager

Document Information

Modified date:
15 July 2020

UID

swg21607533