IBM Support

MustGather: Performance, hang, or high CPU issues with WebSphere Application Server on Linux

Troubleshooting


Problem

If you are experiencing performance, hang, or high CPU issues with WebSphere Application Server on Linux, this MustGather will assist you in collecting the data necessary to diagnose and resolve the issue.

Resolving The Problem

If you are experiencing performance, hang, or high CPU issues with WebSphere Application Server on Linux, this MustGather will assist you in collecting the data necessary to diagnose and resolve the issue. There are two scripts that can be used to collect the performance diagnostic information Please expand the following section and download one of the scripts and use it to collect information during the problem

Collecting data

Complete the following three steps:

(1) Collecting the required data:

a) If you have not already done so, enable verboseGC and restart the problematic server(s).
b) Download linperf.sh from this technote:  
This script will create a file named linperf_RESULTS.tar.gz and and ten to twelve javacore text files. This script should be executed as the root user. As with any script, you may need to add execute permissions before executing the script (chmod).
c) At the time of the problem, run the script using the process ID (pid) number(s).  Examples:
If your current process ID (pid) is 23456, then run the command:
./linperf.sh 23456
If you have multiple pids that you want to collect data for, then separate them with a space:
./linperf.sh 23456 9876 4567
** Note: The script will NOT work using the server name, it must be the current process ID (pid) number.  

(2) Collecting log files:
Collect the server logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log) from the problematic server(s):
a) profile_root/logs/server_name/*
b) If you have verbosegc redirected to other files, please send those as well
(3) Submitting required data:

Zip/Tar all the files gathered:
  • linperf_RESULTS.tar.gz
  • All javacores (10 to 12) created by linperf.sh
  • Server logs (SystemOut.log, SystemErr.log, native_stderr.log, native_stdout.log, verbosegc)
  • server.xml for the server(s) that you are providing data for
  • /var/log/messages (Linux OS files)
    ** Note: The javacores are NOT packed up in the linperf_RESULTS.tar.gz file so you need to zip those up and send them

Send the results to IBM Support: "Exchanging information with IBM Support"

Frequently Asked Questions:
  • What is the impact of enabling verboseGC?
    verboseGC Data is critical to diagnosing these issues. This can be enabled on production systems because it has a negligible impact on performance (< 2%).

    What is the linperf_RESULTS.tar.gz file and where can I find it?
    The linperf_RESULTS.tar.gz file is created while running the linperf.sh script and contains output from the commands called by the script. It will be created in the directory from which you execute the script.

    What are 'javacores' and where do I find them?
    javacores are snapshots of the JVM activity and are essential to troubleshooting these issues. These files will usually be found in the <profile_root>. If you don't find the files here, you can search your entire system for them using the following command:

    find / -name "*javacore*"
 
  • Monitor process sizes and paging usage (Only if asked to do so)
    • The linmon.sh script will collect data every 5 minutes until it is stopped manually. Run the following command before the issue occurs to start the script:

      ./linmon.sh

      This will create two files: ps_mon.out and vmstat_mon.out.

Exchanging data with IBM Support

To diagnose or identify a problem, it is sometimes necessary to provide Technical Support with data and information from your system. In addition, Technical Support might also need to provide you with tools or utilities to be used in problem determination. You can submit files using one of following methods to help speed problem diagnosis:

 

Previous linperf.sh version:
2019_linperf.sh


Read first and related MustGathers

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Hangs\/Performance Degradation","Platform":[{"code":"PF016","label":"Linux"}],"Version":"9.0.0.0;8.5;8.0;7.0;6.1","Edition":"Base;Express;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SS7JFU","label":"WebSphere Application Server - Express"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":" ","Platform":[{"code":"PF016","label":"Linux"}],"Version":"8.5;8.0;7.0;6.1;6.0;5.1","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Business Unit":{"code":"BU053","label":"Cloud \u0026 Data Platform"},"Component":"Java SDK","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
17 November 2021

UID

swg21115785