Troubleshooting
Problem
The WebSphere Application Server JVM process terminated without warning or a system core is generated. The following MustGather will assist you in collecting the critical data to troubleshoot issues with an IBM WebSphere Application Server JVM crash on the IBM i platform.
Symptom
High-level symptoms include:
- The application appears to be hung or not available, and the application is not responding to incoming requests.
- The performance of the application is degrading.
- System core generated, JVM process disappears or terminates.
- The application appears to be hung or not available, and the application is not responding to incoming requests.
- The performance of the application is degrading.
- System core generated, JVM process disappears or terminates.
Cause
By "crash" we mean that the Application Server has not been shutdown or restarted by Administrative action, but the Process ID (PID) of the Application Server does not exist in the process table (ps -ef) or has changed to a different number than it was before the problem happened. A "crash" is when the Application Server process on which the Java Virtual Machine (JVM) runs enters an illegal state, and is terminated by the operating system. The IBM i operating system will usually produce a "core" file, which is an image of the running process at the time that it crashed.
Diagnosing The Problem
In most cases a non-truncated core file is necessary to determine the cause of a crash with any certainty. A "core" file is not the same as a "javacore" file. Every system core file must be processed by the jextract utility on the same partition where it was created so that it can be analyzed by IBM.
Resolving The Problem
WebSphere Application Server Crash specific MustGather information:
For all Java Virtual Machine (JVM) crash issues, collect the following information. There is a list of links provided at the bottom of this document that routes you to the how-to documents for steps that require additional instructions.
Collect the following information when the application server process (job) is experiencing the problem.
Note: If you are running the Network Deployment (ND) edition of WebSphere Application Server, please complete the following steps for the Deployment Manager AND the Federated Node experiencing the problem.
Multiple JVMs are available on IBM i 5.4 and higher. The original JVM is known as the "Classic JVM." The new JVM is called "IBM Technology for Java." The approach to gather data is different for each JVM, so you'll need to figure out which JVM is in use before you proceed. The easiest way to find this is by displaying the job log for the job experiencing the problem.
For example, if the server is called 'server1'; I would execute the following from the CL command line:
WRKJOB SERVER1 <enter>
Choose option 1 to select the active job.
Choose option 10 to display the job log.
Press F10 to see all messages
The joblog will show:
"Java Virtual Machine is IBM Technology for Java. PID(xxx)"
or
"Java Virtual Machine is Classic."
Appendix B: How to send diagnostic information to IBM support
For all Java Virtual Machine (JVM) crash issues, collect the following information. There is a list of links provided at the bottom of this document that routes you to the how-to documents for steps that require additional instructions.
Collect the following information when the application server process (job) is experiencing the problem.
Note: If you are running the Network Deployment (ND) edition of WebSphere Application Server, please complete the following steps for the Deployment Manager AND the Federated Node experiencing the problem.
- Execute the following commands to collect system information to be sent to IBM:
- WRKPTFGRP (use F6 to print or send a screen shot showing the installed status of PTFs)
- DSPPTF OUTPUT(*PRINT)
- WRKSYSSTS OUTPUT(*PRINT)
- WRKSBS OUTPUT(*PRINT)
- WRKSHRPOOL OUTPUT(*PRINT)
- WRKACTJOB OUTPUT(*PRINT)
- Gather the System core, javacore, heapdump, and Snap dumps produced by the JVM during the crash using the following Appendix. Note: The jextract utility must be ran against the core.xxx.dmp file on the same box that created the System core dump. You will then upload the zip file produced from the utility.
- "Appendix A: How to collect JVM dump files for a JVM experiencing a crash."
- Collector tool output
- Follow instructions to send diagnostic information to IBM support. "Appendix B: How to send diagnostic information to IBM support."
Multiple JVMs are available on IBM i 5.4 and higher. The original JVM is known as the "Classic JVM." The new JVM is called "IBM Technology for Java." The approach to gather data is different for each JVM, so you'll need to figure out which JVM is in use before you proceed. The easiest way to find this is by displaying the job log for the job experiencing the problem.
For example, if the server is called 'server1'; I would execute the following from the CL command line:
WRKJOB SERVER1 <enter>
Choose option 1 to select the active job.
Choose option 10 to display the job log.
Press F10 to see all messages
The joblog will show:
"Java Virtual Machine is IBM Technology for Java. PID(xxx)"
or
"Java Virtual Machine is Classic."
Java Virtual Machine is IBM Technology for Java
- On the operating system command line, type the following:
| STRSST Press the Enter key. |
|
| Sign on SST with the QSECOFR or 22222222 profile and the corresponding password. | |
| From the SST main menu, select Option 1, Start a Service Tool. | |
| Select Option 5, Licensed Internal Code log. | |
| Select Option 1, Select entries from the Licensed Internal Code (LIC) log. | |
| At the screen titled Specify Licensed Internal Code Log Selection Values, pressing the Enter key will display the most recent VLOG entries. However, VLOGs can also be displayed by date range, type, or entry ID. Enter a major code of 4700 to search for just the i5/OS PASE VLOG entries. |
- Locate and gather all dumps produced by the JVM when it experienced the crash. This could include a javacore, core, and Snap dump. Note: The dumps produced by the JVM should already exist. The default location where you can find the System core dump are listed below. The filenames of the three dumps will be in the form of "javacore.xxx.txt", "core.xxx.dmp", and "Snap.xxx.trc".
- WebSphere Application Server all versions, all editions: /QIBM/UserData/Websphere/AppServer/<version>/<edition>/profiles/<profileName>/
- If you need to gather the one or more of the dumps manually, please refer to the following technical document: IBM Technology for Java (J9) Serviceability - JVMINFO & How to gather System, Java, Heap, and GcCycle Dumps in IBM i 5.4 and later.
- Run the jextract utility against the System core dump file on the same box experiencing the Application Server crash. You do not need to run this utility against the other files. The jextract utility will collect the System Core dump file and other files used to analyze the JVM environment and archive them to a zip file. Please refer to the following technical document on how to run the jextract utility: IBM Technology for Java - jextract
- Gather the joblog of the WebSphere Application Server job. Use WRKJOB <jobname> to find the job. You might have to select job from a list, then enter 4 to Work with spool files. If you cannot find the job and/or spool file, then the message logging levels of the job might have not been set properly. Refer to Appendix B: How to send diagnostic information to IBM support.
- Look for and print any License Internal Code(LIC) Logs in the System Service Tools (SST) with a major code of 4700 (i5/OS PASE).
- To view and print the LIC logs, do the following:
- If there are any 4700 major code vlogs that occurred around the time of the crash, enter a 1 in the option column and press ENTER to print them to a spool file.
- The resulting file will be a QPCSMPRT spool file in the user's output queue.
Java Virtual Machine is Classic
| On the operating system command line, type the following: STRSST Press the Enter key. |
|
| Sign on SST with the QSECOFR or 22222222 profile and the corresponding password. | |
| From the SST main menu, select Option 1, Start a Service Tool. | |
| Select Option 5, Licensed Internal Code log. | |
| Select Option 1, Select entries from the Licensed Internal Code (LIC) log. | |
| At the screen titled Specify Licensed Internal Code Log Selection Values, pressing the Enter key will display the most recent VLOG entries. However, VLOGs can also be displayed by date range, type, or entry ID. Enter a major code of 4300 to search for just the Java Virtual Machine VLOG entries. |
Appendix B: How to send diagnostic information to IBM support
Java Virutal Machine is IBM Technology for Java
- Collect the <dumpfile>.zip file from the location specified in the output of the jextract utility.
- Collect all other JVM dump files (javacore, heapdump, Snap, etc.)
- Collect the Collector host_name-cell_name-node_name-profile_name.JAR output file. NOTE: The collector program writes its output JAR file to the current directory.
- Collect the WRKACTJOB, WRKPTFGRP, DSPPTF, WRKSYSSTS, WRKSBS, WRKSHRPOOL, Joblog, and Vlog spool files.
- To retrieve the file, open System i Navigator, expand your system, log in as the user profile who created the spool file, expand Basic Operations, and then left-click on Printer Output to display the output files. Locate the QPDSPAJB (WRKACTJOB), QSYSPRT (WRKPTFGRP), QSYSPRT (DSPPTF), QPDSPSTS (WRKSYSSTS), QPDSPSBS (WRKSBS), QSYSPRT (WRKSHRPOOL), QPJOBLOG (joblog), and QCPSMPRT (Vlog) spool files. Finally, drag and drop the files from System i Navigator to your Desktop. Plain text files will be created.
- Send the diagnostic information to IBM support using MustGather: Instructions for Sending Data to IBM i Support.
Java Virtual Machine is Classic
- Collect the Collector host_name-cell_name-node_name-profile_name.JAR output file. NOTE: The collector program writes its output JAR file to the current directory.
- Gather the joblog of the WebSphere Application Server job. Use WRKJOB <jobname> to find the job. You might have to select job from a list, then enter 4 to Work with spool files. If you cannot find the job and/or spool file, then the message logging levels of the job might have not been set properly. Refer to Appendix B: How to send diagnostic information to IBM support.
- Look for and print any License Internal Code(LIC) Logs in the System Service Tools (SST) with a major code of 4300 (Java Virtual Machine).
- To view and print the LIC logs, do the following:
- If there are any 4300 major code vlogs that occurred around the time of the crash, enter a 1 in the option column and press ENTER to print them to a spool file.
- The resulting file will be a QPCSMPRT spool file in the user's output queue.
- Collect the WRKACTJOB, WRKPTFGRP, DSPPTF, WRKSYSSTS, WRKSBS, WRKSHRPOOL, Joblog, and Vlog spool files.
- To retrieve the files, open IBM i Navigator, expand your system, log in as the user profile who created the spool file, expand Basic Operations, and then left-click on Printer Output to display the output files. Locate the QPDSPAJB (WRKACTJOB), QSYSPRT (WRKPTFGRP), QSYSPRT (DSPPTF), QPDSPSTS (WRKSYSSTS), QPDSPSBS (WRKSBS), QSYSPRT (WRKSHRPOOL), QPJOBLOG (joblog), and QPCSMPRT (Vlog) spool files. Finally, drag and drop the files from the System i Navigator to your Desktop. Plain text files will be created.
- Send the diagnostic information to IBM Support using MustGather: Instructions for Sending Data to IBM i Support
Related Information
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Component":"Crash","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"9.0.5;9.0;8.5.5;8.5;8.0;7.0;6.1","Edition":"Base;Express;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}}]
Was this topic helpful?
Document Information
Modified date:
04 February 2020
UID
swg21382901