IBM Support

WebSphere Application Server process exits without leaving a footprint in log files or no core dumps

Troubleshooting


Problem

When running WebSphere Application Server, a Java™ process exits without producing a system core or user dump and no entries are seen in the SystemOut.log in the logs.

Cause

There could be several causes of a Java process exiting without a footprint (no core file produced on UNIX or no user.dmp file produced on Windows) :
  1. The nodeagent has restarted the unresponsive server
  2. Java Native Interface (JNI) code has called JVM to shutdown
  3. Some code within the Java process has called System.exit() or Runtime.exit()
  4. Some external process has triggered a signal that will kill the Java process (SIGTERM, oom-killer, etc)

Resolving The Problem

Follow the points below to debug the problem when you suspect that the Java process has exited without leaving any footprint.

Follow the steps in order, however specific OS steps are located at the bottom of the document.
  1. Make sure that the operating system wasn't unintentionally restarted
    Or if a user who owns the running application server process had logged-off the system

    Restarting the operating system or logging off will cause any processes owned by that user to halt operation immediately (services will be gracefully stopped if configured as such). This will make it seem that a crash has occurred without a footprint.

    Check the audit/event logs for any logoff or restart activity. If using a virtual machine, check the audit logs for restart activity.

    If running on Windows, see the note at the bottom of the document about Windows Services.

    If #1 is ruled out, continue to the next steps.

     
  2. Check to see if generic JVM argument -Xrs is configured.
    This option allows the OS signal handlers to handle most process signals and may unintentionally terminate the JVM without a footprint. Removing the argument should return the JVM to normal operation.

     
  3. Ensure that the nodeagent has not restarted the unresponsive server.
    This is only applicable for Network Deployment installations.

    Per the default monitoring policy, the nodeagent will restart any unresponsive server. This may look as if the Java process exited without leaving any footprint. Check the nodeagent logs and see if it has restarted the server.

    You can check the monitoring policy setting through an administrative console page, click Servers > Application Servers > server_name. Then, under Server Infrastructure, click Java and Process Management > Monitoring Policy.

     
  4. Check if application code has called System.exit()
    Or if application code is triggering a graceful shutdown of the JVM.

    When System.exit() (or similarly Runtime.exit()) is called, this immediately terminates the JVM, without logging what had happened. This will make it seem like a system crash, but without the core dump or javacores that would be associated with such an event. Steps below go into how to trigger javacores on calls to System.exit().

    NOTE: If using a network deployment edition of WebSphere Application Server, the nodeagent may restart the server automatically.

    The next substeps goes into how the SDK can report an unintentional shutdown of the JVM.

     
    • Enable dumps to be generated upon server exit/shutdown. There are several ways to enable this as each step below will have slightly different results.
      1. Enable a Generic JVM argument to trigger thread dumps (javacores) on shutdown. It will also log the incident in the SystemErr.log with the current thread that initiated the shutdown, as usually a shutdown hook message will be seen, sometimes with a thread stack:

        EXAMPLE MESSAGE:
        [3/14/15 03:14:15:926 MST] 000000ab SystemErr     R java.lang.Exception: Thread[Thread-3141,5,main]: shutdown hook stack trace:
        -Dcom.ibm.ws.runtime.dumpShutdown=true
         
      2. On AIX, Linux, and Windows, use the IBM SDK’s Xtrace mechanism to trigger a javadump on System.exit() (may optionally add another argument for Runtime.exit())
        -Xtrace:trigger=method{java/lang/System.exit,javadump}

        ** The above two options will generate a javacore whenever the Java Virtual Machine stops.  We are only interested in javacores if they are created by an unexpected stop of the server.
      3. Check if Java Native Interface (JNI) code have called JVM to shutdown (Only if requested by support) 
        Verbose JNI trace reports information about use of native methods and other Java Native Interface activity. If the application's native code made a call to "JNI_DestroyJavaVM" or System.exit(), it will be logged.

        The verbose JNI can log all the interactions between the native code and Java. However, if there are issues in the native code, JNI trace will not be all that helpful in this particular case.
        Follow the steps below to enable verbose JNI :
        a) Select Servers
        b) Select Server Types.
        c) Select WebSphere Application Servers > name_of_appserver> Process Definition.
        d) Select Java Virtual Machine.
        e) Place a checkmark next to Verbose JNI, then click apply.
        f) At the top of the administrative client, click Save to save the changes.
        g) Restart the application server.
        The verbose JNI output will normally be logged in the native_stderr.log file for the application server, and may produce a lot of output. Clearing out the logs is recommended before you start the server with this setting in place.
         
      4. HP-UX
        TUSC
        This is a system trace tool usually available on HP-UX systems similar to truss. The following command below should output the system calls and signals along with timestamps.
        tusc -T "%Y-%m-%dT%H:%M:%S" -f -o /tmp/tusc/output/folder/tusc.out PID
        Note: A more verbose output is available with the -v option.
         
      5. LINUX
        OOM KILLER
        The OOM killer can terminate a process without a footprint. This mechanism will normally kill unimportant processes if the system encounters an Out of Memory event, but may end up terminating the application server JVM.  A system administrator can retrieve the /var/log/messages file to review for evidence of such an event.
        EXAMPLE MESSAGES:
        Mar 14 03:14:15 xxxxx kernel: yyyyy invoked oom-killer: ...
        Mar 14 03:14:15 xxxxx kernel: Out of Memory: Killed process 31415

        STRACE
        Collect strace on a running process. This will produce an output file in which the strace data will be stored. The command allows entry of a specific directory where the strace output will be generated:
        strace -tt -f -o /tmp/strace/output/folder/strace.out -p PID
        ** We will engage our Java Development team to analyze strace output
      6. SOLARIS
        TRUSS
        This utility can track the signals the application server process receives. You can attach truss to an already running process. Start with the command below which will output data with date stamps, and track forked processes.
        truss -d -f -o /tmp/truss/output/folder/truss.out -p PID

        STRACE
        A version of strace is available for Solaris, but it may not be present on all systems. You can run this against an already running process, and this will produce an output file in which the strace data will be stored. The command allows entry of a specific directory where the strace output will be generated.
        strace -tt -f -o /tmp/strace/output/folder/strace.out -p PID

         
      7. WINDOWS
Submitting required data
Gather the following files:
  • Server logs and native logs (SystemOut.log, SystemErr.log, native_stdout.log, native_stderr.log)
  • All javacore.*.txt files
  • Collect all strace/truss or related data produced and engage the appropriate OS Support team
  • /var/log/messages (UNIX/Linux Only)
Note: You can archive or compress the preceding files.

Send the results to IBM support.

 

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Java SDK","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.5;8.0;7.0","Edition":"Base;Express;Liberty;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Java SDK","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
01 August 2023

UID

swg21144595

Page Feedback