Resolving The Problem
System core dump files should generate in WebSphere Application Server during a crash, or if manually triggered, and in some OutOfMemory instances. A good system core dump is needed to diagnose crashes, some OutOfMemory issues, and some other issues as needed. A few conditions can cause the core dumps to be truncated and unusable. You may need to have your Linux OS Administrator make these changes for you.
NOTE: There is a different technote that discusses issues where the process does not record a crash event.
1. SET ULIMITS
See Also: Guidelines for Setting Ulimits
The ulimits for core (-c) and fsize (-f) need to be tuned so that the hard and soft limits are set to unlimited. This may require root access to change.
Global settings are set in the file /etc/security/limits.conf.
The format for setting each limit is as follows:
<domain> <type> <item> <value>
<domain> controls which users or groups will have these limits
<type> is either the string "soft" or "hard" limits.
The hyphen "-" can also be used which represents both soft and hard limits
Example: wasadmin soft core unlimited
2. DISK SPACE
Check your partitions where WebSphere Application Server resides and make sure there is enough space for the dump to be produced. Usually an error message will be seen in the native_stderr.log that indicates if the core was unable to be written.
To check all of your partitions, execute this command (the -k is for kilobytes):
3. CORE PATTERN CONFIGURATION
In some cases (which has been seen with -Xdisableexplicitgc configured), the core_pattern setting may have extra options added to its configuration which may need to be removed, such as:
/proc/sys/kernel/core_pattern = core
** "core" with no other options
4. DISABLE SIGNAL HANDLERS
To force the operating system to handle all signals sent to the JVM process, you can disable all JVM signal handlers.
For IBM SDK 5.0 and later, set this JVM argument:
NOTE: On SDK 6.0, to prevent unintentional crashes due to SIGTRAP, clear the shared class cache by executing <WAS_HOME>/bin/clearClassCache.sh
What happens if I do not have write permission in the profile's root directory, or the directory I am redirecting javacores, heapdumps, and system core files to?
This will result in a failure when writing these files to the system. Check for an error in the native_stderr.log, as it may try to write the dump to an alternate folder (such as /tmp).
Even with all ulimit settings set to unlimited, core files are truncated at 2GB?
There is a limitation on 32-bit processes which can be worked around if you enable large file support..
Using a 64-bit version of WebSphere Application Server also resolves this limitation, although if you run out of disk space the dump can still be truncated.
Can I test my configuration to see if a core can be generated?
Yes. The preferred way is to generate the core via the Admin Console. See "Collecting Java dumps and core files using the administrative console"
kill -6 PID
kill -11 PID
An alternative is to use the gcore command. This produces a core file and keeps the process running.
30 November 2020