MustGather information for Linux

When a problem occurs, the more information known about the state of the system environment, the easier it is to reach a diagnosis.

The IBM® Support Assistant Data Collector is the recommended utility for collecting Java™ diagnostics files for a problem event. The IBM Support Assistant Data Collector collects diagnostic files such as dumps and log files, and helps you to send the information to IBM, if required. For more information, see Using diagnostic tools.

The following sections tell you about the data you can collect to help the IBM service team for Java solve the problem.

Producing system dumps

You can use the -Xdump:system command-line option to obtain system dumps based on a trigger. See Using dump agents for more information.

You can also use a Linux® system utility to generate system dumps:
  1. Determine the Process ID of your application using the ps command. See The ps command.
  2. At a shell prompt, type gcore -o <dump file name> <pid>

A system dump file is produced for your application. The application is suspended while the system dump is written.

Collecting system dumps (core files)

Collect system dumps to help diagnose many types of problem. Process the system dump using the jextract utility to generate a file for service (see Dump extractor (jextract)).
Note: The IBM Support Assistant Data Collector runs the jextract utility on system dumps as part of its collection process.

Producing Javadumps

In some conditions, like a crash, a Javadump is produced, usually in the current directory.

In other conditions, like a hang, you might need to prompt the JVM to generate a javadump by sending the JVM a SIGQUIT symbol:
  1. Determine the Process ID of your application using the ps command. See The ps command.
  2. At a shell prompt, type kill -QUIT <pid>

For more information about this process, see Using Javadumps in the J9 VM reference.

Producing Heapdumps

The JVM can generate a Heapdump at the request of the user, for example by calling com.ibm.jvm.Dump.HeapDump() from inside the application, or by default when the JVM terminates because of an OutOfMemoryError. You can specify finer control of the timing of a Heapdump with the -Xdump:heap option. For example, you can request a heapdump after a certain number of full garbage collections have occurred. The default heapdump format (phd files) is not human-readable and you process it using available tools such as Heaproots. For more information, see Using Heapdumps in the J9 VM reference.

Producing Snap traces

Under default conditions, a running JVM collects a small amount of trace data in a special wraparound buffer. This data is dumped to file when the JVM terminates unexpectedly or an OutOfMemoryError occurs. You can use the -Xdump:snap option to vary the events that cause a snap trace to be produced. The snap trace is in normal trace file format and requires the use of the supplied standard trace formatter so that you can read it. See Snap traces for more information about the contents and control of snap traces.

Producing JIT dumps

A general protection fault (GPF) or abort event generates a small binary dump of JIT diagnostic data. For more information, see JIT dumps.

Using system logs

The kernel logs system messages and warnings. The system log is located in the /var/log/messages file. Use it to observe the actions that led to a particular problem or event. The system log can also help you determine the state of a system. Other system logs are in the /var/log directory.

Determining the operating environment

This section looks at the commands that can be useful to determine the operating environment of a process at various stages of its life cycle.

uname -a
Displays operating system and hardware information.
df
Displays free disk space on a system.
free
Displays memory use information.
ps -ef
Displays a full process list.
lsof
Displays open file handles.
top
Displays process information (such as processor, memory, states) sorted by default by processor usage.
vmstat
Displays general memory and paging information.

The uname, df, and free output is the most useful. The other commands can be run before and after a crash, or during a hang, to determine the state of a process. These commands can provide useful diagnostic information.

Sending information to Java Support

When you have collected the output of the commands listed in the previous section, put that output into files.

Compress the files before sending them to Java Support, because the files could be large. You should compress the files at a very high ratio.

The following command builds an archive from files {file1,..,fileN} and compresses them to a file with a name in the format filename.tgz:

tar czf filename.tgz file1 file2...filen

Collecting additional diagnostic data

Depending on the type of problem, the following data can also help you diagnose problems. The information available depends on the way in which Java is started and also the system environment. If these debugging aids are not active, you must change your configuration and restart Java to reproduce the problem.

/proc file system

The /proc file system gives direct access to kernel level information. The /proc/<pid> directory contains detailed diagnostic information about the process with PID (process ID) <pid>, where <pid> is the ID of the process.

The command cat /proc/<pid>/maps lists memory segments (including native heap) for a given process.

strace and ltrace

Use the commands strace and ltrace to collect further diagnostic data. See Tracing tools.