Generating GPFS trace reports

Use the mmtracectl command to configure trace-related configuration variables and to start and stop the trace facility on any range of nodes in the GPFS™ cluster.

To configure and use the trace properly:

Issue the mmlsconfig dataStructureDump command to verify that a directory for dumps was created when the cluster was configured. The default location for trace and problem determination data is /tmp/mmfs. Use mmtracectl, as instructed by the IBM® Support Center, to set trace configuration parameters as required if the default parameters are insufficient. For example, if the problem results in GPFS shutting down, set the traceRecycle variable with --trace-recycle as described in the mmtracectl command in order to ensure that GPFS traces are performed at the time the error occurs.
If desired, specify another location for trace and problem determination data by issuing this command:
```
mmchconfig dataStructureDump=path_for_storage_of_dumps
```
To start the tracing facility on all nodes, issue this command:
```
mmtracectl --start
```
Re-create the problem.
When the event to be captured occurs, stop the trace as soon as possible by issuing this command:
```
mmtracectl --stop
```
The output of the GPFS trace facility is stored in /tmp/mmfs, unless the location was changed using the mmchconfig command in Step 1. Save this output.
If the problem results in a shutdown and restart of the GPFS daemon, set the traceRecycle variable as necessary to start tracing automatically on daemon startup and stop the trace automatically on daemon shutdown.

If the problem requires more detailed tracing, the IBM Support Center might ask you to modify the GPFS trace levels. Use the mmtracectl command to establish the required trace classes and levels of tracing. The syntax to modify trace classes and levels is as follows:

mmtracectl --set --trace={io | all | def | "Class Level [Class Level ...]"}

For example, to tailor the trace level for I/O, issue the following command:

mmtracectl --set --trace=io

Once the trace levels are established, start the tracing by issuing:

mmtracectl --start

After the trace data has been gathered, stop the tracing by issuing:

mmtracectl --stop

To clear the trace settings and make sure tracing is turned off, issue:

mmtracectl --off

Other possible values that can be specified for the trace Class include:

afm: active file management
alloc: disk space allocation
allocmgr: allocation manager
basic: 'basic' classes
brl: byte range locks
cksum: checksum services
cleanup: cleanup routines
cmd: ts commands
defrag: defragmentation
dentry: dentry operations
dentryexit: daemon routine entry/exit
disk: physical disk I/O
disklease: disk lease
dmapi: Data Management API
ds: data shipping
errlog: error logging
eventsExporter: events exporter
file: file operations
fs: file system
fsck: online multinode fsck
ialloc: inode allocation
io: physical I/O
kentryexit: kernel routine entry/exit
kernel: kernel operations
klockl: low-level vfs locking
ksvfs: generic kernel vfs information
lock: interprocess locking
log: recovery log
malloc: malloc and free in shared segment
mb: mailbox message handling
mmpmon: mmpmon command
mnode: mnode operations
msg: call to routines in SharkMsg.h
mutex: mutexes and condition variables
nsd: network shared disk
perfmon: performance monitors
pgalloc: page allocator tracing
pin: pinning to real memory
pit: parallel inode tracing
quota: quota management
rdma: rdma
sanergy: SANergy
scsi: scsi services
sec: cluster security
shared: shared segments
smb: SMB locks
sp: SP message handling
super: super_operations
tasking: tasking system but not Thread operations
thread: operations in Thread class
tm: token manager
ts: daemon specific code
user1: miscellaneous tracing and debugging
user2: miscellaneous tracing and debugging
vbhvl: behaviorals
vnode: vnode layer of VFS kernel support
vnop: one line per VNOP with all important information

Values that can be specified for the trace Class, relating to vdisks, include:

vdb: vdisk debugger
vdisk: vdisk
vhosp: vdisk hospital

For more information about vdisks and IBM Spectrum Scale™ RAID, see IBM Spectrum Scale RAID: Administration.

The trace Level can be set to a value from 0 through 14, which represents an increasing level of detail. A value of 0 turns tracing off. To display the trace level in use, issue the mmfsadm showtrace command.

On AIX®, the –aix-trace-buffer-size option can be used to control the size of the trace buffer in memory.

On Linux nodes only, use the mmtracectl command to change the following:

The trace buffer size in blocking mode.
For example, to set the trace buffer size in blocking mode to 8K, issue:
```
mmtracectl --set --tracedev-buffer-size=8K
```
The raw data compression level.
For example, to set the trace raw data compression level to the best ratio, issue:
```
mmtracectl --set --tracedev-compression-level=9
```
The trace buffer size in overwrite mode.
For example, to set the trace buffer size in overwrite mode to 500M, issue:
```
mmtracectl --set --tracedev-overwrite-buffer-size=500M
```
When to overwrite the old data.
For example, to wait to overwrite the data until the trace data is written to the local disk and the buffer is available again, issue:
```
mmtracectl --set --tracedev-write-mode=blocking
```
--tracedev-write-mode=blocking specifies that if the trace buffer is full, wait until the trace data is written to the local disk and the buffer becomes available again to overwrite the old data. This is the default. --tracedev-write-mode=overwrite specifies that if the trace buffer is full, overwrite the old data.

Note: Before switching between --tracedev-write-mode=overwrite and --tracedev-write-mode=blocking, or vice versa, run the mmtracectl --stop command first. Next, run the mmtracectl --set --tracedev-write-mode command to switch to the desired mode. Finally, restart tracing with the mmtracectl --start command.