Generating GPFS trace reports
Use the mmtracectl command to configure trace-related configuration variables and to start and stop the trace facility on any range of nodes in the GPFS cluster.
- Issue the
mmlsconfig dataStructureDump command to
verify that a directory for dumps was created when the cluster was
configured. The default location for trace and problem determination
data is /tmp/mmfs. Use mmtracectl,
as instructed by the IBM® Support
Center, to set trace configuration parameters as required if the default
parameters are insufficient. For example, if the problem results in GPFS shutting down, set the traceRecycle variable
with --trace-recycle as described in the mmtracectl command
in order to ensure that GPFS traces
are performed at the time the error occurs. If desired, specify another location for trace and problem determination data by issuing this command:
mmchconfig dataStructureDump=path_for_storage_of_dumps
- To start the tracing facility on all nodes, issue this command:
mmtracectl --start
- Re-create the problem.
- When the event to be captured occurs, stop the trace as soon as
possible by issuing this command:
mmtracectl --stop
- The output of the GPFS trace facility is stored in /tmp/mmfs, unless the location was changed using the mmchconfig command in Step 1. Save this output.
- If the problem results in a shutdown and restart of the GPFS daemon, set the traceRecycle variable as necessary to start tracing automatically on daemon startup and stop the trace automatically on daemon shutdown.
mmtracectl --set --trace={io | all | def | "Class Level [Class Level ...]"}
mmtracectl --set --trace=io
mmtracectl --start
mmtracectl --stop
mmtracectl --off
- afm
- active file management
- alloc
- disk space allocation
- allocmgr
- allocation manager
- basic
- 'basic' classes
- brl
- byte range locks
- ccr
- cluster configuration repository
- cksum
- checksum services
- cleanup
- cleanup routines
- cmd
- ts commands
- defrag
- defragmentation
- dentry
- dentry operations
- dentryexit
- daemon routine entry/exit
- disk
- physical disk I/O
- disklease
- disk lease
- dmapi
- Data Management API
- ds
- data shipping
- errlog
- error logging
- eventsExporter
- events exporter
- file
- file operations
- fs
- file system
- fsck
- online multinode fsck
- ialloc
- inode allocation
- io
- physical I/O
- kentryexit
- kernel routine entry/exit
- kernel
- kernel operations
- klockl
- low-level vfs locking
- ksvfs
- generic kernel vfs information
- lock
- interprocess locking
- log
- recovery log
- malloc
- malloc and free in shared segment
- mb
- mailbox message handling
- mmpmon
- mmpmon command
- mnode
- mnode operations
- msg
- call to routines in SharkMsg.h
- mutex
- mutexes and condition variables
- nsd
- network shared disk
- perfmon
- performance monitors
- pgalloc
- page allocator tracing
- pin
- pinning to real memory
- pit
- parallel inode tracing
- quota
- quota management
- rdma
- rdma
- sanergy
- SANergy
- scsi
- scsi services
- sec
- cluster security
- shared
- shared segments
- smb
- SMB locks
- sp
- SP message handling
- super
- super_operations
- tasking
- tasking system but not Thread operations
- thread
- operations in Thread class
- tm
- token manager
- ts
- daemon specific code
- user1
- miscellaneous tracing and debugging
- user2
- miscellaneous tracing and debugging
- vbhvl
- behaviorals
- vnode
- vnode layer of VFS kernel support
- vnop
- one line per VNOP with all important information
- vdb
- vdisk debugger
- vdisk
- vdisk
- vhosp
- vdisk hospital
The trace Level can be set to a value from 0 through 14, which represents an increasing level of detail. A value of 0 turns tracing off. To display the trace level in use, issue the mmfsadm showtrace command.
On AIX®, the –aix-trace-buffer-size option can be used to control the size of the trace buffer in memory.
- The trace buffer size in blocking mode. For example, to set the trace buffer size in blocking mode to 8K, issue:
mmtracectl --set --tracedev-buffer-size=8K
- The raw data compression level. For example, to set the trace raw data compression level to the best ratio, issue:
mmtracectl --set --tracedev-compression-level=9
- The trace buffer size in overwrite mode. For example, to set the trace buffer size in overwrite mode to 500M, issue:
mmtracectl --set --tracedev-overwrite-buffer-size=500M
- When to overwrite the old data. For example, to wait to overwrite the data until the trace data is written to the local disk and the buffer is available again, issue:
mmtracectl --set --tracedev-write-mode=blocking
--tracedev-write-mode=blocking
specifies that if the trace buffer is full, wait until the trace data is written to the local disk and the buffer becomes available again to overwrite the old data. This is the default.--tracedev-write-mode=overwrite
specifies that if the trace buffer is full, overwrite the old data.Note: Before switching between --tracedev-write-mode=overwrite and --tracedev-write-mode=blocking, or vice versa, run the mmtracectl --stop command first. Next, run the mmtracectl --set --tracedev-write-mode command to switch to the desired mode. Finally, restart tracing with the mmtracectl --start command.