Suboptimal performance when a tracing is active on a cluster

Tracing is usually enabled on the IBM Storage Scale cluster for troubleshooting purposes. However, running a trace on a node might cause performance degradation.

Problem identification

Issue the mmlsconfig command and verify whether GPFS tracing is configured. The following sample output displays a cluster in which tracing is configured:

# mmlsconfig | grep trace
trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3
tracedevOverwriteBufferSize 1073741824
tracedevWriteMode overwrite 268435456

Issue the # ps -aux | grep lxtrace | grep mmfs command to determine whether GPFS tracing process is running on a node. The following sample output shows that GPFS tracing process is running on the node:

# ps -aux | grep lxtrace | grep mmfs
root     19178  0.0  0.0  20536   128 ?        Ss   14:06   0:00 
/usr/lpp/mmfs/bin/lxtrace-3.10.0-229.el7.x86_64 on 
/tmp/mmfs/lxtrace.trc.c80f1m5n08ib0 --overwrite-mode --buffer-size 
268435456

Problem resolution and verification

When the traces have met their purpose and are no longer needed, use one of the following commands to stop the tracing on all nodes:

  • Use this command to stop tracing:

    mmtracectl --stop –N all

  • Use this command to clear all the trace setting variables and stop the tracing:

    mmtracectl --off –N all