GPFS logs

The GPFS™ log is a repository of error conditions that are detected on each node, as well as operational events such as file system mounts. The GPFS log is the first place to look when you start debugging the abnormal events. As GPFS is a cluster file system, events that occur on one node might affect system behavior on other nodes, and all GPFS logs can have relevant data.

The GPFS log can be found in the /var/adm/ras directory on each node. The GPFS log file is named mmfs.log.date.nodeName, where date is the time stamp when the instance of GPFS started on the node and nodeName is the name of the node. The latest GPFS log file can be found by using the symbolic file name /var/adm/ras/mmfs.log.latest.

The GPFS log from the prior startup of GPFS can be found by using the symbolic file name /var/adm/ras/mmfs.log.previous. All other files have a time stamp and node name appended to the file name.

At GPFS startup, log files that are not accessed during the last 10 days are deleted. If you want to save old log files, copy them elsewhere.

Many GPFS log messages can be sent to syslog on Linux. The systemLogLevel attribute of the mmchconfig command determines the GPFS log messages to be sent to the syslog. For more information on the mmchconfig command, see mmchconfig command.

This example shows normal operational messages that appear in the GPFS log file on Linux node:

Removing old /var/adm/ras/mmfs.log.* files:
Unloading modules from /lib/modules/3.0.13-0.27-default/extra
Unloading module tracedev
Loading modules from /lib/modules/3.0.13-0.27-default/extra
Module                  Size  Used by
mmfs26               2155186  0
mmfslinux             379348  1 mmfs26
tracedev               48513  2 mmfs26,mmfslinux
Tue Oct 27 11:45:47.149 2015: [I] mmfsd initializing. {Version: 4.2.0.0 Built: Oct 26 2015 15:19:01} ...
Tue Oct 27 11:45:47.150 2015: [I] Tracing in blocking mode
Tue Oct 27 11:45:47.151 2015: [I] Cleaning old shared memory ...
Tue Oct 27 11:45:47.152 2015: [I] First pass parsing mmfs.cfg ...
Tue Oct 27 11:45:47.153 2015: [I] Enabled automated deadlock detection.
Tue Oct 27 11:45:47.154 2015: [I] Enabled automated deadlock debug data collection.
Tue Oct 27 11:45:47.155 2015: [I] Enabled automated expel debug data collection.
Tue Oct 27 11:45:47.156 2015: [I] Initializing the main process ...
Tue Oct 27 11:45:47.169 2015: [I] Second pass parsing mmfs.cfg ...
Tue Oct 27 11:45:47.170 2015: [I] Initializing the page pool ...
Tue Oct 27 11:45:47.500 2015: [I] Initializing the mailbox message system ...
Tue Oct 27 11:45:47.521 2015: [I] Initializing encryption ...
Tue Oct 27 11:45:47.522 2015: [I] Encryption: loaded crypto library: IBM 
CryptoLite for C v4.10.1.5600 (c4T3/GPFSLNXPPC64).
Tue Oct 27 11:45:47.523 2015: [I] Initializing the thread system ...
Tue Oct 27 11:45:47.524 2015: [I] Creating threads ...
Tue Oct 27 11:45:47.529 2015: [I] Initializing inter-node communication ...
Tue Oct 27 11:45:47.530 2015: [I] Creating the main SDR server object ...
Tue Oct 27 11:45:47.531 2015: [I] Initializing the sdrServ library ...
Tue Oct 27 11:45:47.532 2015: [I] Initializing the ccrServ library ...
Tue Oct 27 11:45:47.538 2015: [I] Initializing the cluster manager ...
Tue Oct 27 11:45:48.813 2015: [I] Initializing the token manager ...
Tue Oct 27 11:45:48.819 2015: [I] Initializing network shared disks ...
Tue Oct 27 11:45:51.126 2015: [I] Start the ccrServ ...
Tue Oct 27 11:45:51.879 2015: [N] Connecting to 192.168.115.171 js21n07 <c0p1>
Tue Oct 27 11:45:51.880 2015: [I] Connected to 192.168.115.171 js21n07 <c0p1>
Tue Oct 27 11:45:51.897 2015: [I] Node 192.168.115.171 (js21n07) is now the Group Leader.
Tue Oct 27 11:45:51.911 2015: [N] mmfsd ready
Tue Oct 27 11:45:52 EDT 2015: mmcommon mmfsup invoked. 
Parameters: 192.168.115.220 192.168.115.171 all

The mmcommon logRotate command can be used to rotate the GPFS log without shutting down and restarting the daemon. After the mmcommon logRotate command is issued, /var/adm/ras/mmfs.log.previous will contain the messages that occurred since the previous startup of GPFS or the last run of mmcommon logRotate. The /var/adm/ras/mmfs.log.latest file starts over at the point in time that mmcommon logRotate was run.

Depending on the size and complexity of your system configuration, the amount of time to start GPFS varies. If you cannot access a file system that is mounted, examine the log file for error messages.