NFS logs

The clustered export services (CES) NFS server writes log messages in the /var/log/ganesha.log file at runtime.

Operating system's log rotation facility is used to manage NFS logs. The NFS logs are configured and enabled during the NFS server packages installation.

The following example shows a sample log file:

# tail -f /var/log/ganesha.log
2015-05-31 17:08:04 : epoch 556b23d2 : cluster1.ibm.com : ganesha.nfsd-27204[main] nfs_rpc_cb_init_ccache 
:NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:0)
2015-05-31 17:08:04 : epoch 556b23d2 : cluster1.ibm.com : ganesha.nfsd-27204[main] nfs_Start_threads 
:THREAD :EVENT :Starting delayed executor.
2015-05-31 17:08:04 : epoch 556b23d2 : cluster1.ibm.com : ganesha.nfsd-27204[main] nfs_Start_threads 
:THREAD :EVENT :gsh_dbusthread was started successfully
2015-05-31 17:08:04 : epoch 556b23d2 : cluster1.ibm.com : ganesha.nfsd-27204[main] nfs_Start_threads 
:THREAD :EVENT :admin thread was started successfully
2015-05-31 17:08:04 : epoch 556b23d2 : cluster1.ibm.com : ganesha.nfsd-27204[main] nfs_Start_threads 
:THREAD :EVENT :reaper thread was started successfully
2015-05-31 17:08:04 : epoch 556b23d2 : cluster1.ibm.com : ganesha.nfsd-27204[main] nfs_Start_threads 
:THREAD :EVENT :General fridge was started successfully
2015-05-31 17:08:04 : epoch 556b23d2 : cluster1.ibm.com : ganesha.nfsd-27204[reaper] nfs_in_grace 
:STATE :EVENT :NFS Server Now IN GRACE
2015-05-31 17:08:04 : epoch 556b23d2 : cluster1.ibm.com : ganesha.nfsd-27204[main] nfs_start 
:NFS STARTUP :EVENT :-------------------------------------------------
2015-05-31 17:08:04 : epoch 556b23d2 : cluster1.ibm.com : ganesha.nfsd-27204[main] nfs_start 
:NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
2015-05-31 17:08:04 : epoch 556b23d2 : cluster1.ibm.com : ganesha.nfsd-27204[main] nfs_start 
:NFS STARTUP :EVENT :-------------------------------------------------
2015-05-31 17:09:04 : epoch 556b23d2 : cluster1.ibm.com : ganesha.nfsd-27204[reaper] nfs_in_grace 
:STATE :EVENT :NFS Server Now NOT IN GRACE

Log levels can be displayed by using the mmnfs configuration list | grep LOG_LEVEL command. For example:

mmnfs configuration list | grep LOG_LEVEL

The system displays output similar to the following example:

LOG_LEVEL: EVENT

By default, the log level is EVENT. Additionally, the following NFS log levels can also be used; starting from lowest to highest verbosity:

FATAL
MAJ
CRIT
WARN
INFO
DEBUG
MID_DEBUG
FULL_DEBUG
Note: The FULL_DEBUG level increases the size of the log file. Use it in the production mode only if instructed by the IBM® Support.

Increasing the verbosity of the NFS server log impacts the overall NFS I/O performance.

To change the logging to the verbose log level INFO, use the following command:

mmnfs configuration change LOG_LEVEL=INFO

The system displays output similar to the following example:

NFS Configuration successfully changed. NFS server restarted on all NFS nodes.

This change is cluster-wide and restarts all NFS instances to activate this setting. The log file now displays more informational messages, for example:

2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_rpc_dispatch_threads 
:THREAD :INFO :5 rpc dispatcher threads were started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :gsh_dbusthread was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :admin thread was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :reaper thread was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :General fridge was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[reaper] nfs_in_grace 
:STATE :EVENT :NFS Server Now IN GRACE
2015-06-03 12:49:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_start 
:NFS STARTUP :EVENT :-------------------------------------------------
2015-06-03 12:49:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_start 
:NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
2015-06-03 12:49:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_start 
:NFS STARTUP :EVENT :-------------------------------------------------
2015-06-03 12:50:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[reaper] nfs_in_grace 
:STATE :EVENT :NFS Server Now NOT IN GRACE

To display the currently configured CES log level, use the following command:

mmces log level

The system displays output similar to the following example:

CES log level is currently set to 0

The log file is /var/adm/ras/mmfs.log.latest. By default, the log level is 0 and other possible values are 1, 2, and 3. To increase the log level, use the following command:

mmces log level 1

NFS-related log information is written to the standard GPFS™ log files as part of the overall CES infrastructure. This information relates to the NFS service management and recovery orchestration within CES.