NFS logs

The clustered export services (CES) NFS server writes log messages in the /var/log/ganesha.log file at runtime.

Operating system's log rotation facility is used to manage NFS logs. The NFS logs are configured and enabled during the NFS server packages installation.

The following example shows a sample log file:
# tail -f /var/log/ganesha.log
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs_Init_admin_thread :NFS CB 
:EVENT :Admin thread initialized
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs4_start_grace 
:STATE :EVENT :NFS Server Now IN GRACE, 
duration 59
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT
 :Callback creds directory (/var/run/ganesha) already exists
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs_rpc_cb_init_ccache 
:NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328378:0)
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[reaper]
 nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
Log levels can be displayed by using the mmnfs config list | grep LOG_LEVEL command. For example:
mmnfs config list | grep LOG_LEVEL
The system displays output similar to the following example:
LOG_LEVEL: EVENT
By default, the log level is EVENT. Additionally, the following NFS log levels can also be used; starting from lowest to highest verbosity:
  • FATAL
  • MAJ
  • CRIT
  • WARN
  • EVENT
  • INFO
  • DEBUG
  • MID_DEBUG
  • FULL_DEBUG
    Note: The FULL_DEBUG level increases the size of the log file. Use it in the production mode only if instructed by the IBM® Support.

Increasing the verbosity of the NFS server log impacts the overall NFS I/O performance.

To change the logging to the verbose log level INFO, use the following command:

mmnfs config change LOG_LEVEL=INFO

The system displays output similar to the following example:
NFS Configuration successfully changed. NFS server restarted on all NFS nodes on which NFS 
server is running.
This change is cluster-wide and restarts all NFS instances to activate this setting. The log file now displays more informational messages, for example:
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_rpc_dispatch_threads 
:THREAD :INFO :5 rpc dispatcher threads were started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :gsh_dbusthread was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :admin thread was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :reaper thread was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :General fridge was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[reaper] nfs_in_grace 
:STATE :EVENT :NFS Server Now IN GRACE
2015-06-03 12:49:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_start 
:NFS STARTUP :EVENT :-------------------------------------------------
2015-06-03 12:49:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_start 
:NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
2015-06-03 12:49:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_start 
:NFS STARTUP :EVENT :-------------------------------------------------
2015-06-03 12:50:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[reaper] nfs_in_grace 
:STATE :EVENT :NFS Server Now NOT IN GRACE

To display the currently configured CES log level, use the following command:

mmces log level

The system displays output similar to the following example:
CES log level is currently set to 0
The log file is /var/adm/ras/mmfs.log.latest. By default, the log level is 0 and other possible values are 1, 2, and 3. To increase the log level, use the following command:

mmces log level 1

NFS-related log information is written to the standard GPFS log files as part of the overall CES infrastructure. This information relates to the NFS service management and recovery orchestration within CES.