NFS logs

The clustered export services (CES) NFS server writes log messages in the /var/log/ganesha.log file at runtime.

Operating system's log rotation facility is used to manage NFS logs. The NFS logs are configured and enabled during the NFS server packages installation.

The following example shows a sample log file:
# tail -f /var/log/ganesha.log
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs_Init_admin_thread :NFS CB 
:EVENT :Admin thread initialized
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs4_start_grace 
:STATE :EVENT :NFS Server Now IN GRACE, 
duration 59
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT
 :Callback creds directory (/var/run/ganesha) already exists
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs_rpc_cb_init_ccache 
:NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328378:0)
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[reaper]
 nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main]
 nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
2018-04-09 11:28:18 : epoch 000100a2 : rh424a : gpfs.ganesha.nfsd-20924[main] 
nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
Log levels can be displayed by using the mmnfs config list | grep LOG_LEVEL command. For example: 
mmnfs config list | grep LOG_LEVEL
The system displays output similar to the following example:
LOG_LEVEL: EVENT
By default, the log level is EVENT. Additionally, the following NFS log levels can also be used; starting from lowest to highest verbosity:
  • FATAL
  • MAJ
  • CRIT
  • WARN
  • EVENT
  • INFO
  • DEBUG
  • MID_DEBUG
  • FULL_DEBUG
    Note: The FULL_DEBUG level increases the size of the log file. Use it in the production mode only if instructed by the IBM® Support.

Increasing the verbosity of the NFS server log impacts the overall NFS I/O performance.

To change the logging to the verbose log level INFO, use the following command:

mmnfs config change LOG_LEVEL=INFO

The system displays output similar to the following example:
NFS Configuration successfully changed. NFS server restarted on all NFS nodes on which NFS 
server is running.
This change is cluster-wide and restarts all NFS instances to activate this setting. The log file now displays more informational messages, for example:
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_rpc_dispatch_threads 
:THREAD :INFO :5 rpc dispatcher threads were started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[disp] rpc_dispatcher_thread 
:DISP :INFO :Entering nfs/rpc dispatcher
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :gsh_dbusthread was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :admin thread was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :reaper thread was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_Start_threads 
:THREAD :EVENT :General fridge was started successfully
2015-06-03 12:49:31 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[reaper] nfs_in_grace 
:STATE :EVENT :NFS Server Now IN GRACE
2015-06-03 12:49:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_start 
:NFS STARTUP :EVENT :-------------------------------------------------
2015-06-03 12:49:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_start 
:NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
2015-06-03 12:49:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[main] nfs_start 
:NFS STARTUP :EVENT :-------------------------------------------------
2015-06-03 12:50:32 : epoch 556edba9 : cluster1.ibm.com : ganesha.nfsd-21582[reaper] nfs_in_grace 
:STATE :EVENT :NFS Server Now NOT IN GRACE
Start of change

Enabling logrotate for NFS

To enable logrotate for NFS based on the log file size, complete the following steps:
  1. Modify the default NFS logrotate configuration located at /etc/logrotate.d/ganesha with the following changes:
    1. Add the size option with an appropriate size value, such as 100M or 1G, that indicates the maximum size for log rotation.
    2. Remove the weekly or daily option, as it is not required for the size-based rotation.
    3. Replace the dateext option with dateformat -%Y%m%d%H%M%S to specify the desired date format for rotated log files.
    For example,
    # cat /etc/logrotate.d/ganesha
/var/log/ganesha.log {
size 10M
rotate 52
copytruncate
dateformat -%Y%m%d%H%M%S
compress missingok
}
  2. Add the logrotate service to the crontab by issuing the following command:
    # crontab -u <user> -e
    For example, to add the logrotate service for a root user and set it to run every 5 minutes:
    # crontab -u root -e */5 * * * * /etc/cron.daily/logrotate
    After these steps, logrotate is configured to rotate the nfs-ganesha log file based on the specified size, and the logrotate service is scheduled to run at a desired frequency.
End of change

To display the currently configured CES log level, use the following command:

mmces log level

The system displays output similar to the following example:
CES log level is currently set to 0
The log file is /var/adm/ras/mmfs.log.latest. By default, the log level is 0 and other possible values are 1, 2, and 3. To increase the log level, use the following command:

mmces log level 1

NFS-related log information is written to the standard GPFS log files as part of the overall CES infrastructure. This information relates to the NFS service management and recovery orchestration within CES.