NFS events

The following table lists the events that are created for the NFS component.

Table 1. Events for the NFS component
Event Event
Type
Severity Call Home Details
dbus_error STATE_CHANGE WARNING no Message: The DBus availability check failed.
Description: Failed to query DBus when the NFS service is registered.
Cause: The DBus was detected as down, which might cause several issues on the local node.
User Action: Stop the NFS service, restart the DBus, and start the NFS service again.
dbus_error_pod STATE_CHANGE WARNING no Message: {id}: The DBus availability check failed.
Description: Failed to query DBus when the NFS service is registered.
Cause: The DBus was detected as down, which might cause several issues on the local node.
User Action: Stop the NFS service, restart the DBus, and start the NFS service again.
dir_statd_perm_ok STATE_CHANGE INFO no Message: The permissions of the local NFS statd directory are correct.
Description: The permissions of the local NFS statd directory are correct.
Cause: N/A
User Action: N/A
dir_statd_perm_problem STATE_CHANGE WARNING no Message: The permissions of the local NFS statd directory might be incorrect for operation. {0}={1} (reference={2}).
Description: The permissions of the local NFS statd directory might be incorrect for operation.
Cause: The permissions of the local NFS statd directory might be incorrect for operation.
User Action: Check permissions of the local NFS statd directory.
disable_nfs_service INFO_EXTERNAL INFO no Message: The CES NFS service is disabled.
Description: The NFS service is disabled on this node. Disabling a service also removes all configuration files, which are different from stopping a service.
Cause: The user disabled the NFS service by using the mmces service disable nfs command.
User Action: N/A
enable_nfs_service INFO_EXTERNAL INFO no Message: The CES NFS service was enabled.
Description: The NFS service is enabled on this node. Enabling a protocol service also automatically installs the required configuration files with the current valid configuration settings.
Cause: The user enabled the NFS service by using the mmces service enable nfs command.
User Action: N/A
ganeshaexit INFO_EXTERNAL INFO no Message: The CES NFS service is stopped.
Description: An NFS server instance was terminated.
Cause: An NFS instance was terminated or killed.
User Action: Restart the NFS service when the root cause for this issue is solved.
ganeshagrace INFO_EXTERNAL INFO no Message: The CES NFS service is set to a grace mode.
Description: The NFS server is set to a grace mode for a limited time, which gives the time to previously connected clients to recover their file locks.
Cause: The grace period is always cluster-wide. The NFS export configurations might change, and one or more NFS servers get restarted.
User Action: N/A
knfs_available_warn STATE_CHANGE WARNING no Message: The kernel NFS service state is not masked or disabled.
Description: The kernel NFS service is available, but there is a risk that it can be started, and can cause issues.
Cause: The kernel NFS service might be available.
User Action: Check the NFS setup. The nfs.service should be deactivated or masked to avoid conflicts with the IBM Storage Scale NFS server.
knfs_disabled_ok STATE_CHANGE INFO no Message: The kernel NFS service is disabled, but it should be masked.
Description: The kernel NFS service is disabled, but there is a risk that it can be started.
Cause: N/A
User Action: N/A
knfs_masked_ok STATE_CHANGE INFO no Message: The kernel NFS service is masked.
Description: The kernel NFS service is masked to avoid the service from being accidentally started.
Cause: N/A
User Action: N/A
knfs_running_warn STATE_CHANGE WARNING no Message: The kernel NFS service state is active.
Description: The kernel NFS service is active and can cause conflicts with the IBM Storage Scale NFS server.
Cause: The kernel NFS service is active.
User Action: Check the NFS setup. The nfs.service is either deactivated or masked to avoid conflicts with the IBM Storage Scale NFS server.
mountd_rpcinfo_ok STATE_CHANGE INFO no Message: The NFS mountd service is listed by rpcinfo.
Description: A required service is listed.
Cause: N/A
User Action: N/A
mountd_rpcinfo_unknown STATE_CHANGE WARNING no Message: The NFS mount service is not listed by rpcinfo.
Description: A required service is not listed.
Cause: The mountd service is not listed by rpcinfo, but expected to run.
User Action: Check whether the issue is caused by a rpcbind crash or kernel NFS usage. Run the systemctl restart nfs-ganesha.service command.
nfs_hung STATE_CHANGE WARNING FTDC upload Message: The NFS service is unresponsive.
Description: A check showed that the CES NFS service, which is supposed to be running, is unresponsive. This event restarts NFS automatically.
Cause: Process might be hung or busy.
User Action: Check the service state for any restart-related errors.
nfs3_down INFO WARNING no Message: The NFS v3 NULL check failed.
Description: The NFS v3 NULL check failed when it was expected to be functioning. This check verifies whether the NFS server reacts to NFS v3 requests. The NFS v3 protocol must be enabled for this check. If this down state is detected, then further checks are done to figure out whether the NFS server is still working. If the NFS server seems not to be working, then a failover is triggered. If NFS v3 and NFS v4 protocols are configured, then only the v3 NULL test is performed.
Cause: The NFS server is either under high load or hung up, which restricts the processing of the request.
User Action: Check the health state of the NFS server and restart, if necessary.
nfs3_up INFO INFO no Message: NFS v3 NULL check is successful.
Description: The NFS v3 NULL check is successful.
Cause: N/A
User Action: N/A
nfs4_down INFO WARNING no Message: NFS v4 NULL check failed.
Description: The NFS v4 NULL check failed. This check verifies whether the NFS server reacts to NFS v4 requests. The NFS v4 protocol must be enabled for this check. If this down state is detected, then further checks are done to figure out whether the NFS server is still working. If the NFS server seems not to be working, then a failover is triggered.
Cause: The NFS server is either under high load or hung up, which restricts the processing of the request.
User Action: Check the health state of the NFS server and restart, if necessary.
nfs4_up INFO INFO no Message: The NFS v4 NULL check is successful.
Description: The NFS v4 NULL check is successful.
Cause: N/A
User Action: N/A
nfs_active STATE_CHANGE INFO no Message: The NFS service is now active.
Description: The NFS service must be up and running, and in a healthy state to provide the configured file exports.
Cause: N/A
User Action: N/A
nfs_active_pod STATE_CHANGE INFO no Message: {id}: The NFS service is now active.
Description: The NFS service must be up and running, and in a healthy state to provide the configured file exports.
Cause: N/A
User Action: N/A
nfs_dbus_error STATE_CHANGE WARNING no Message: NFS check by using the DBus failed.
Description: The NFS service must be registered on DBus to be fully working, which is currently not the case.
Cause: The NFS service is registered on DBus, but there was a problem while accessing it.
User Action: Check the health state of the NFS service and restart the NFS service. Check the log files for reported issues.
nfs_dbus_error_pod STATE_CHANGE WARNING no Message: {id}: NFS check by using the DBus failed.
Description: The NFS service must be registered on DBus to be fully working, which is currently not the case.
Cause: The NFS service is registered on DBus, but there was a problem while accessing it.
User Action: Check the health state of the NFS service and restart the NFS service. Check the log files for reported issues.
nfs_dbus_failed STATE_CHANGE WARNING no Message: NFS check by using the DBus did not return the expected message.
Description: The NFS service configuration settings or log configuration settings are queried by using the DBus. The result is checked for expected keywords.
Cause: The NFS service is registered on DBus, but the check by using the DBus did not return the expected result.
User Action: Stop the NFS service and start it again. Check the log configuration of the NFS service.
nfs_dbus_failed_pod STATE_CHANGE WARNING no Message: {id}: NFS check by using the DBus did not return the expected message.
Description: The NFS service configuration settings or log configuration settings are queried by using the DBus. The result is checked for expected keywords.
Cause: The NFS service is registered on DBus, but the check by using the DBus did not return the expected result.
User Action: Stop the NFS service and start it again. Check the log configuration of the NFS service.
nfs_dbus_ok STATE_CHANGE INFO no Message: The NFS check by using the DBus is successful.
Description: The check to verify whether the NFS service is registered on DBus and working, is successful.
Cause: N/A
User Action: N/A
nfs_dbus_ok_pod STATE_CHANGE INFO no Message: {id}: The NFS check by using the DBus is successful.
Description: The check to verify whether the NFS service is registered on DBus and working, is successful.
Cause: N/A
User Action: N/A
nfs_exported_fs_chk STATE_CHANGE_EXTERNAL INFO no Message: The Cluster State Manager (CSM) cleared the 'nfs_exported_fs_down' event.
Description: Declared NFS exported file systems are either available again on this node, or not available on any node.
Cause: The CSM cleared an 'nfs_exported_fs_down' event on this node to display the self-detected state of this node.
User Action: N/A
nfs_exported_fs_down STATE_CHANGE_EXTERNAL ERROR no Message: One or more declared NFS exported file systems are not available on this node.
Description: One or more declared NFS exported file systems are not available on this node. Other nodes might have those file systems available.
Cause: One or more declared NFS exported file systems are not available on this node.
User Action: Check NFS export-related local and remote file system states.
nfs_exports_clear_state STATE_CHANGE INFO no Message: Clear local NFS export down state temporarily.
Description: Clear local NFS export down state temporarily, because an 'all nodes have the same problem' message is received.
Cause: N/A
User Action: N/A
nfs_exports_down STATE_CHANGE WARNING no Message: One or more declared file systems for NFS exports are not available.
Description: One or more declared file systems for NFS exports are unavailable.
Cause: One or more declared file systems for NFS exports are unavailable.
User Action: Check local and remote file system states.
nfs_exports_up STATE_CHANGE INFO no Message: All declared file systems for NFS exports are available.
Description: All declared file systems for NFS exports are available.
Cause: N/A
User Action: N/A
nfs_in_grace STATE_CHANGE WARNING no Message: NFS is in a grace mode.
Description: The monitor detected that CES NFS is in a grace mode. During this time, the NFS state is shown as degraded.
Cause: The NFS service was started or restarted.
User Action: N/A
nfs_in_grace_pod STATE_CHANGE WARNING no Message: {id}: NFS is in a grace mode.
Description: The monitor detected that CES NFS is in a grace mode. During this time, the NFS state is shown as degraded.
Cause: The NFS service was started or restarted.
User Action: N/A
nfs_not_active STATE_CHANGE ERROR FTDC upload Message: NFS service is inactive.
Description: A check showed that the CES NFS service, which is supposed to be running, is inactive.
Cause: Process might be hung.
User Action: Restart the CES NFS service.
nfs_not_active_pod STATE_CHANGE ERROR FTDC upload Message: {id}: NFS service is inactive.
Description: A check showed that the CES NFS service, which is supposed to be running, is inactive.
Cause: Process might be hung.
User Action: Restart the NFS service.
nfs_not_dbus STATE_CHANGE WARNING no Message: NFS service is unavailable as the DBus service.
Description: The NFS service is currently not registered on DBus. In this mode, the NFS service is not fully working. Exports cannot be added or removed, and not set in the grace mode, which is important for data consistency.
Cause: The NFS service might be started while the DBus was down.
User Action: Stop the NFS service, restart the DBus, and start the NFS service again.
nfs_not_dbus_pod STATE_CHANGE WARNING no Message: {id}: NFS service is unavailable as the DBus service.
Description: The NFS service is currently not registered on DBus. In this mode, the NFS service is not fully working. Exports cannot be added or removed, and not set in the grace mode, which is important for data consistency.
Cause: The NFS service might be started while the DBus was down.
User Action: Stop the NFS service, restart the DBus, and start the NFS service again.
nfs_openConnection INFO WARNING no Message: NFS has invalid open connection to CES IP {0}.
Description: The NFS server did not close a connection to a released socket.
Cause: The NFS server has an open connection to a not existing CES IP.
User Action: Check for unexpected misbehavior of clients and reconnect them.
nfs_rpcinfo_ok STATE_CHANGE INFO no Message: The NFS program is listed by rpcinfo.
Description: The required program is listed.
Cause: N/A
User Action: N/A
nfs_rpcinfo_unknown STATE_CHANGE WARNING no Message: The NFS program is not listed by rpcinfo.
Description: The required program is not listed.
Cause: The NFS program is not listed by rpcinfo, but expected to run.
User Action: Check whether the issue is caused by a rpcbind crash.
nfs_sensors_active TIP INFO no Message: The NFS perfmon sensor {0} is active.
Description: The NFS perfmon sensors are active. This event's monitor is running only once an hour.
Cause: The NFS perfmon sensors' period attribute is greater than 0.
User Action: N/A
nfs_sensors_inactive TIP TIP no Message: The NFS perfmon sensor {0} is inactive.
Description: The NFS perfmon sensors are inactive. This event's monitor is running only once an hour.
Cause: The NFS perfmon sensors' period attribute is 0.
User Action: Set the period attribute of the NFS sensors to a value greater than 0. For more information, use the mmperfmon config update SensorName.period=N command where 'SensorName' is the name of a specific NFS sensor and 'N' is a natural number greater than 0. Consider that this TIP monitor is running only once per hour and it might take up to one hour to detect the changes in the configuration.
nfs_sensors_not_configured TIP TIP no Message: The NFS perfmon sensor {0} is not configured.
Description: The NFS perfmon sensor does not exist in the mmperfmon config show command.
Cause: The NFS perfmon sensor is not configured in the sensors' configuration file.
User Action: Include the sensors into the perfmon configuration by using the mmperfmon config add --sensors /opt/IBM/zimon/defaults/ZIMonSensors_nfs.cfg command. An example for the configuration file can be found in the mmperfmon command page in the Command Reference Guide.
nfs_unresponsive STATE_CHANGE WARNING no Message: The NFS service is unresponsive.
Description: A check showed that the CES NFS service, which is supposed to be running, is unresponsive.
Cause: Process might be hung or busy.
User Action: Restart the CES NFS service when this state persists.
nfs_unresponsive_pod STATE_CHANGE WARNING no Message: {id}: The NFS service is unresponsive.
Description: A check showed that the CES NFS service, which is supposed to be running, is unresponsive.
Cause: Process might be hung or busy.
User Action: Restart the CES NFS service when this state persists.
nfsd_down STATE_CHANGE ERROR FTDC upload Message: NFSD process is not running.
Description: Checks for an NFS service process.
Cause: The NFS server process was not detected.
User Action: Check the health state of the NFS server and restart, if necessary. The process might get hung or be in a dysfunctional state. Ensure that the kernel NFS server is not running.
nfsd_no_restart INFO WARNING no Message: NFSD process cannot be restarted. Reason: {0}.
Description: An expected NFS service process was not running, but cannot be restarted.
Cause: The NFS server process was not detected and cannot be restarted.
User Action: Check the health state of the NFS server and restart, if necessary. Check the issues, which lead to the unexpected failure. Ensure that the kernel NFS server is not running.
nfsd_restart INFO WARNING FTDC upload Message: NFSD process restarted.
Description: An expected NFS service process was not running and then restarted.
Cause: The NFS server process was not detected and restarted.
User Action: Check the health state of the NFS server and restart, if necessary. Check the issues, which lead to the unexpected failure. Ensure that the kernel NFS server is not running.
nfsd_up STATE_CHANGE INFO no Message: NFSD process is running.
Description: The NFS server process is detected.
Cause: N/A
User Action: N/A
nfsd_warn INFO WARNING no Message: NFSD process monitoring returned an unknown result.
Description: The NFS server process monitoring returned an unknown result.
Cause: The NFS server process state cannot be determined due to a problem.
User Action: Check the health state of the NFS server and restart, if necessary.
nfsserver_found_pod INFO_ADD_ENTITY INFO no Message: The NFS server {id} was found.
Description: A NFS server was detected.
Cause: A NFS Server was detected.
User Action: N/A
nfsserver_vanished_pod INFO_DELETE_ENTITY INFO no Message: The NFS Server {id} has vanished.
Description: A declared NFS server was not detected.
Cause: A NFS server is not in use for an IBM Storage Scale file system, which can be a valid situation.
User Action: N/A
nlockmgr_rpcinfo_ok STATE_CHANGE INFO no Message: The NFS nlockmgr service is listed by rpcinfo.
Description: A required service is listed.
Cause: N/A
User Action: N/A
nlockmgr_rpcinfo_unknown STATE_CHANGE WARNING no Message: The NFS nlockmgr service is not listed by rpcinfo.
Description: A required service is not listed.
Cause: The nlockmgr service is not listed by rpcinfo, but expected to run.
User Action: Check whether the issue is caused by a rpcbind crash or kernel NFS usage. Run the systemctl restart nfs-ganesha.service command.
portmapper_down STATE_CHANGE ERROR no Message: Portmapper port 111 is inactive.
Description: The portmapper is needed to provide the NFS services to clients.
Cause: The portmapper is not running on port 111.
User Action: Check whether the portmapper service is running, and if any services are conflicting with the portmapper service on this system.
portmapper_down_pod STATE_CHANGE ERROR no Message: {id}: Portmapper port 111 is inactive.
Description: The portmapper is needed to provide the NFS services to clients.
Cause: The portmapper is not running on port 111.
User Action: Check whether the portmapper service is running, and if any services are conflicting with the portmapper service on this system.
portmapper_up STATE_CHANGE INFO no Message: Portmapper port is now active.
Description: The portmapper is running on port 111.
Cause: N/A
User Action: N/A
portmapper_up_pod STATE_CHANGE INFO no Message: {id}: Portmapper port is now active.
Description: The portmapper is running on port 111.
Cause: N/A
User Action: N/A
portmapper_warn INFO WARNING no Message: Portmapper port monitoring (111) returned an unknown result.
Description: The portmapper process monitoring returned an unknown result.
Cause: The portmapper status cannot be determined due to a problem.
User Action: Restart the portmapper, if necessary.
portmapper_warn_pod INFO WARNING no Message: {id}: Portmapper port monitoring (111) returned an unknown result.
Description: The portmapper process monitoring returned an unknown result.
Cause: The portmapper status cannot be determined due to a problem.
User Action: Restart the portmapper, if necessary.
postIpChange_info_n INFO_EXTERNAL INFO no Message: IP addresses are modified.
Description: Notification that IP addresses are moved around the cluster nodes.
Cause: CES IP addresses are moved or added to the node, and activated.
User Action: N/A
rpc_mountd_inv_user STATE_CHANGE WARNING no Message: The mount port {0} does not belong to IBM Storage Scale NFS.
Description: The required daemon process for NFS has an unexpected owner.
Cause: The mount daemon process does not belong to NFS Ganesha.
User Action: Check the NFS setup and ensure that the kernel NFS is not activated.
rpc_mountd_ok STATE_CHANGE INFO no Message: The mountd service has the expected user.
Description: A required daemon process for NFS is functional.
Cause: N/A
User Action: N/A
rpc_nfs_inv_user STATE_CHANGE WARNING no Message: The NFS port {0} does not belong to IBM Storage Scale NFS.
Description: The required program for NFS has an unexpected owner.
Cause: The NFS program does not belong to NFS Ganesha.
User Action: Check the NFS setup and ensure that the kernel NFS is not activated.
rpc_nfs_ok STATE_CHANGE INFO no Message: The NFS service has the expected user.
Description: A required program for NFS is functional.
Cause: N/A
User Action: N/A
rpc_nlockmgr_inv_user STATE_CHANGE WARNING no Message: The nlockmgr port {0} does not belong to IBM Storage Scale NFS.
Description: The required daemon process for NFS has an unexpected owner.
Cause: The lock manager process does not belong to NFS Ganesha.
User Action: Check the NFS setup and ensure that the kernel NFS is not activated.
rpc_nlockmgr_ok STATE_CHANGE INFO no Message: The nlockmgr service has the expected user.
Description: A required daemon process for NFS is functional.
Cause: N/A
User Action: N/A
rpc_rpcinfo_warn INFO WARNING no Message: The rpcinfo check returned an unknown result.
Description: The rpcinfo NFS program check returned an unknown result.
Cause: The rpcinfo status cannot be determined due to a problem.
User Action: Restart the portmapper, if necessary. Check NFS configuration.
rpcbind_down STATE_CHANGE WARNING no Message: The rpcbind process is not running.
Description: The rpcbind process is used by NFS v3 and v4.
Cause: The rpcbind process is not running.
User Action: Start rpcbind manually by using the systemctl restart rpcbind command in case it does not auto-starts. A restart of NFS might be needed to register its services to rpcbind.
rpcbind_down_pod STATE_CHANGE WARNING no Message: {id}: The rpcbind process is not running.
Description: The rpcbind process is used by NFS v3 and v4.
Cause: The rpcbind process is not running.
User Action: Start rpcbind manually by using the systemctl restart rpcbind command in case it does not auto-starts. A restart of NFS might be needed to register its services to rpcbind.
rpcbind_restarted INFO INFO no Message: The rpcbind process is restarted.
Description: The rpcbind process is restarted.
Cause: The rpcbind process is restarted by the NFS monitor.
User Action: Check whether NFS processing works as expected. Otherwise, restart NFS to get registered to rpcbind.
rpcbind_unresponsive STATE_CHANGE ERROR no Message: The rpcbind process is unresponsive. Attempt to restart.
Description: The rpcbind process does not work. A restart can help.
Cause: The rpcbind process is unresponsive.
User Action: Start rpcbind manually by using the systemctl restart rpcbind command in case it does not auto-starts.
rpcbind_unresponsive_pod STATE_CHANGE ERROR no Message: {id}: The rpcbind process is unresponsive. Attempt to restart.
Description: The rpcbind process does not work. A restart can help.
Cause: The rpcbind process is unresponsive.
User Action: Start rpcbind manually by using the systemctl restart rpcbind command in case it does not auto-starts.
rpcbind_up STATE_CHANGE INFO no Message: The rpcbind process is running.
Description: The rpcbind process is running.
Cause: N/A
User Action: N/A
rpcbind_up_pod STATE_CHANGE INFO no Message: {id}: The rpcbind process is running.
Description: The rpcbind process is running.
Cause: N/A
User Action: N/A
rpcbind_warn STATE_CHANGE WARNING no Message: The rpcbind check failed with an issue.
Description: The rpcbind check failed with an issue, which might be temporary.
Cause: The rpcbind process is unresponsive.
User Action: N/A
rpcbind_warn_pod STATE_CHANGE WARNING no Message: {id}: The rpcbind check failed with an issue.
Description: The rpcbind check failed with an issue, which might be temporary.
Cause: The rpcbind process is unresponsive.
User Action: N/A
rquotad_down INFO INFO no Message: rpc.rquotad is not running.
Description: Currently, not in use.
Cause: N/A
User Action: N/A
rquotad_up INFO INFO no Message: rpc.rquotad is running.
Description: Currently, not in use.
Cause: N/A
User Action: N/A
start_nfs_service INFO_EXTERNAL INFO no Message: CES NFS service is started.
Description: Notification about an NFS service start.
Cause: The NFS service was started by issuing the mmces service start nfs command.
User Action: N/A
start_nfs_service_pod INFO_EXTERNAL INFO no Message: {id}: NFS monitoring service is started.
Description: Notification about an NFS service start.
Cause: The NFS monitoring service was started.
User Action: N/A
statd_down STATE_CHANGE ERROR no Message: The rpc.statd process is not running.
Description: The statd process is used by NFS v3 to handle file locks.
Cause: The statd process is not running.
User Action: Stop and start the NFS service, which also attempts to start the statd process.
statd_multiple STATE_CHANGE WARNING no Message: The rpc.statd process is running multiple times.
Description: The statd process is used by NFS v3 to handle file locks.
Cause: The statd process, which is running multiple times, either indicates an issue with rpcbind or manually starts.
User Action: Stop and start the NFS service. This stops and restarts the statd processes.
statd_up STATE_CHANGE INFO no Message: The rpc.statd process is running.
Description: The statd process is running, which is used by NFS v3 to handle file locks.
Cause: N/A
User Action: N/A
statd_wrong STATE_CHANGE WARNING no Message: The rpc.statd process is misconfigured.
Description: The statd process is used by NFS v3 to handle file locks.
Cause: The statd process was not started by NFS startup. The command line parameter mmstatdcallout is missing or has an unexpected owner.
User Action: Stop and start the NFS service. This stops and restarts the statd processes.
stop_nfs_service INFO_EXTERNAL INFO no Message: CES NFS service is stopped.
Description: Notification about an NFS service stop.
Cause: The NFS service was stopped by using the mmces service stop nfs command.
User Action: N/A
stop_nfs_service_pod INFO_EXTERNAL INFO no Message: {id}: NFS monitoring service is stopped.
Description: Notification about an NFS service stop.
Cause: The NFS monitoring service was stopped.
User Action: N/A