Enhanced debug data collection for hang scenarios

You can use enhanced debug data collection in IBM Storage Scale CES NFS to gather additional diagnostic information when NFS-Ganesha becomes unresponsive or hung.

Overview

Additional diagnostic data can be collected when NFS-Ganesha becomes unresponsive whereas the service process remains active. These diagnostics investigate hang conditions and identify root causes more efficiently.

Depending on the configuration, the system can collect lightweight process diagnostic information, NFS-Ganesha core dumps, and GPFS traces.

Configuration parameters

The following parameters in the mmsysmonitor.conf file control additional debug data collection during NFS hang scenarios:
collectnfscoreonhang
Controls whether the sysmon service captures a Ganesha core dump through gcore when NFS is detected as unresponsive. This flag changes what is collected on a hang event, not what is collected periodically. The default value is false.

When the option is enabled, the collection can include network statistics, Ganesha statistics, cluster-wide waiter information, GPFS statistics, process limits, stack traces, and the core dump through gcore.

collectgpfstraceonhang
Controls whether GPFS trace data is collected when NFS is detected as hung or unresponsive. The default value is true.
This flag is enabled by default to capture GPFS deadlock traces and internal state during hang scenarios.
This trace data helps to identify GPFS-level issues that can contribute to NFS unresponsiveness, such as lock contention, thread deadlocks, or resource exhaustion.

Unresponsive event handling and data collection

The mmsysmon service continuously monitors NFS-Ganesha health by checking process responsiveness through NFS NULL requests and I/O activity. If the NFS-Ganesha process becomes unresponsive or does not respond, the sysmon service triggers an nfs_unresponsive event.

  1. Health checks run periodically at regular intervals.
  2. If NFS-Ganesha does not respond within the UnrespTimeout period of 70 seconds, the service is marked as unresponsive.
  3. The maxwaittime setting of 240 seconds defines the maximum time to wait before declaring NFS-Ganesha as hung.
  4. After an unresponsive condition is detected, the nfsdebug.snap script is triggered automatically.
  5. The script collects diagnostic data based on the enabled collectnfscoreonhang and collectgpfstraceonhang configuration flags.
  6. Data collection starts immediately after detection to capture the NFS-Ganesha state during the nfs_hang condition.

Operational considerations

The flags collectnfscoreonhang and collectgpfstraceonhang can be enabled or disabled independently based on troubleshooting needs.

Warning: Ensure that you have sufficient free space in /var/adm/ras/ and /tmp/ before you enable these options, and monitor disk usage during active troubleshooting sessions.

To configure FTDC for CES NFS, see Configuring debug data collection for NFS hung scenarios.