Managing deadlocks
IBM Spectrum Scale™ provides
functions for automatically detecting potential deadlocks, collecting
deadlock debug data, and breaking up deadlocks.
The distributed nature of GPFS™, the complexity of the locking infrastructure, the dependency on the proper operation of disks and networks, and the overall complexity of operating in a clustered environment all contribute to increasing the probability of a deadlock.
Deadlocks can be disruptive in certain situations, more so than other type of failure. A deadlock effectively represents a single point of failure that can render the entire cluster inoperable. When a deadlock is encountered on a production system, it can take a long time to debug. The typical approach to recovering from a deadlock involves rebooting all of the nodes in the cluster. Thus, deadlocks can lead to prolonged and complete outages of clusters.
To troubleshoot deadlocks, you must have specific types of debug data that must be collected while the deadlock is in progress. Data collection commands must be run manually before the deadlock is broken. Otherwise, determining the root cause of the deadlock after that is difficult. Also, deadlock detection requires some form of external action, for example, a complaint from a user. Waiting for a user complaint means that detecting a deadlock in progress might take many hours.
In GPFS V4.1
and later, automated deadlock detection, automated deadlock data collection,
and deadlock breakup options are provided to make it easier to handle
a deadlock situation.
- Debug data for deadlocks
Debug data for potential deadlocks is automatically collected. System administrators must monitor and manage the file systems where debug data is stored. - Automated deadlock detection
Automated deadlock detection flags unexpected long waiters as potential deadlocks. Effective deadlock detection thresholds are self-tuned to reduce false positive detection. You can register a user program for the deadlockDetected event to receive automatic notification. - Automated deadlock data collection
Automated deadlock data collection gathers crucial debug data when a potential deadlock is detected. - Automated deadlock breakup
Automated deadlock breakup helps resolve a deadlock situation without human intervention. To break up a deadlock, less disruptive actions are tried first; for example, causing a file system panic. If necessary, more disruptive actions are then taken; for example, shutting down a GPFS mmfsd daemon. - Deadlock breakup on demand
Deadlocks can be broken up on demand, which allows a system administrator to choose the appropriate time to start the breakup actions.