How to get started with troubleshooting
Troubleshooting the issues that are reported in the system is easier when you follow the process step-by-step.
About this task
When you experience some issues with the system, go through the following steps to get started with the troubleshooting:
Procedure
- Check the events that are reported in various nodes of the cluster by using the mmhealth cluster show and mmhealth node show commands.
- Check the user action corresponding to the active events and take the appropriate action. For more information on the events and corresponding user action, see Events.
- Check for events that happened before the event you are trying to investigate. They might give you an idea about the root cause of problems. For example, if you see an event nfs_in_grace and node_resumed a minute before you get an idea about the root cause why NFS entered the grace period, it means that the node resumed after a suspend.
- Collect the details of the issues through logs, dumps, and traces. You can use various CLI commands like gpfs.snap, esssnap and the GUI page to collect the details of the issues reported in the system.
- Based on the type of issue, browse through the various topics that are listed in the troubleshooting section and try to resolve the issue.
- If you cannot resolve the issue by yourself, contact IBM® Support.
Example
Example command
output
# essinstallcheck -N localhost
Start of install check
nodelist: localhost
Getting package information.
[WARN] Package check cannot be performed other than on EMS node.
Checking nodes.
================== Summary of node: localhost =============================
[INFO] Getting system profile setting.
Installed version: ess3000_6.1.0.0_0315-22_dme
[OK] Linux kernel installed: 4.18.0-193.40.1.el8_2.x86_64
[OK] Systemd installed: 239-29.el8.x86_64
[OK] Networkmgr installed: 1.22.8-4.el8.x86_64
[OK] Mellanox OFED level: MLNX_OFED_LINUX-4.9-2.2.4.0
[OK] System Firmware: 2.02.000_0B0G_1.73_FB300052_0C32.official_FW1255_FW1255
[OK] System profile setting: scale
[OK] System profile verification PASSED.
[OK] Memory inspection passed for ESS 3000 nodes: localhost
[OK] CPU inspection passed for ESS 3000 nodes: localhost
[OK] Boot drive health inspection passed for ESS 3000 nodes: localhost
[OK] Drive partition health inspection passed for ESS 3000 nodes: localhost
[OK] Drive format correct for ESS 3000 nodes: localhost
[OK] GNR Level: 5.1.0.2 efix4
Performing Spectrum Scale RAID configuration check.
[OK] mmvdisk settings match best practices.
[OK] GNR callback events:
postRGTakeover,postRGRelinquish,rgOpenFailed,rgPanic,pdFailed,pdRecovered,pdReplacePdisk,pdPathDown,daRebuildFailed
[OK] Network adapter MT4121 firmware: 16.28.2006, net adapter count: 4
[OK] Network adapter firmware
Obtaining storage firmware versions from IO nodes. May take a long time...
[OK] Enclosure 5141-AF8 firmware: 1111, enclosure count: 1
[OK] Drive a8241014065c firmware: SN5ASN5A, drive count: 1
[OK] Drive a8221014063b firmware: SN1OSN1O, drive count: 11
[OK] Storage system firmware
[OK] Node is not reserving KVM memory.
[WARN] IBM Electronic Service Agent (ESA) test cannot be checked other than EMS node.
[OK] Software Callhome group defined.
[OK] Software callhome configured correctly on ESS system.
[OK] Software Callhome Connectivity Verification Passed.
End of install check
[PASS] essinstallcheck passed successfully
# mmhealth cluster show
Component Total Failed Degraded Healthy Other
--------------------------------------------------------------------------------------
NODE 4 0 0 2 2
GPFS 4 0 0 2 2
NETWORK 4 0 0 4 0
FILESYSTEM 1 0 0 1 0
DISK 4 0 0 4 0
CALLHOME 1 0 0 1 0
FILESYSMGR 1 0 0 1 0
GUI 1 0 1 0 0
NATIVE_RAID 2 0 0 2 0
PERFMON 3 0 0 3 0
THRESHOLD 3 0 0 3 0
# mmhealth node show
Node name: fab3d-hs.example.net
Node status: HEALTHY
Status Change: 8 hours ago
Component Status Status Change Reasons
--------------------------------------------------------------
GPFS HEALTHY 8 hours ago -
NETWORK HEALTHY 8 hours ago -
FILESYSTEM HEALTHY 8 hours ago -
DISK HEALTHY 8 hours ago -
NATIVE_RAID HEALTHY 8 hours ago -
PERFMON HEALTHY 8 hours ago -
THRESHOLD HEALTHY 8 hours ago -
# gnrhealthcheck
################################################################
# gnrhealthcheck invoked: Wed Mar 17 10:09:11 MST 2021
################################################################
################################################################
# Beginning topology checks.
################################################################
Topology checks successful.
################################################################
# Beginning enclosure checks.
################################################################
Enclosure checks successful.
################################################################
# Beginning recovery group checks.
################################################################
Recovery group checks successful.
################################################################
# Beginning pdisk checks.
################################################################
Pdisk checks successful.
################################################################
# Beginning IBM Power RAID checks.
################################################################
IBM Power RAID checks successful.
################################################################
# Beginning the NVMe Controller checks.
################################################################
The NVMe Controller checks are successful.
################################################################
# Beginning SSD endurance checks
################################################################
The SSD endurance checks are successful.# essinstallcheck -N ems1-ib,essio11-ib,essio12-ib --get-version
Start of install check
nodelist: ems1-ib essio11-ib essio12-ib
Node: ems1-ib Installed version: ess5000_6.1.0.0_0311-20_dme
Node: essio11-ib Installed version: ess5000_6.1.0.0_0311-20_dme
Node: essio12-ib Installed version: ess5000_6.1.0.0_0311-20_dme
[PASS] essinstallcheck passed successfully
# mmhealth cluster show
Component Total Failed Degraded Healthy Other
--------------------------------------------------------------------------------------
NODE 4 0 1 2 1
GPFS 4 0 0 3 1
NETWORK 4 0 3 1 0
FILESYSTEM 2 0 0 2 0
DISK 8 0 0 8 0
CALLHOME 1 1 0 0 0
CES 1 0 0 1 0
CESIP 1 0 0 1 0
FILESYSMGR 1 0 0 1 0
NATIVE_RAID 2 0 2 0 0
PERFMON 3 1 0 2 0
THRESHOLD 3 0 0 3 0
# mmhealth node show
Node name: ems1-ib.example.net
Node status: DEGRADED
Status Change: 5 days ago
Component Status Status Change Reasons
------------------------------------------------------------------------------------------------------
CALLHOME FAILED 5 days ago callhome_heartbeat_failed
GPFS HEALTHY 5 days ago -
NETWORK DEGRADED 5 days ago ib_rdma_port_width_low(mlx5_0/1, mlx5_1/1)
FILESYSTEM HEALTHY 5 days ago -
PERFMON FAILED 5 days ago pmcollector_down
THRESHOLD HEALTHY 5 days ago -
# gnrhealthcheck
################################################################
# gnrhealthcheck invoked: Wed Mar 17 12:20:26 CDT 2021
################################################################
################################################################
# Beginning topology checks.
################################################################
Topology checks successful.
################################################################
# Beginning enclosure checks.
################################################################
Enclosure checks successful.
################################################################
# Beginning recovery group checks.
################################################################
Recovery group checks successful.
################################################################
# Beginning pdisk checks.
################################################################
Found recovery group ess5k_7894E4A pdisk n001v001 has 0 paths.
################################################################
# Beginning IBM Power RAID checks.
################################################################
IBM Power RAID checks successful.
################################################################
# Beginning the NVMe Controller checks.
################################################################
The NVMe Controller checks are successful.
################################################################
# Beginning SSD endurance checks
################################################################
The SSD endurance checks are successful.