IBM Support

VIOS SSP cluster -status errors "clcmd_com: Could not read the response header. Not able to send command. Error: -37. Unable to get storage pool information (node <VIOS_node_name>)"

Troubleshooting


Problem

'cluster -status -clustername <cluster_name>' fails with:

clcmd_com: Could not read the response header.
Not able to send command. Error: -37.
Unable to get storage pool information (node <VIOS_node_name>)

OR

Unable to get storage pool information (node <VIOS_node_name>)

Symptom

VIOS cluster with nodes at 2.2.6.  However, if the cluster involves a mixture of different VIOS levels, i.e. 2.2.3.51, 2.2.4.10, and 2.2.6.10, the errors may be seen on nodes running the older levels, not just the ones at 2.2.6.

'cluster -status -clustername <cluster_name>' command returns errors intermittently, even though the command output shows the cluster state, node state and pool state as "OK".  

Sample output:

From srvh01vio01 (2.2.6.10)

$ cluster -status -clustername IBM_TEST_CL
Cluster Name         State
IBM_TEST_CL          OK

    Node Name        MTM           Partition Num  State  Pool State
    srvh03vio01      7891-74XSERIAL#-A         1  OK     OK
    srvh03vio02      7891-74XSERIAL#-A         2  OK     OK
    srvh01vio01      7891-74XSERIAL#-B         1  OK     OK
    srvh01vio02      7891-74XSERIAL#-B         2  OK     OK
    srvh05vio01      8406-71YSERIAL#-C         2  OK     OK
    srvh07vio01      8406-71YSERIAL#-D         2  OK     OK
    srv750vio02      8233-E8BSERIAL#-E         3  OK     OK
    srv800vio01      8286-42ASERIAL#-F       127  OK     OK
    srv800vio02      8286-42ASERIAL#-F       128  OK     OK
    srv750vio01      8233-E8BSERIAL#-E         2  OK     OK

Same command from other nodes may  reports errors as shown below.

From srv750vio01 (2.2.4.10):

$ cluster -status -clustername IBM_TEST_CL
...snip...
Unable to get storage pool information (node srvh03vio02)
Unable to get storage pool information (node srvh01vio02)

From srvh03vio01 (2.2.6.10):

$ cluster -status -clustername IBM_TEST_CL
...snip...
clcmd_com: Could not read the response header.
Not able to send command. Error: -37.
Unable to get storage pool information (node srvh05vio01)
Unable to get storage pool information (node srvh07vio01)
Unable to get storage pool information (node srv750vio02)
Unable to get storage pool information (node srv800vio01)
Unable to get storage pool information (node srv800vio02)
Unable to get storage pool information (node srv750vio01)

Subsequent retry later on from the same node may come back OK (with no errors).

Cause

Known issue in VIOS 2.2.6 related to clcomd hang fixed with HYPER APAR IJ04279.

Environment

VIOS nodes running 2.2.6.  Affected releases include 2.2.6.0 thru 2.2.6.23

Diagnosing The Problem

If the VIOS cluster has nodes at 2.2.6, determine the level.  Login to VIOS as padmin and run:

$ ioslevel

Resolving The Problem

IJ04279: POSSIBLE HANG DURING POWERHA VERIFY & SYNC AND OTHER PROCESSES

If you have any VIOS nodes at any of the affected ioslevels, you can temporarily clear the errors by using the local fix noted in the APAR details, which involves cycling clcomd on all the nodes.  (This is not expected to have any effect on the function of the CAA cluster and related products (RSCT, SSP)):

$ oem_setup_env
# stopsrc -s clcomd
# startsrc -s clcomd

To permanently prevent further hangs, update to VIOS 2.2.6.31 . This ioslevel also delivers the following recommended APARs:
IJ02423: CLCMD HANG
IJ03620: SRCLOOP IS GENERATING DEFUNCT PROCESSES RANDOMLY

If there are VIOS nodes in the cluster running a VIOS version below 2.2.6, IBM recommends updating all VIOS nodes in the cluster to 2.2.6.31.  VIOS fixes are available in FixCentral .

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSPHKW","label":"PowerVM Virtual I\/O Server"},"Component":"","Platform":[{"code":"PF002","label":"AIX"}],"Version":"VIOS 2.2.6.","Edition":"","Line of Business":{"code":"LOB57","label":"Power"}}]

Document Information

More support for:
PowerVM Virtual I/O Server

Software version:
VIOS 2.2.6.

Operating system(s):
AIX

Document number:
740017

Modified date:
20 October 2021

UID

ibm10740017

Manage My Notification Subscriptions

Page Feedback