PH57529: AN ACCELERATOR MULTI-NODE CLUSTER IS DOWN AND NONE OF THE DATA NODES IS REACHABLE

APAR status

• Closed as documentation error.

Error description

• A customer running accelerator maintenance level 7.5.9 was
  alerted that all nodes (LPARs) of an accelerator multi-node
  cluster were offline:
  
  -- IBM Db2 Warehouse Cluster Status --
  +----------+---------------+------+---------+---------+
  | NodeName | IP            | Type | Role    | State   |
  +----------+---------------+------+---------+---------+
  | data1    | 192.xxx.yy.za | DATA | Unknown | Unknown |
  | data2    | 192.xxx.yy.zb | DATA | Unknown | Unknown |
  | data3    | 192.xxx.yy.zc | DATA | Unknown | Unknown |
  | data4    | 192.xxx.yy.zd | DATA | Unknown | Unknown |
  | data5    | 192.xxx.yy.zz | DATA | Unknown | Unknown |
  | head     | 192.xxx.yy.1  | HEAD | Unknown | Unknown |
  +----------+---------------+------+---------+---------+
  
  The accelerator Admin UI was showing red dots (meaning:
  components unavailable) under the following sections: Appliance
  authentication service, Appliance data service, and Db2
  Accelerator service. As a connection to the accelerator was
  impossible, the customer was not able to create a trace."
  
  The subject issue affects at least the accelerator maintenance
  levels 7.5.9, 7.5.10.x, and 7.5.11.x.
  A fix will be provided with Accelerator maintenance level
  7.5.12.
  
  Additional keywords:
  TS013934601 wolverine logutil.py directory DT244003
  GH/Everest/Customer-Cases/issues/590
  
  Additional information for IBM support:
  The analysis of the log data shows the 'systemctl restart
  wolverine' is failing because the given log file is a directory:
  (date/time) head wolverineÝ13699¨: File
  "/usr/local/lib/python3.6/site-packages/wolverine/utils/logutil.
  py", line 67, in addLogFile
  (date/time) head wolverineÝ13699¨: with open(fileName, 'a') as
  f:
  (date/time) head wolverineÝ13699¨: IsADirectoryError: ÝErrno 21¨
  Is a directory: '/wolverine/ha'
  Error from "dbdiag collect --sets ha.log"
  Ýroot@head - Db2wh wolverine¨# dbdiag collect --sets ha.log
  ----------------------------------------------------------------
  ----------------
                      dbdiag v1.0.19.1 (20220630175713blocal)
  ----------------------------------------------------------------
  ----------------
  Log File: '/scratch/dbdiag/log/dbdiag_20230818_192044_59897.log'
  Report generated by head at 2023-08-18 19:20:44
  Getting node(s) information
    Failed to get node(s) information from host 'head'. Trying
    nodes file.
    Attempting to get nodes info using host 'data1'
  ERROR: Specified component set 'ha.log' is not found
  

Local fix

• The following manual patch was applied successfully:
  in
  /usr/local/lib/python3.6/site-packages/wolverine/ha/service.py
  get_log_file()
  returns this string:
  "/wolverine/head/log/ha.log"
  

Problem summary

• Problem Sumamry:
  See Error Description.
  
  Users Affected:
  Users of Db2 Analytics Accelerator on IBM Z.
  
  Problem Scenario:
  See Error Description.
  
  Problem Symptoms:
  See Error Description.
  

Problem conclusion

• The issue has been fixed with Accelerator maintenance level
  7.5.12. Upgrade your accelerator environments accordingly.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels