Question & Answer
Db2 pureScale deployment is simplified by removing the requirement of configuring pingable IP interfaces on high speed interconnect switches. The adapter port liveliness test has been automated. This automated adapter liveliness detection does not work on certain configurations.
When an adapter is detected down, RSCT detects and logs it in the system logs as follows:
Sep 7 05:28:42 host1 daemon:notice cthats: (Recorded using libct_ffdc.a cv 2):::Error ID: 6zV5DL.urMYP/X1M/8J.1h....................:::Reference ID: :::Template ID: 173c787f:::Details File: :::Location: rsct,nim_control
.C,18.104.22.168,5929 :::TS_LOC_DOWN_ST Possible malfunction on local adapter Adapter interface name hca0 Adapter offset 2 Adapter IP address 10.1.1.101
In turn, Db2 is notified and logs this event in db2diag.log:
2018-09-07-05.28.42.412244+540 E2325A594 LEVEL: Event
PID : 14483680 TID : 542 PROC : ca-wdog 128 [db2inst1]
INSTANCE: db2inst1 NODE : 128
EDUID : 542 EDUNAME: ca-wdog 128 [db2inst1]
FUNCTION: DB2 UDB, high avail services, rocmHCAMonitorCallback, probe:911
MESSAGE : ADM7537I The status of the following adapter changed. Adapter name:
"hca0". New status: "offline". Number of adapters that are
currently online: "1". Host name: "host1".
CHANGE : Communication adapter port
If an adapter is down and these log entries are missing in the above specified files, then Db2 is unable to automatically detect the adapter as down. Depending on which adapter is affected, some of the expected symptoms are:
- The resource on the host with the down adapter being marked Failed Offline
- FODC_Panic of the member
- CF in ERROR state
- Total Cluster Outage
Due to the limitations of some OSes and/or platforms, adapter state is not reflected to enable automated detection.
On the following environments, pingable IP interfaces need to be configured to enable Reliable Scalable Cluster Technology (RSCT) to monitor the network -
- All Db2 pureScale supported environments with LHEA virtualisation
- Intel and Power Linux environments with SEA virtualisation
- Intel and Power Linux environments with SRIOV virtualisation
On these environments, refer to IBM Db2 Knowledge Center to setup netmon.cf file properly.
20 June 2019