How To
Summary
Method to collect cluster node data (eg. cpuinfo, meminfo, systemctl status kubelet, journal, dmesg, etc) through the CLUSTERNODES secondary mode of the CP4AIOps MustGather.
Environment
OCP Server = 4.16.45
OC Client = 4.16.38
CP4AIOps MustGather = 1.28.18
CP4AIOps = 4.10.1
OC Client = 4.16.38
CP4AIOps MustGather = 1.28.18
CP4AIOps = 4.10.1
Steps
(1) Download the latest CP4AIOps MustGather (link).
OR
Upgrade the current CP4AIOps MustGather to the latest version:
./waiops-mustgather.sh -UD
(2) Data collection for cluster nodes is enabled with the "-d" option.
The alias "install" is configured with the "-d" option, so if executed, the command gathers cluster node data:
# ./waiops-mustgather.sh -O .listalias
=========================================================================================================================================================================================
ALIAS_NAME DESCRIPTION COMMAND
=========================================================================================================================================================================================
...
install Data collection for general installation problem waiops-mustgather.sh -cypfd -C postgres-edb.sh -Y
NOTE:
To execute the "install" alias:
./waiops-mustgather.sh -O install
(3) If only cluster data is needed, the user can execute the MustGather with only the "-d" option (CLUSTERNODES secondary mode):
./waiops-mustgather.sh -d
(4) By default, the CLUSTERNODES secondary mode ("-d" option) collects the following data:
If the "-S" option is used (to allow collection of sensitive data), the following data is added:
(a)[OS info] cat /etc/os-release
(b)[CPU info] cat /proc/cpuinfo
(c)[Memory info] cat /proc/meminfo
(d)[Disk utilization] df -h
(e)[kubelet status] systemctl status kubelet --no-pager -l
(f)[dmesg] dmesg
(g)[journal] journalctl -b 1 --no-pager -n "$JOURNAL_SIZE" (by default, JOURNAL_SIZE = 2000)
(h)[CRIO images] crictl images
(i)[CRIO containers] crictl ps -a
(j)[Process] ps -auxww
(k)[IP address] ifconfig -a
(l)[/var/lib/kubelet] ls -lR /var/lib/kubelet
(m)[Memory utilization] vmstat
(n)[timedatectl] timedatectl
If the "-S" option is used (to allow collection of sensitive data), the following data is added:
(a) [Firewall/Routing] iptables -L -nv --line-number
(b) [Firewall/Routing] iptables -L -nv -t nat --line-number
(c) [CNI] find /etc/cni/ -type f
(d) [CNI] find /etc/kubernetes/cni/ -type f
If the underlying platform is Openshift (OCP), the following data is added:
(a) [Node sizing] cat /etc/node-sizing-enabled.env
(b) [Node sizing] cat /etc/node-sizing.env
If the underlying platform is k3s, the following data is added:
(a) [k3s logs] cat /var/log/k3s.log
(5) On k3s platform (Linux installation), the following aliases are configured:
# ./waiops-mustgather.sh -O .listalias | grep k3s
k3s-clusterdata (Linux/k3s install) Cluster node data collection waiops-mustgather.sh -DRMd -W 'CLUSTERNODES_SCP_FILES="\\\$HOME/.aiopsctl%%/var/lib/rancher/k3s/agent/containerd/containerd.log"##CLUSTERNODES_SCP_FILES_EXCL="^kubectl$%%"##CLUSTERNODES_SCP_FILES_EXCL_MODE="child%%"'
k3s-install (Linux/k3s install) Data collection for general installation problem waiops-mustgather.sh -cypfd -C postgres-edb.sh -Y -W 'CLUSTERNODES_SCP_FILES="\\\$HOME/.aiopsctl%%/var/lib/rancher/k3s/agent/containerd/containerd.log"##CLUSTERNODES_SCP_FILES_EXCL="kubectl%%"##CLUSTERNODES_SCP_FILES_EXCL_MODE="child%%"'
(6) The cluster data can be found in the 2-CLUSTERNODES directory:
Eg.
# pwd
/tmp/mg/waiops-clusternodes-02092025-062442
# ls -l
total 36
drwxr-xr-x 2 root root 25 Sep 2 06:24 0-README-FIRST
drwxr-xr-x 6 root root 4096 Sep 2 06:24 1-RUNTIME-DATA
drwxr-xr-x 2 root root 24576 Sep 2 06:26 2-CLUSTERNODES
drwxr-xr-x 2 root root 6 Sep 2 06:24 CLUSTER_DATA
drwxr-xr-x 2 root root 6 Sep 2 06:24 CUSTOM_DATA
drwxr-xr-x 2 root root 6 Sep 2 06:24 OTHER_NAMESPACES
drwxr-xr-x 2 root root 6 Sep 2 06:26 _POST_DATA_COLLECT
drwxr-xr-x 6 root root 4096 Sep 2 06:24 PROD_NAMESPACES
# ls -l 2-CLUSTERNODES/
total 27560
-rw-r--r-- 1 root root 9792 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.cpuinfo
-rw-r--r-- 1 root root 11410 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.crictl-images
-rw-r--r-- 1 root root 38831 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.crictl-ps
-rw-r--r-- 1 root root 54609 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.df
-rw-r--r-- 1 root root 81196 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.dmesg
-rw-r--r-- 1 root root 21204 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.ifconfig
-rw-r--r-- 1 root root 287242 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.journalctl-2000
-rw-r--r-- 1 root root 5869 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.kubelet
-rw-r--r-- 1 root root 1587 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.meminfo
-rw-r--r-- 1 root root 76 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.nodesizing
-rw-r--r-- 1 root root 101 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.nodesizing-enabled
-rw-r--r-- 1 root root 254 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.ocdebug.err.0
-rw-r--r-- 1 root root 752 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.os-release
-rw-r--r-- 1 root root 183403 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.ps
-rw-r--r-- 1 root root 359249 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.stats
-rw-r--r-- 1 root root 300 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.timedatectl
-rw-r--r-- 1 root root 244 Sep 2 06:24 master0.waiops410a.cp.fyre.ibm.com.vmstat
...
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSE9G0Q","label":"IBM Cloud Pak for AIOps"},"ARM Category":[{"code":"a8m3p000000LRdhAAG","label":"Watson AIOps-\u003EAI Manager-\u003EScripts-\u003EMustgather"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"4.10.0;4.10.1;4.11.0;4.2.0;4.2.1;4.3.0;4.4.0;4.4.1;4.5.0;4.5.1;4.6.0;4.6.1;4.7.0;4.7.1;4.8.0;4.8.1;4.9.0;4.9.1"}]
Was this topic helpful?
Document Information
Modified date:
02 September 2025
UID
ibm17243777