IBM Cloud Private (ICP) healthcheck tool

Product Documentation

Abstract

In IBM Cloud Private 3.1.0 and beyond, the healthcheck command runs the bulk collection of IBM Cloud Private platform node system logs, pod logs, and container logs.

Content

To run the "healthcheck" command, run the following commands from the cluster folder on the boot node.
For example, "cd /opt/ibm-cloud-private-3.2.1/cluster/"

Remove any old health check files to reduce the sized of the propagated tarZ file
```
rm -rf logs/healthcheck/*
```

Create the health check folders.

mkdir -p logs/healthcheck/pods logs/healthcheck/pv logs/healthcheck/system

If the IBM Cloud Private version is before 3.2.0, run the following command to collect the node descriptions. Otherwise, move on to Step 4.
```
kubectl describe nodes > logs/healthcheck/nodes-description.txt 
kubectl get nodes -o wide > logs/healthcheck/nodes-status.txt 
```
Collect the cluster logs with the healthcheck command.
```
sudo docker run --net=host -t -e LICENSE=accept -v "$(pwd)":/installer/cluster ibmcom/icp-inception-$(uname -m | sed 's/x86_64/amd64/'):<version> healthcheck -v 
```
Replace the "version" with the ibmcom/icp-inception-* image deployed on your registry.
You can find the version by running the command. docker images | grep inception
Archive the logs for upload and make sure to include the config.yaml and the hosts' file
```
 tar czf healthcheck.tgz ./logs/ config.yaml hosts
```
Upload the healthcheck.tgz archive to the case.

Notes:

The healthcheck command in Step 4 runs an Ansible script similar to the installation and places the log files in the <installationirectory>/cluster/logs/healthcheck folder. Only kubs-syster related information is gathered using the healthcheck
In case anything went wrong with the healthcheck command, a log of the script actions is also collected <installation-directory>/cluster/logs/ in the file healthcheck.log.<date>. Increase the logging levels by appending more v's (up to four) to the end of the command (-vvv).

Using the script to collect cluster information:

The healthcheck tool does not include logs from user-deployed applications or system logs. If you need to share the logs for the pods non kube-system namespace , you can use the following script to collect information for all namespaces if requested by the IBM support team.

Save the following shell script in a file. For example "MG-ICP.sh" maintaining the space for the loops

 
  
export MGDIR=icp-logs-$(date '+%y%b%dT%H-%M-%S')
export LOGLIMIT="--tail=1000"
mkdir -p $MGDIR
kubectl top nodes > $MGDIR/topnode-list.txt
kubectl get nodes -o wide > $MGDIR/getnode-list.txt
kubectl describe nodes > $MGDIR/node-describe.txt
kubectl get namespaces > $MGDIR/namespaces.txt
kubectl get pods --all-namespaces -o wide > $MGDIR/all-pods-list.txt
kubectl get pods --all-namespaces -o wide | grep -v Running | grep -v Completed > $MGDIR/notrunning-pods-list.txt
for NS in `kubectl get ns | awk 'NR>1{ORS=" "; print $1}'`; do
 export NS=$NS; mkdir $MGDIR/$NS; echo gathering info from namespace $NS
 kubectl get all,secrets,cm,ing,events -n $NS -o wide &> $MGDIR/$NS/all-list.txt
 kubectl get pods -n $NS | awk 'NR>1{print "kubectl -n $NS describe pod "$1" > $MGDIR/$NS/"$1"-describe.txt && echo described "$1}' | bash
 kubectl get pods -n $NS -o go-template='{{range $i := .items}}{{range $c := $i.spec.containers}}{{println $i.metadata.name $c.name}}{{end}}{{end}}' > $MGDIR/$NS/container-list.txt
 awk '{print "kubectl -n $NS logs "$1" -c "$2" $LOGLIMIT -p > $MGDIR/$NS/"$1"_"$2"_previous.log && echo gathered previous logs of "$1"_"$2}' $MGDIR/$NS/container-list.txt | bash
 awk '{print "kubectl -n $NS logs "$1" -c "$2" $LOGLIMIT > $MGDIR/$NS/"$1"_"$2".log && echo gathered logs of "$1"_"$2}' $MGDIR/$NS/container-list.txt | bash
done

Change permission to execute "chmod +x MG-ICP.sh" and run the script.

 
tar czf $MGDIR.tgz $MGDIR/

Upload the file to your IBM case

Related Information

IBM Cloud Private MustGather to collect logs , obtain support and troubleshooti…

Document Location

Worldwide

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBS6K","label":"IBM Cloud Private"},"Component":"Troubleshooting","Platform":[{"code":"PF016","label":"Linux"}],"Version":"3.1.0+","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Product Synonym

IBM Cloud Private; MustGather; ICP, debug; healthcheck

Was this topic helpful?

Document Information

Modified date:
26 September 2022

UID

ibm10870296

Tips