Debug data collection

IBM Storage Scale Container Storage Interface driver provides the storage-scale-driver-snap.sh tool to collect the debug data.

This tool gathers the state of required Kubernetes resources such as nodes, pods, service accounts, and so on and collects Deployment and DaemonSet logs from all nodes. It collects the definition of resources in the given namespace with the label, product=ibm-spectrum-scale-csi. The collected logs are stored in the given output directory.

Issue the following command and download the tool for CSI 2.14.x: 
curl -O https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.14.4/tools/storage-scale-driver-snap.sh

Usage of the tool

storage-scale-driver-snap.sh [-l | -n | -o | -p | -s | -v | -h]
-l: Collects logs from driver pods that are running only on the same node as sidecar pods. Logs will be collected from all the driver pods if the -l option is not specified.
-n: Namespace from which the debug data for CSI resources is to be collected. If not specified, the default namespace is used. The tool returns error if CSI is not running under the given namespace.
-o: Output directory where debug data will be stored. If not specified, the debug data is stored in current directory.
-p: Previous instance of the container in a pod. If set to False, the logs for the previous instance of the container in a pod are not collected.
-s: Only returns the newer logs, which correspond to a specific time frame like the last 2 hours (2h) or 4 days (4d). By default, returns all logs.
-v: Prints the CSI version that is used on this cluster.
-h: Prints help information about the usage of the tool for debug data collection.

The resultant folder should contain the following file structure with debug information:

├── cluster-scoped-resources
│   ├── clusterrolebindings
│   ├── clusterroles
│   ├── csinodes
│   ├── nodes
│   ├── pv
│   ├── storageclass
│   └── volumeattachment
├── kube-system
│   ├── daemonsets.json
│   ├── deployments.json
│   ├── events.json
│   ├── pods.json
│   ├── replicasets.json
│   ├── replication-controllers.json
│   └── services.json
├── namespaces
│   └── ibm-spectrum-scale-csi-driver
├── nodes.json
└── version
In addition to these logs, the following details are necessary for troubleshooting:
  • Include the following details if an issue is related to PVC:
    kubectl describe namespace < namespace name >
kubectl get pvc < pvc name > -o yaml
kubectl describe pvc < pvc name >
kubectl describe pv < pv name from above pvc >
kubectl describe StorageClass < storage class name for pvc >
  • Include the following details in addition to the previous if an issue is related to a pod:
    kubectl describe pod < pod name >
kubectl get pod < pod name > -o yaml
kubectl describe VolumeAttachment < volume attachment name >
  • Include the following details in addition to the previous if an issue is related to a snapshot:
    kubectl describe VolumeSnapshot < snapshot name >
kubectl get VolumeSnapshot < snapshot name > -o yaml
kubectl describe VolumeSnapshotContent <snapshot content name >
kubectl describe VolumeSnapshotClass < snapshot name >