Log package

A quick overview of the IBM Fusion HCI log packages.

The following log packages are available in the Logs page:
  1. Nodes
  2. Network switches
  3. Global Data Platform
  4. System health check
  5. Administration
  6. Backup and restore
  7. Data Cataloging
  8. Data Foundation

Every log package contains a HealthManagerCR folder expect for the Backup & Restore service.

By default, every log package contains a Must Gather folder except for the on-boarded services such as IBM Data Cataloging, Global Data Platform and Backup & Restore. The Data Foundation service must gather logs collected in the data-foundation-ext log collection set. For more information about Must Gather, see Must Gather section.
Note:
  • The Must Gather folder contains common data that is collected for all the log packages.
  • The isf-serviceability-operator-manager service account uses cluster admin role as it is required to collect oc adm must-gather logs.

Nodes

Logs related to compute nodes, firmware upgrades, and node health. The nodes log package mainly contains the folders for the following namespace:
ibm-spectrum-fusion-ns
It contains the logs and object definition files for all major resources in this namespace. For example, routes, pods logs, deployments, deamon sets, persistent volumes, and config maps.
Note: It includes the Kubernetes events that are associated with the resource. If any resource does not exist, then skip it.
openshift-machine-api
It contains the logs and object definition files for all major resources in this namespace. For example, routes, pods logs, deployments, deamon sets, persistent volumes, and config maps.
openshift-machine-config-operator
It contains the logs and object definition files for all major resources in this namespace. For example, routes, pods logs, deployments, deamon sets, persistent volumes, and config maps.
IMM-logs
Important: The IMM logs package is collected only for the Lenovo nodes.
The bmc-logs logs folder contains the IMM_NAME.ffdc and IMM_NAME.inventory files for every IMM.
Note: The log contents vary based on the Node vendor imm logs for Lenovo and Support Assist package for Dell.
Dell Support Assist logs
Important: The Dell Support Assist Logs package is collected only for the Dell nodes.
The Dell support assist logs contains the debug logs, hardware logs, and storage logs for the Dell nodes.
OCP must gather
The folder contains OpenShift® Container Platform must-gather logs for analysis.

Nodes log packages also contains the custom resources such as baremetalhosts, machinesets, machines, spectrumfusions, computeprovisionworkers, computemonitorings, rackinfoes, computeinits, computeconfigurations, computemaintenances, computepowerops, computediscoveries, computefirmwares, and updatemanagers.

Network switches

Logs related to appliance network errors and connectivity. The network switches log package mainly contains the folders for the following namespaces:

ibm-spectrum-fusion-ns
It contains the logs and object definition files for all major resources in this namespace. For example, routes, pods logs, deployments, deamon sets, persistent volumes, and config maps.
Note: It includes the Kubernetes events that are associated with the resource. If any resource does not exist, then skip it.
openshift-monitoring
It contains the logs and object definition files for all major resources in this namespace. For example, routes, pods logs, deployments, deamon sets, persistent volumes, and config maps.
Vlans
The Vlans folder contains a vlans.json file that includes a Vlan list.
Links
The Links directory contains a links.json file that includes a link list.
Switch
The Switch directory contains a switches.json file that includes only the definition of the switches.
Switch logs
The switch logs contain the following three directories that include the actual switch logs:
  • clncl
  • genic
  • scpcl

Network log packages also contain the custom resources such as networkinits, switchfirmwares, network-attachment-definitions, and updatemanagers.

Global Data Platform

Logs related to storage availability. The Global Data Platform log package mainly contains the folders for the following namespaces:
Important: By default, it collects the logs from the IBM Fusion namespace and the namespace where the service is deployed.
  • ibm-spectrum-scale-namespace
  • ibm-spectrum-scale-csi-namespace
  • ibm-spectrum-scale-operator-namespace
  • openshift-monitoring

Global Data Platform log packages contain both the custom and cluster resources. The custom resources such as spectrumfusions, scalemanagers, scaleclusters, and nodes. The cluster resources such as storageclasses.

Follow the steps to collect Velero namespace logs:
  1. On the Global Data Platform service, manually add the Velero namespace under the namespaces in the serviceability section of the FusionServiceDefinition instance.
  2. It ensures that the Velero namespace logs are collected as part of log collection for the Global Data Platform service.
  3. The Velero namespace is specified in ramen-dr-cluster-operator-config configmap in the ibm-spectrum-fusion-ns namespace.
It also includes the GPFS snap logs and must-gather that is related to IBM Storage Scale.
Important: To obtain GPFS logs, gather them from IBM Fusion HCI user interface and not from OpenShift Container Platform.

System health check

Logs related to overall assessment of the health of IBM Fusion HCI. The system health check log package mainly contains the folders for the following namespaces:

Node Metrics
It contains the object definition files for the NodeMetricsList type.
ibm-spectrum-fusion-pod-metric
It contains the PodMetrics files for every pod present in this namespace.
Switch logs
The switch logs contain the following three directories that include the actual switch logs:
  • clncl
  • genic
  • scpcl

It also includes the Baremetalhosts, Machinesets, Machines, and Node network configuration policy folders with the relevant log files.

Administration

Logs related to Red Hat® OpenShift, administration, and audit logging. The administration log package mainly contains the folder for the following namespace:

Audit logs
Audit logs contain the Control-0-rack, Kube, OAuth, and OpenShift folders with the relevant log files.

Backup and restore

Logs related to the backup and restore of your workload applications. The backup and restore log package mainly contains the folders for the following namespaces:
Important: By default, it collects the logs from the IBM Fusion namespace and the namespace where the service is deployed.
ibm-backup-restore
It contains the logs and object definition files for all major resources in this namespace. For example, routes, pods logs, deployments, deamon sets, persistent volumes, and config maps.
Note: It includes the Kubernetes events that are associated with the resource. If any resource does not exist, then it can be skipped.

Data Cataloging

Logs related to the IBM Data Cataloging service. The IBM Data Cataloging log package mainly contains the folders for the following namespace:
Important: By default, it collects the logs from the IBM Fusion namespace and the namespace where the service is deployed.
ibm-data-cataloging
It contains the logs and object definition files for all major resources in this namespace. For example, routes, pods logs, deployments, deamon sets, persistent volumes, and config maps.
Note: It includes the Kubernetes events that are associated with the resource. If any resource does not exist, then skip it.

Data Foundation

Logs related to storage availability. The Red Hat OpenShift Data Foundation log package mainly contains the folders for the following namespaces:
Important: By default, it collects the logs from the IBM Fusion namespace and the namespace where the service is deployed.

Red Hat OpenShift Data Foundation log packages contains both the custom and cluster resources. The custom resources such as spectrumfusions, odfmanagers, odfclusters, fusionservicedefinitions, fusionserviceinstances, storageclusters, cephclusters, localvolumediscoveries, localvolumediscoveryresults, localvolumesets, catalogsources, volumesnapshotclasses, and clustercsidrivers.

The cluster resources such as persistentvolumes, clusterrolebindings, storageclasses, and storageclassclaims.

Follow the steps to collect Velero namespace logs:
  1. On the Data Foundation service, manually add the Velero namespace under the namespaces in the serviceability section of the FusionServiceDefinition instance.
  2. It ensures that the Velero namespace logs are collected as part of log collection for the Data Foundation service.
  3. The Velero namespace is specified in ramen-dr-cluster-operator-config configmap in the ibm-spectrum-fusion-ns namespace.

It includes the Openshift-storage and Openshift-local-storage folder with the relevant log files.

Note: Though Data Foundation log status shows partial due to the unavailability of one or more namespaces, you can still upload and download partial logs.

Must Gather

The Must Gather folder contains the common data that is collected for all the log packages. It mainly contains the following files, which might be helpful in debugging to get more information about the environment. It contains information about nodes, nodes config, kickstart.json, rackinfo config map, and persistent volumes related files.