Logging and monitoring
IBM Fusion Data Foundation uses built-in capabilities for monitoring and logging as provided by the Red Hat OpenShift Container Platform.
Once deployed in IBM Strage Fusion, the Red Hat OpenShift Data Foundation operator manifests
itself as a new IBM Fusion Data Foundation navigation entry in the Red Hat
OpenShift administrator’s view under the Storage main entry. This view provides a
handy "USE" (Usage-Saturation-Events) dashboard for the operational status of IBM Fusion Data
Foundation.
Data for these visualizations is gathered through the built-in monitoring stack. Metrics are
exported from the NooBaa and the S3 service monitors, the ODF operator for the default ODF storage
cluster Storage System, and the Rook Ceph manager. Detailed metrics can be obtained through the
Administrator’s Observe / Metrics UI in the Red Hat OpenShift Console.
In addition to the storage dashboard, the Red Hat OpenShift Data Foundation Operator extends
built-in monitoring with several alerts. These alerts are named after the respective service. The
interested administrator can find them by navigating to Observe / Alert - Alerting
Rules and using Ceph or Noobaa as name filter.
Normal Red Hat OpenShift Logging mechanisms are available. Log information from Red Hat OpenShift
Data Foundation-related Pods can be retrieved through the usual oc logs
<pod-name> command. However, a consolidated logging experience is achieved by using Red
Hat OpenShift Logging.
However, as Red Hat OpenShift Data Foundation provides an integrated storage solution for the cluster, it is worth customizing the monitoring and optionally, the logging environment.
Monitoring Data Persistence using Red Hat OpenShift Data Foundation
When the Red Hat OpenShift monitoring stack is deployed initially to a new Red Hat OpenShift cluster, it does not start with persistent data stores for the collected metrics data. This means that whenever one of the provided Prometheus Pods is rescheduled, its own Time Series DataBase (TSDB) is re-created from scratch. In other words, historical monitoring data is lost. Notably, for production clusters, it is essential to keep persistent monitoring data. This aspect is not only important for Red Hat OpenShift Data Foundation monitoring, but also for monitoring the whole Red Hat OpenShift cluster. Fortunately, with the presence of Red Hat OpenShift Data Foundation, it is easy to persist monitoring data.
The OpenShift documentation on monitoring describes how to set up persistent storage for the various components of the monitoring stack. While the documentation suggests the use of Physical Volumes (PV) obtained through the Local Storage Operator (LSO), the Red Hat OpenShift Data Foundation solution provides some advantages. Notably consider the following: While LSO provided volumes are bound to distinguished cluster nodes, Pods for the monitoring components will be finally tied to the cluster node where their respective data resides. But with Red Hat OpenShift Data Foundation volumes, this restriction is no longer true: The data can "move" to the node where the Pod resides, which dramatically improves the flexibility and resiliency of the monitoring stack.
Using ocs-storagecluster-ceph-rbd as storageClassName instead
of the suggested local-storage value in the documentation example places the
monitoring data to Red Hat OpenShift Data Foundation provisioned volumes. As this storage class also
provides volume expansion, it is easy to adjust the PVC size to the actual required size for the
desired monitoring retention time. This is clearly a further
advantage over the use of LSO provisioned PVCs.
Consolidated Logging using Red Hat OpenShift Logging Operator with Red Hat OpenShift Data Foundation
As mentioned before, plain Kubernetes / Red Hat OpenShift logging on container level is by default available. However, a consolidated logging solution based on the Red Hat OpenShift Logging Operator is a recommended way to manage production environments. Similar to the monitoring stack, logging data also should be persisted. Again, Red Hat OpenShift Data Foundation is helpful here.
Logging(Red Hat documentation) shows how to connect the
logstore entities with their Persistent Volume Claims. Using the
ocs-storagecluster-ceph-rbd storage class for Red Hat OpenShift Data Foundation
provided storage gives the same flexibility and fault tolerance as discussed in the previous section
on monitoring data.