IBM Support

OpenShift Monitoring

White Papers


Abstract

Prometheus, Grafana and Alertmanager is the default platform monitoring stack in Redhat Openshift/IBM Cloud Pak for Data (CP4D). This white paper presents a case study on how to use this stack to monitor both infrastructure and application components which is a crucial Day 2 operation to ensure system availability and performance. The use case 1 focuses on Portworx Storage and use case 2 describes monitoring/alerting on etcd cluster which is a critical metadata store in many CP4D applications.

Content

  




 

Monitoring OpenShift in IBM Cloud Pak for Data

Abdul Hafeez, Chengmin Ding, David Sagal

IBM Corporation, Herndon VA





























 

 

Table of Contents

Introduction

A Closer Look At Prometheus, Grafana and Alertmanager

  Prometheus and ServiceMonitors

  Grafana

  Alertmanager and Alert Rules

Prerequisites and Assumptions

Example Use Case 1: Portworx Storage Monitoring

  Prometheus Setup

  Grafana Setup

  A Customized Portworx Volume Dashboard in Grafana

  Alertmanager Setup

  Testing Alertmanager with Dbench

Example Use Case 2: etcd Monitoring

  Prometheus Setup

  Grafana Setup

  Alertmanager Setup

  Testing etcd Alerts with Dbench/FIO

Conclusion

References

 

List of Figures

Figure 1: A representation of the architecture of IBM Cloud Pak for Data

Figure 2: A representation of the architecture of RedHat OpenShift

Figure 3: A representation of the architecture of Prometheus

Figure 3.1: A representation of the Prometheus Operator’s interaction with ServiceMonitors

Figure 3.2: The Service object which needs to be monitored

Figure 3.3: The ServiceMonitor for the Service in Figure 3.2

Figure 4: An example of a Grafana dashboard containing several different types of panels

Figure 5: An example of an alert rule

Figure 6.1: A ServiceMonitor for Portworx

Figure 6.2: The Portworx service corresponding to the ServiceMonitor in figure 6.1

Figure 7: Portworx metrics shown in the Prometheus UI

Figure 8.1: Navigating to the create->import option in the Grafana UI

Figure 8.2: Pasting in the contents of a JSON configuration file

Figure 8.3: Selecting Prometheus as the data source for a dashboard

Figure 8.4: A newly created dashboard visible in the list of dashboards

Figure 8.5: Reaching the "save as" button through the settings menu

Figure 8.6: Naming our new copy of the Portworx Volume Dashboard

Figure 8.7: Accessing the "Volumes" variable in order to edit it

Figure 8.8: The newly edited regular expression and the result of the change

Figure 8.9: Our new variable label in our dashboard

Figure 8.10: Creating a new row

Figure 8.11: A newly created row

Figure 8.12: Copying a panel

Figure 8.13: Using the paste option to paste a panel

Figure 8.14: Result after moving a panel into a row

Figure 8.15: The menu used for editing a panel

Figure 8.16: The working panel after modifying the PromQL expression

Figure 8.17: Selecting the repeat option for each value of the "Volumes" variable

Figure 8.18: Our panel's title after having the "Volumes" variable's value added to it

Figure 8.19: The new dashboard with only 1 PVC name selected

Figure 8.20: The new dashboard with 3 PVC names selected

Figure 9: Portworx alert rules shown in the Prometheus UI

Figure 9.1: Retrieving the Slack Incoming Webhook URL 

Figure 10.1: The dbench-pv-claim pvc and dbench job being created

Figure 10.2: A newly created pvc and its associated volume name

Figure 10.3: The volume selected in the dropdown menu and its activity its volume usage

Figure 10.4: A firing alert in slack

Figure 11.1: A new instance of etcd being shown in Prometheus after our ServiceMonitor is installed

Figure 11.2: The script used to install FIO being run

Figure 11.3: Four screen windows are created

Figure 11.4: The metric being shown in Prometheus has surpassed a value of 8 which should trigger our example alert

Figure 11.5: Our example etcd alert firing in Slack

List of Configuration Files

Introduction

IBM Cloud Pak for Data (CP4D) is the new Data & AI middleware that provides a comprehensive set of preintegrated software for big data integration and analytics needs.  It is an open platform that can have additional addons or custom applications to be deployed on top of it.

It is a complicated system and has many layers as seen in figure 1. In order to support smooth daily operations of the platform or quickly identify the root cause of a problem, it is important to monitor and collect information from different layers as it is very difficult to determine the cause of an issue if there is not much context available. For example, in cases where strange behavior is occurring in certain applications, if metrics are only being collected at the cartridge (or addon) level, the only thing that is known for sure is that there is issue occurring at application UI and difficult to identify the real problem. If metrics are collected at the control plane level, such as from RedHat OpenShift, there can be much more information uncovered about the root cause of the problem.

image-20200424163255-1

Figure 1 - A representation of the architecture of IBM Cloud Pak for Data

From a monitoring perspective, we can focus on the OpenShift layer that supports CP4D. In figure 2, we can see that there are more sublayers within the OpenShift layer.

image-20200424163255-2

Figure 2 - A representation of the architecture of RedHat OpenShift

To create an effective monitoring system, we can use the three components recommended in the official OpenShift monitoring documentation [12].
 

  • We will need to collect and store metrics from the components that we want to watch. To do this, we will use Prometheus.
  • We will then need to visualize the data we’ve collected from our system using dashboards we create with Grafana.
  • We’ll also need our monitoring system to send us alerts when there is a critical issue. We can create alerts with Alertmanager

A Closer Look at Prometheus, Grafana and Alertmanager


Prometheus is a systems monitoring tool that we can use to collect and store metrics, which can then be used with tools like Alertmanager (which is part of the Prometheus ecosystem) and Grafana. These metrics are stored as time-series data. The Prometheus ecosystem consists of many different parts which are shown in figure 3. Some key components of this ecosystem that we can leverage are:
 

  • The Prometheus server which collects (by pulling) and stores data.
  • An alert system which creates rules based on Prometheus metrics and pushes alerts to the Alertmanager
  • PromQL, a query language which can be used to create specific expressions that can be used by dashboarding solutions like Grafana

image-20200424163255-3

Figure 3 - A representation of the architecture of Prometheus

A ServiceMonitor is a Kubernetes custom resource definition that monitors different services in the environment and is watched by the Prometheus Operator. This means that through a ServiceMonitor, we can monitor a variety of different applications.


 

image-20200424163255-4

Figure 3.1 - A representation of the Prometheus Operators's interaction with ServiceMonitors

To correctly configure a ServiceMonitor to work with the service it monitors, there are a few things that need to be taken into account.


image-20200424163255-5                              image-20200424163255-6

Figure 3.2 - The Service object which needs to be monitored                                                                               Figure 3.3 - The ServiceMonitor for "example-app"

The ServiceMonitor will use its label selector to determine which service objects to monitor. In our example, the service object in figure 3.2 has the label, “app: example-app”. To match that in the corresponding ServiceMonitor in figure 3.3, under the label selector, we must add the same label, “app: example-app”.

  

Grafana is a visualization and analytics software. It can take data from a time series database (such as Prometheus), and create elegant dashboards made up of panels. Panels are based on queries. There are many different types of panels, and each of them are suited better for different types of data. These types include graphs, tables, stats, gauges, bar gauges, logs, singlestat and heatmap. There are many premade dashboards available on Grafana’s website which can simply be imported. Panels and dashboards can also be defined using JSON files or directly entering expressions into each panel.

image-20200424163255-7

Figure 4 - An example of a Grafana dashboard containing several different types of panels

 

Alertmanager is a tool that manages alerts sent by applications like Prometheus. Its main features include grouping, silencing and inhibitions. It can group alerts based on certain properties they have. For example, if there is a case where hundreds of services fail, instead of getting hundreds of separate alert notifications, you’d get one compact notification. Silences can simply mute certain alerts for a period of time. If an alert meets the specifications of an active silence, it will not be sent. Inhibitions are a way of stopping certain alerts from firing when there are certain other alerts firing. For example, if there is an alert that is triggered by a cluster not being reachable, then all other alerts from that cluster will be muted because we already know that the cluster cannot be reached along with anything it contains. Alert rules’ expressions are the rules that are evaluated to determine the need for an alert. These are defined in our Prometheus instance by a YAML file as seen in figure 5.

image-20200424163255-8

Figure 5 - An example of an alert rule

Prerequisites and Assumptions
This document assumes that the required components (Prometheus, Grafana, and Alertmanager) have already been installed and configured correctly during the deployment of OpenShift and that all ports and protocols required for access to the installed components have been configured. 

A comprehensive list of ports Prometheus uses can be found here although the range for core components is 9090-9094 and for exporters, 9100-9999 can be used.
Grafana does not need any additional configuration from a ports perspective provided the route to the service was configured during the deploment.
Alertmanager may need to use tcp port 25 for access to a mail server or to leverage Slack, it will require port 443 for the webhook API. Other than that, it should not need any additional configuration beyond what is configured for prometheus. 

Example Use Case 1: Portworx Storage Monitoring

Prometheus Setup

The first step to monitoring a service is to allow Prometheus to watch it. To do this, a ServiceMonitor can be created by following the steps below:

Create a file named “service-monitor.yaml” and enter the following contents:

  apiVersion: monitoring.coreos.com/v1  kind: ServiceMonitor  metadata:    labels:      k8s-app: portworx-prometheus-sm    name: portworx-prometheus-sm    namespace: openshift-monitoring  spec:    endpoints:    - port: px-api      targetPort: 9001   - port: px-kvdb      targetPort: 9019    namespaceSelector:      matchNames:      - kube-system    selector:      matchLabels:        name: portworx

Configuration file 1: A .yaml file used to configure a ServiceMonitor to monitor Portworx

Once the service-monitor.yaml file has been created, it can be applied with the following command:

  oc apply -f service-monitor.yaml

Now that the YAML file has been applied, we should see something similar to the ServiceMonitor object in figure 6.1.

image-20200424163255-9        image-20200424163255-10         
Figure 6.1 - Our new Portworx ServiceMonitor.                                                          Figure 6.2 – portworx, our corresponding service object

Notice that the label selector specifies the same label as the “portworx” service object, since that it what’s being monitored.

Now Portworx metrics should be visible in the Prometheus UI as seen in figure 7.

image-20200424163255-11

Figure 7 - Portworx metrics shown in the Prometheus UI

 

Grafana Setup

Once the metrics are being collected by Prometheus, we can now use those metrics in Grafana. Regarding dashboards in Grafana, we can either create one based on PromQL expressions, or import one using a JSON file. In this case, JSON configurations of a few different dashboards have already been provided by Portworx [13].

The first step is to hover over the plus sign on the left side of the Grafana Home Dashboard, then click on import.


image-20200424163255-12  image-20200424163255-13
Figure 8.1 – Navigating to the create->import option in the Grafana UI

Once we’ve reached the import screen, we can simply paste in the contents of the JSON file from Portworx, then click load.

image-20200424163255-14
Figure 8.2 - Pasting in the contents of the JSON configuration file

Once we reach the options screen, we have to select Prometheus as the data source.

image-20200424163255-15
Figure 8.3 - Select Prometheus as the data source for this dashboard

Once we click the green import button, our dashboard should be visible in the dashboards section on the left menu.

image-20200424163255-16 image-20200424163255-17
Figure 8.4 - Our new dashboard highlighted by a red arrow(left) and our dashboard(right).

 A Customized Portworx Volume Dashboard in Grafana
Motivation for a Customized Dashboard
One of the benefits of using Grafana is that the user can customize dashboards and rearrange them in order to monitor more specific pieces of data. For example, we could have a scenario where we want to monitor a few metrics from a few different Portworx volumes, and compare them beside each other. One of the big issues with the pre-made Portworx volume dashboard is that it shows volumes by their volume name(something like "pvc-12414-345325235"), which can be hard to identify quickly. To be able to easily identify volumes associated with certain applications, we'll need something a little bit different from the default Portworx volume dashboard, which shows data from across the entire cluster, and is not built for monitoring events of any specific task.
Creating the Dashboard
The pre-made Portworx volume dashboard has a lot of great metrics in it, so we can just modify it to suit our needs. The first step to this is creating a copy of the Portworx Volume Dashboard. To create a copy of a dashboard, we open our dashboard, then click on the settings gear in the top right corner, and click on the "save as" button in the settings menu.
customgrafanafigure1_0.png  customgrafanafigure2.png
Figure 8.5 - Reaching the "save as" button through the settings menu
We can rename our new copy of the dashboard to something a little bit more descriptive as we save it.
customgrafanafigure3.png
Figure 8.6 - Naming our new copy of the Portworx Volume Dashboard
Now that we have a copy of the dashboard, we can safely modify it without ruining the original copy. 
Modifying a Variable
To make the data more easily readable by humans, we can change the "Volumes" variable. To do this, we must once again click on the settings gear in the top right corner of the dashboard. Once we're in settings, we need to click on the "Variables" section on the left side menu. Then we can click on the "Volumes" variable to edit it. 
volumesvariable1.pngvolumevariable2.png
Figure 8.7 - Accessing the "Volumes" variable in order to edit it
The variable is chosen from a regular expression which is applied on the output of Prometheus when a certain PromQL query is entered. This output can also be seen in the Prometheus web UI in the graph/console once a query/expression has been entered. To change the variables in order to make them more expressive, we can modify the regex to search for a PVC name instead of a volume name.
volumevariable3.png
Figure 8.8 - The newly edited regular expression and the result of the change.
Now that we've changed the regular expression and label, at the bottom of the variable's menu, the preview of values should be a set of descriptive names instead of just a clump of numbers and letters. We should also be able to see the new label in the dropdown menu in the dashboard as seen in figure 10.9. 
volumevariable4.png
Figure 8.9 - Our new variable label in our dashboard
Creating a New Row
Now, if we collapse the "All Volumes" and "Volume: ----" rows, we can see that there are two total rows here. We can add a new row to include a section of more specific metrics we want to look at. To do this, click on the "add panel" icon in the top right corner of the dashboard and select "row".
addrow1
Figure 8.10 - Creating a new row
Now we should see our new row created above the old ones with a default name. By clicking on the gear icon to the right of the row name, we can rename the row.
addrow2
Figure 8.11 - A newly created row
Creating a new panel
Now that we have a new row, we need to populate it with new panels. To do this, we'll be copying some panels from the "Volume: ----" row and modifying them. First, we have to open that row, and select a panel we want to duplicate. We can start with the Volume Throughput panel. First click on the down arrow next to its name, select "more" and then "copy"
addpanel1
Figure 8.12- Copying a panel
Now that we've copied the panel, we can go back into our new row, click on the "add panel" icon again and choose the "paste" option. The previously copied panel's name should be there, and can be clicked in order to make a copy of that panel.
addpanel2
Figure 8.13 - Using the paste option to paste a panel
Now we can simply drag the new panel into our new row.
addpanel3
Figure 8.14 - Result after moving a new panel into a row
Modifying a panel
We can see that there are no data points in this panel (and all other panels that use the Volume variable). That is because the panel's PromQL expression is still using the "Volumes" variable under the pretense that it refers to volume names, but we changed it to refer to PVC names. To fix this issue, we can click on the panel's name and click on the "edit" option in the dropdown menu.
editpanel1
Figure 8.15- The menu used for editing a panel
The important segment of the expression is {volumename=~"[[Volumes]]"}. The expression will only include whichever volume name matches the value of the selected "Volume" variable. To correct this, we can change it to {pvc=~"[[Volumes]]"}.
editpanel2
Figure 8.16 - The working panel after modifying the PromQL expression
We also want to be able to compare metrics for different volumes. To help with this, we can go to the "General" tab and make the panel repeat for each value of "Volumes" horizontally. This way, whenever we select a new volume in the PVC Name dropdown menu at the top of our dashboard, a new instance of this panel will be generated horizontally, and we'll be able to see the same metric for different volumes side by side. 
editpanel3
Figure 8.17 - Selecting the repeat option for each value of the "Volumes" variable
We can also change the name of the panel to include the name of each PVC. We do this by adding [[Volumes]] to the title. This is one way to refer to a variable. Once that has been added to the title of the panel, it will display the value of the variable that is used to filter that query.
editpanel4
Figure 8.18 - Our panel title after having the "Volumes" variable's value added to it.
Final Dashboard
We can repeat the entire process for each of the panels that we want to copy over and modify. In the end, the dashboard will look similar to what we have in figures 8.19 and 8.20.
finaldashboard1
Figure 8.19 - The new dashboard with only one PVC name selected
finaldashboard2
Figure 8.20 - The new dashboard with 3 PVC names selected
​

Alertmanager Setup

In order to get alerts about Portworx, we need Prometheus to use alert rules specific to Portworx. Portworx has already provided a set of useful alerts [14]. To use these alerts, we need a yaml file with alert rule expressions as seen in figure 5.

First, we need to create a file named “prometheus-rules.yaml” and enter the following contents:

  apiVersion: monitoring.coreos.com/v1  kind: PrometheusRule  metadata:     labels:      prometheus: k8s      role: alert-rules    name: prometheus-rules      namespace: openshift-monitoring   spec:    groups:    - name: portworx.rules      rules:      - alert: PortworxVolumeUsageCritical        annotations:          description: Portworx volume {{$labels.volumeid}} on {{$labels.host}} is over            80% used for more than 10 minutes.          summary: Portworx volume capacity is at {{$value}}% used.        expr: 100 * (px_volume_usage_bytes / px_volume_capacity_bytes) > 80        for: 5m        labels:          issue: Portworx volume {{$labels.volumeid}} usage on {{$labels.host}} is high.          severity: critical      - alert: PortworxVolumeUsage        annotations:          description: Portworx volume {{$labels.volumeid}} on {{$labels.host}} is over            70% used for more than 10 minutes.          summary: Portworx volume {{$labels.volumeid}} on {{$labels.host}} is at {{$value}}%            used.        expr: 100 * (px_volume_usage_bytes / px_volume_capacity_bytes) > 70        for: 5m        labels:          issue: Portworx volume {{$labels.volumeid}} usage on {{$labels.host}} is critical.          severity: warning      - alert: PortworxVolumeWillFill        annotations:          description: Disk volume {{$labels.volumeid}} on {{$labels.host}} is over            70% full and has been predicted to fill within 2 weeks for more than 10            minutes.          summary: Portworx volume {{$labels.volumeid}} on {{$labels.host}} is over            70% full and is predicted to fill within 2 weeks.        expr: (px_volume_usage_bytes / px_volume_capacity_bytes) > 0.7 and predict_linear(px_cluster_disk_available_bytes[1h],          14 * 86400) < 0        for: 10m        labels:          issue: Disk volume {{$labels.volumeid}} on {{$labels.host}} is predicted to            fill within 2 weeks.          severity: warning      - alert: PortworxStorageUsageCritical        annotations:          description: Portworx storage {{$labels.volumeid}} on {{$labels.host}} is            over 80% used for more than 10 minutes.          summary: Portworx storage capacity is at {{$value}}% used.        expr: 100 * (1 - px_cluster_disk_utilized_bytes / px_cluster_disk_total_bytes)          < 20        for: 5m        labels:          issue: Portworx storage {{$labels.volumeid}} usage on {{$labels.host}} is            high.          severity: critical      - alert: PortworxStorageUsage        annotations:          description: Portworx storage {{$labels.volumeid}} on {{$labels.host}} is            over 70% used for more than 10 minutes.          summary: Portworx storage {{$labels.volumeid}} on {{$labels.host}} is at {{$value}}%            used.        expr: 100 * (1 - (px_cluster_disk_utilized_bytes / px_cluster_disk_total_bytes))          < 30        for: 5m        labels:          issue: Portworx storage {{$labels.volumeid}} usage on {{$labels.host}} is            critical.          severity: warning      - alert: PortworxStorageWillFill        annotations:          description: Portworx storage {{$labels.volumeid}} on {{$labels.host}} is            over 70% full and has been predicted to fill within 2 weeks for more than            10 minutes.          summary: Portworx storage {{$labels.volumeid}} on {{$labels.host}} is over            70% full and is predicted to fill within 2 weeks.        expr: (100 * (1 - (px_cluster_disk_utilized_bytes / px_cluster_disk_total_bytes)))          < 30 and predict_linear(px_cluster_disk_available_bytes[1h], 14 * 86400) <          0        for: 10m        labels:          issue: Portworx storage {{$labels.volumeid}} on {{$labels.host}} is predicted            to fill within 2 weeks.          severity: warning      - alert: PortworxStorageNodeDown        annotations:          description: Portworx Storage Node has been offline for more than 5 minutes.          summary: Portworx Storage Node is Offline.        expr: max(px_cluster_status_nodes_storage_down) > 0        for: 5m        labels:          issue: Portworx Storage Node is Offline.          severity: critical      - alert: PortworxQuorumUnhealthy        annotations:          description: Portworx cluster Quorum Unhealthy for more than 5 minutes.          summary: Portworx Quorum Unhealthy.        expr: max(px_cluster_status_cluster_quorum) > 1        for: 5m        labels:          issue: Portworx Quorum Unhealthy.          severity: critical      - alert: PortworxMemberDown        annotations:          description: Portworx cluster member(s) has(have) been down for more than            5 minutes.          summary: Portworx cluster member(s) is(are) down.        expr: (max(px_cluster_status_cluster_size) - count(px_cluster_status_cluster_size))          > 0        for: 5m        labels:          issue: Portworx cluster member(s) is(are) down.          severity: critical

Configuration File 2: A .yaml file used to create rules for Prometheus to use in order to alert

Now the following command can be used to apply these rules:

  oc apply -f prometheus-rules.yaml


 

Under the “alerts” tab in the Prometheus UI, we can see our newly applied Portworx alert rules.

image-20200424163255-18
Figure 9 – Portworx alert rules shown in the Prometheus UI

Now we need a way to receive notifications of firing alerts. It’s convenient to receive alerts through incoming webhooks in Slack. We need to make some configuration changes to Alertmanager to do that. First, we need to add Incoming Webhooks into a Slack workspace and retrieve our webhook URL. To do this, go to Slack's app directory and add the Incoming Webhooks app to your workspace. From there, the app can be edited and added to a channel. In the edit menu, there is a field containing the webhook URL. We need to note this URL because we'll need to use it in our Alertmanager configuration.

slack_iw
Figure 9.1 - Retrieving the Slack Incoming Webhook URL

To create a new configuration, create a file named “alertmanager.yaml” and enter the following contents:

  global:      http_config:       proxy_url: <proxy_server_url>     route:    group_by: ['alertname']    group_wait: 10s    group_interval: 10s    receiver: 'slack-notifications'     receivers:  - name: 'slack-notifications'    slack_configs:    - api_url: <slack_incoming_webhook_url>      username: 'Alertmanager'      channel: <slack_channel_name>      send_resolved: true      title: |-        [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}        {{- if gt (len .CommonLabels) (len .GroupLabels) -}}          {{" "}}(          {{- with .CommonLabels.Remove .GroupLabels.Names }}            {{- range $index, $label := .SortedPairs -}}              {{ if $index }}, {{ end }}              {{- $label.Name }}="{{ $label.Value -}}"            {{- end }}          {{- end -}}          )        {{- end }}      text: >-        {{ with index .Alerts 0 -}}          :chart_with_upwards_trend: *<{{ .GeneratorURL }}|Graph>*          {{- if .Annotations.runbook }}   :notebook: *<{{ .Annotations.runbook }}|Runbook>*{{ end }}

Configuration File 3 – a .yaml file used to configure Alertmanager to send alerts to Slack (the proxy server field may not be needed for some clusters).

Once that’s been entered, the new configuration can be applied with the following command: 

  oc -n openshift-monitoring create secret generic alertmanager-main -–from-file=alertmanager.yaml –-dry-run -o=yaml | oc -n openshift-monitoring replace secret –filename=-

Testing Alertmanager with Dbench
Dbench can be used with FIO to trigger a test alert in our Slack channel. We’ll do this by creating a test alert rule that will be triggered when a threshold is met for only 2 seconds rather than several minutes. After that, we’ll use dbench to simulate a volume being over 70% used which should trigger an alert.

First, we can create a file named “example-rule.yaml” and enter the following contents:

  apiVersion: monitoring.coreos.com/v1  kind: PrometheusRule  metadata:    labels:      prometheus: k8s      role: alert-rules    name: example-rule.yaml    namespace: openshift-monitoring  spec:    groups:    - name: portworx.rules      rules:      - alert: ExamplePortworxVolumeUsage        annotations:          description: Portworx volume {{$labels.volumeid}} on {{$labels.host}} is over            70% used for more than 2 seconds.          summary: Portworx volume {{$labels.volumeid}} on {{$labels.host}} is at {{$value}}%            used.        expr: 100 * (px_volume_usage_bytes / px_volume_capacity_bytes) > 70        for: 2s        labels:          issue: Portworx volume {{$labels.volumeid}} usage on {{$labels.host}} is critical.          severity: warning

Configuration File 4: A .yaml file used to apply an alert rule for testing Alertmanager

Next, apply the rule using the following command: 
  oc apply -f example-rule.yaml

Now that the test alert rule has been applied, we can use dbench to trigger it. We’ll create a PersistentVolumeClaim with a capacity of 2Gi, and a dbench job with a FIO_SIZE of 1700M. This should cause our created volume to be over 70% used and trigger our test alert. First, we need to create a file named “dbench.yaml” and enter the following contents:
  kind: PersistentVolumeClaim  apiVersion: v1  metadata:    name: dbench-pv-claim  spec:    storageClassName: <storage_class_name>    # storageClassName: gp2    # storageClassName: local-storage    # storageClassName: ibmc-block-bronze    # storageClassName: ibmc-block-silver    # storageClassName: ibmc-block-gold    accessModes:      - ReadWriteOnce    resources:      requests:        storage: 2Gi  ---  apiVersion: batch/v1  kind: Job  metadata:    name: dbench  spec:    template:      spec:        containers:        - name: dbench          image: openebs/fbench:latest          imagePullPolicy: Always          env:            - name: DBENCH_MOUNTPOINT              value: /data          #  - name: DBENCH_QUICK          #    value: "yes"            - name: FIO_SIZE              value: 1700M          #  - name: FIO_OFFSET_INCREMENT          #    value: 256M          #  - name: FIO_DIRECT          #    value: "0"          volumeMounts:          - name: dbench-pv            mountPath: /data        restartPolicy: Never        volumes:        - name: dbench-pv          persistentVolumeClaim:            claimName: dbench-pv-claim    backoffLimit: 4

Configuration File 5: A .yaml file used to create a PVC and dbench job to simulate a volume’s storage capacity being 70% filled

Apply the file with the following command: 

  oc apply -f dbench.yaml

Once the configuration is applied, we should see a new pod and PVC created in our namespace.

image-20200424163255-19

              Figure 10.1 – The dbench-pv-claim PVC and dbench job being created

Now we need to note the name of the volume associated with our new pvc.

image-20200424163255-20

                Figure 10.2  - The newly created pvc and its associated volume name, in this case it is “pvc-5ab2ed7f-7066-11ea-90a2-005056bf3d69”.

We can monitor the volume on the Portworx Volume Dashboard by selecting it in the dropdown menu as seen in figure 10.3 and watching the Volume Usage panel. We should see it fill up to over 70% within a few minutes.

image-20200424163255-21

               Figure 10.3 - The new volume selected in the dropdown menu and its activity in the volume usage panel

Finally, a message should be seen in the Slack channel included in the Alertmanager configuration as in figure 10.4.

image-20200424163255-22

Figure 10.4  - The firing example/test alert being shown in Slack


Example Use Case 2: etcd Monitoring

Prometheus Setup
Once we've chosen a certain external etcd instance in our cluster we want to monitor, we can create a ServiceMonitor the same way we did for the Portworx Storage use case. This time, we can use this configuration to monitor our external etcd instance:

  apiVersion: monitoring.coreos.com/v1  kind: ServiceMonitor  metadata:    labels:      k8s-app: wml-etcd-sm    name: wml-etcd-sm    namespace: openshift-monitoring  spec:    endpoints:      - port: '2379'        targetPort: 2379    namespaceSelector:      matchNames:      - wsl    selector:      matchLabels:        app: wml-etcd
Configuration File 6: A .yaml file used to create a ServiceMonitor for an external etcd cluster
After creating the ServiceMonitor, Prometheus' service account may need to be bound to a new role to allow Prometheus to get, list and watch nodes, endpoints, pods and services in the namespace containing the etcd instance we want to monitor.
Usually, the OpenShift monitoring stack installed by default will have a few etcd instances already being monitored, so we'll see etcd metrics in Prometheus even before our ServiceMonitor is applied. To make sure our new ServiceMonitor worked, we'll need to check if our new instance's metrics are being scraped. We can do this in the Prometheus UI by simply typing in an etcd metric, and seeing whether or not an instance matching our service name is present in the graph/console.

wml-etcd-showing-in-prometheus
 Figure 11.1 - A new instance of etcd being shown in Prometheus after our ServiceMonitor is installed
Grafana Setup
We can also install the Grafana dashboard the same way as the Portworx ones. An etcd dashboard is provided in the official etcd GitHub repository [15]. 
Alertmanager Setup
Since we've already configured Alertmanager with Slack in the Portworx use case, we don't need to do it again. We just need to apply new alert rules for etcd. The same way that we applied alert rules for Portworx, we can apply alert rules for etcd. There is a premade YAML file containing several useful alert rules in the official etcd GitHub repository [16]. 
Testing etcd alerts with Dbench/FIO
We can trigger a realistic etcd alert by using FIO to create high fsync durations during etcd's write-ahead logging. In this cluster, this specific metric usually stays around a certain value, so we will double that value. Also, instead of making the alert fire after the threshold is met for 10 minutes, we will trigger our example alert after just 1 second. We will use a modified version of the pod we used for the Portworx alert testing. 
First, we create a file named "example-etcd-rule.yaml" and enter the following contents: 
  apiVersion: monitoring.coreos.com/v1  kind: PrometheusRule  metadata:    labels:      prometheus: k8s      role: alert-rules    name: example-etcd-rule    namespace: openshift-monitoring  spec:    groups:    - name: etcd      rules:      - alert: Example_etcdHighFsyncDuration        annotations:          message: '[example]etcd cluster "{{ $labels.job }}": 99th percentile fync durations are {{ $value }}s on etcd instance {{ $labels.instance }}.'        expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) > 8        for: 1s        labels:          severity: warning
Configuration File 7: A .yaml file used to create an example alert rule for our etcd alert demo
We can apply the example rule using the same method as we did in the Portworx Storage use case. The next step will be to create a new pod by applying the following YAML file :
  apiVersion: v1  kind: Pod  metadata:    name: wisc-dbench  spec:        containers:        - name: dbench          image: openebs/fbench:latest          command: ["/bin/sh"]          args: ["-c", "while true; do echo hello; sleep 10;done"]          #args: ["-l"]          imagePullPolicy: IfNotPresent          securityContext:            runAsUser: 0          env:            - name: DBENCH_MOUNTPOINT              value: /data          #  - name: DBENCH_QUICK          #    value: "yes"            - name: FIO_SIZE              value: 1M          #  - name: FIO_OFFSET_INCREMENT          #    value: 256M          #  - name: FIO_DIRECT          #    value: "0"            volumeMounts:          - name: dbench-pv            mountPath: /data        restartPolicy: Always        volumes:        - name: dbench-pv          persistentVolumeClaim:            claimName: etcd-pvc
Configuration File 8: A .yaml file used to create a pod used to run FIO and create an artifical load on our etcd cluster.
Once we've created the pod, we can create a shell script inside the pod named "install-fio.sh" with the following contents and run it: 
  #! /bin/sh    apk --no-cache add \          make \          alpine-sdk \          zlib-dev \          libaio-dev \          linux-headers \          coreutils \          libaio && \      git clone https://github.com/axboe/fio && \      cd fio && \      ./configure && \      make -j`nproc` && \      make install && \      cd .. && \      rm -rf fio && \      apk --no-cache del \          make \          alpine-sdk \          zlib-dev \          libaio-dev \          linux-headers \          coreutils
Configuration File 9: A shell script used to install FIO in a pod with the Alpine Linux Package Manager
install-fio-run
 Figure 11.2 - The script used to install FIO being run 
Once that's done, we need to add the screen tool into the container using the following command: 
  apk add screen
Now we can create a new screen session, and create four windows in that session.
4-screen-windows
 Figure 11.3 - Four screen windows are created
In all four screen windows, we'll need to change directories to the /data directory because that is where our etcd volume is mounted. If we run FIO elsewhere, our etcd service will be unaffected and will not trigger an alert. We'll also need to create a directory named "test-data" to store data from FIO. Next, we will use the following FIO command in each window:
  fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=220000m --bs=23000000 --name=mytest
In that command, the "--bs" flag is the block size. In order to increase fsync write durations, the block size is made very big. This causes each write to take much longer than normal. Now we can type our example alert rule's expression in the Prometheus UI to see its progress. After a few minutes, we should see a dramatic increase. 
 
wml-etcd-threshold-met
 Figure 11.4 - The metric being shown in Prometheus has surpassed a value of 8 which should trigger our example alert
We can see that the metric is much higher than normal, and we should also be able to see our example alert firing in slack. At this point, its important to remember to go back into the Dbench pod and kill all of the FIO processes. If they are left to keep running, the etcd cluster may be damaged.
wml-etcd-alert-firing-in-slack
 Figure 11.5 - Our example etcd alert firing in Slack
Conclusion
After using this monitoring system in two different CP4D installations for different use cases, we determined that it is portable and can be replicated without much trouble. Throughout this process of installing monitoring components and testing them, we learned that there are certain applications that may not directly support Prometheus, such as Postgresql and Elasticsearch. For these types of applications, an exporter is needed to provide Prometheus with an endpoint to scrape metrics from. For quite a few components of a CP4D base platform or CP4D application (e.g. Watson Discovery), open source exporters do exist and can be tailored to use in the CP4D / Openshift monitoring framework described here. Built upon Prometheus with Grafana and Alertmanager monitoring stack provided by OpenShift, both CP4D platform and applications could be configured to expose useful operational metrics to support root cause analysis of an issue or preventive measures to avoid system disasters which are essential to Day 2 operations.
References

[6] Bruce, Morgan, and Paulo A.. Pereira. Microservices in Action. Manning Publications Co., 2019.
[14] https://docs.portworx.com/samples/k8s/grafana/prometheus-rules.yaml

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SGMV159","label":"IBM Support for Red Hat OpenStack Platform"},"ARM Category":[],"Platform":[{"code":"PF040","label":"RedHat OpenShift"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB61","label":"IBM Technology Lifecycle Services"}}]

Document Information

Modified date:
12 August 2021

UID

ibm16174375

Page Feedback