IBM Cloud Pak foundational services Monitoring service

Accessing the monitoring dashboard

1. Log in to the console.

   Note: When you log in to the console, you have administrative access to Grafana. Do not create more users within the Grafana dashboard or modify the existing users or org.

2. To access the Grafana dashboard, click Menu > Platform > Monitoring. Alternatively, you can open https://<IP_address>:<port>/grafana, where <IP_address> is the DNS or IP address that is used to access the console. <port> is the port that is used to access the console.

3. To access the Alertmanager dashboard, click Menu > Platform > Alerting. Alternatively, you can open https://<IP_address>:<port>/alertmanager.

4. To access the Prometheus dashboard, open https://<IP_address>:<port>/prometheus.

5. From the Grafana dashboard, open one of the following default dashboards:

   • Etcd by Prometheus

     Etcd Dashboard for Prometheus metrics scraper.

   • Helm Release Metrics

     Provides information about system metrics such as CPU and Memory for each Helm release that is filtered by pods.

   • Namespaces Performance IBM Provided 2.5

     Provides information about namespace performance and status metrics.

   • Performance IBM Provided 2.5 Provides TCP system performance information about Nodes, Memory, and Containers.

   • Kubernetes Cluster Monitoring Monitors Kubernetes clusters that use Prometheus. Provides information about cluster CPU, Memory, and Filesystem usage. The dashboard also provides statistics for individual pods, containers, and systemd services.

   • Kubernetes POD Overview Monitors pod metrics such as CPU, Memory, Network pod status, and restarts.

   • NGINX Ingress controller Provides information about NGINX Ingress controller metrics that can be sorted by namespace, controller class, controller, and ingress.

   • Node Performance Summary Provides information about system performance metrics such as CPU, Memory, Disk, and Network for all nodes in the cluster.

   • Prometheus Stats Dashboard for monitoring Prometheus v2.x.x.

   • MongoDB Overview Provides server status metrics such as Connections, Commands, and Operations.

   • MongoDB ReplSet Provides replica-set metrics such as Members, Member status, Member elections, Replication lag, and Oplog activity.

   • MongoDB WiredTiger Provides storage-engine metrics such as Cache activity, Blockmanager, Sessions, and Page-faults.

     Note: If you configure pods to use host level resources such as host network, the dashboards display the metrics of the host but not the pod itself.

   • IBM Multicloud Manager Monitoring Provides information for metrics such as CPU, Memory, and Network for managed clusters. This dashboard is available only for IBM Multicloud Manager hub clusters.

If you want to view other data, you can create new dashboards or import dashboards from JSON definition files for Grafana.

Metrics collected out of the box

Some exporters are provided to manage metrics. The exporters expose metrics endpoints as Kubernetes services.

• node-exporter Provides the node-level metrics, including metrics for CPU, memory, disk, network, and other components.

• kube-state-metrics Provides the metrics for Kubernetes objects, including metrics for pod, deployment, statefulset, daemonset, replicaset, configmap, service, job, and other objects.

• elasticsearch-exporter Provides metrics for the Elasticsearch logging service, including the status for Elasticsearch cluster, shards, and other components.

• collectd-exporter Provides metrics that are sent from the collected network plug-in.

• MongoDB exporter Provides metrics for the MongoDB service, including server, replica-set, and storage status.

Some Kubernetes pods provide metrics endpoints for Prometheus:

• nginx-ingress-controller Provides metrics for the Nginx ingress controller.

In addition, Prometheus has preconfigured scrape targets that communicate with several targets to scrape metrics:

• cAdvisor Provides container metrics that include CPU, memory, network, and other components.

• Prometheus Provides metrics for the Prometheus server that include metrics for request handle, alert rule evaluation, TSDB status, and other components.

• kubernetes-apiservers Provide metrics for the Kubernetes API servers.

Prometheus displays scrape targets in its user interface as links. These addresses are typically not accessible from a user's browser as they are on the Kubernetes cluster internal network. Only the Prometheus server needs to be able to access the addresses.

Role-based access control (RBAC)

curl -k -s -X POST -H "Authorization:$ACCESS_TOKEN" https://<Cluster Master Host>:<Cluster Master API Port>/grafana/check_stale_users

Installing monitoring service in IBM Cloud Pak for Integration

Monitoring service is installed by default during IBM Cloud Pak for Integration installation. You can also select to install monitoring service from the Catalog or cloudctl.

  apiVersion: v1
  kind: PersistentVolume
  metadata:
      name: monitoring-prometheus-pv
      labels:
          component: prometheus
  .......

Configuring the Prometheus server

You can configure the following Prometheus server parameters during preinstallation or post installation:

• scrape_Interval

  The parameter for the frequency to scrape targets. The default value is 1 minute (1m).

• evaluation_Interval

  The parameter for the frequency to evaluate rules. The default value is 1 minute (1m).

• retention

  The parameter for the frequency to remove old data. The default value is 24 hours (24h).

• resources.limits.memory

  The parameter for the memory limitation for the Prometheus container. The default value is 4096Mi. The Prometheus container crashes if the memory limitation is not fulfilled. You must increase the value of this parameter to ensure that the Prometheus container can work correctly.

monitoring:
  prometheus:
    scrape_Interval: 1m
    evaluation_Interval: 1m
    retention: 24h
    resources:
      limits:
        memory: 4096Mi

kubectl edit prometheus monitoring-prometheus -n kube-system

Alerts

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
    labels:
        component: icp-prometheus
    name: sample-rule
spec:
    groups:
      - name: a.rules
        rules:
          - alert: NodeMemoryUsage
            expr: ((node_memory_MemTotal_bytes - (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes))/ node_memory_MemTotal_bytes) * 100 > 5
            annotations:
              DESCRIPTION: '{{ $labels.instance }}: Memory usage is greater than the 15% threshold.  The current value is: {{ $value }}.'
              SUMMARY: '{{ $labels.instance }}: High memory usage detected'

apiVersion: monitoringcontroller.cloud.ibm.com/v1
kind: AlertRule
metadata:
  name: failed-jobs
spec:
  enabled: true
  data: |-
    groups:
      - name: failedJobs
        rules:
          - alert: failedJobs
            expr: kube_job_failed != 0
            annotations:
              description: 'Job {{ "{{ " }} $labels.exported_job {{ " }}" }} in namespace {{ "{{ " }} $labels.namespace {{ " }}" }} failed for some reason.'
              summary: Failed job

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    component: icp-prometheus
  name: failed-jobs
spec:
  groups:
    - name: failedJobs
      rules:
        - alert: failedJobs
          expr: kube_job_failed != 0
          annotations:
            description: 'Job {{ "{{ " }} $labels.exported_job {{ " }}" }} in namespace {{ "{{ " }} $labels.namespace {{ " }}" }} failed for some reason.'
            summary: Failed job

kubectl create -f {file}

kubectl edit secret alertmanager-monitoring-prometheus-alertmanager -n kube-system

apiVersion: v1
data:
  alertmanager.yaml: CiAgZ2xvYmFsOgogIHJlY2VpdmVyczoKICAgIC0gbmFtZTogZGVmYXVsdC1yZWNlaXZlcgogIHJvdXRlOgogICAgZ3JvdXBfd2FpdDogMTBzCiAgICBncm91cF9pbnRlcnZhbDogNW0KICAgIHJlY2VpdmVyOiBkZWZhdWx0LXJlY2VpdmVyCiAgICByZXBlYXRfaW50ZXJ2YWw6IDNo
kind: Secret
metadata:
  name: alertmanager-monitoring-prometheus-alertmanager
type: Opaque

Managing Grafana dashboards

You can manage Grafana dashboards by operating on a Kubernetes custom resource MonitoringDashboard in IBM Cloud Pak for Integration. The following sample-dashboard.yaml file is an example of a MonitoringDashboard resource definition.

apiVersion: monitoringcontroller.cloud.ibm.com/v1
kind: MonitoringDashboard
metadata:
  name: sample-dashboard
spec:
  enabled: true
  data: |-
    {
        "id": null,
        "uid": null,
        "title": "Marco Test Dashboard",
        "tags": [ "test" ],
        "timezone": "browser",
        "schemaVersion": 16,
        "version": 1
      }

You must provide the following parameter values:

• apiVersion

  monitoringcontroller.cloud.ibm.com/v1

• kind

  MonitoringDashboard

• spec.data

  Contains the content of the Grafana dashboard definition file. For more information about dashboard files, see Dashboard JSON .

• spec-enabled

  Set the flag to specify whether the dashboard is enabled or not enabled.

  You can use kubectl to manage the dashboard. Use the -n option to specify the namespace in which this MonitoringDashboard is to be created. The dashboard will be imported to the corresponding organization in Grafana.

• Create a new dashboard resource in the default namespace using the sample-dashboard.yaml file. The dashboard will be imported into the default organization in Grafana.

  kubectl apply -f sample-dashboard.yaml -n default
  

• Edit the sample dashboard.

  kubectl edit monitoringdashboards/sample-dashboard -n default
  

• Delete the sample dashboard.

  kubectl delete monitoringdashboards/sample-dashboard -n default
  

Configure applications to use monitoring service

Modify the application to expose the metrics.

• For applications that have a metrics endpoint, you must define the metrics endpoint as a Kubernetes service by using the annotation prometheus.io/scrape: 'true'. The service definition resembles the following code:

  apiVersion: v1
  kind: Service
  metadata:
    annotations:
    prometheus.io/scrape: 'true'
    labels:
      app: liberty
    name: liberty
  spec:
    ports:
    - name: metrics
      targetPort: 5556
      port: 5556
      protocol: TCP
    selector:
      app: liberty
    type: ClusterIP
  

  Note: For more information about configuring the metrics endpoint for Prometheus, see CLIENT LIBRARIES in the Prometheus documentation.

• Applications can have more than one port defined in the service definition. You might not want to expose monitoring metrics on some ports or have the ports be discovered by Prometheus. You can add annotation filter.by.port.name: 'true' so the port whose name does not start with metrics is ignored by Prometheus. In the following service definition, Prometheus collects metrics from port metrics, and ignores metrics from port collector.

  apiVersion: v1
  kind: Service
  metadata:
  annotations:
    prometheus.io/scrape: 'true'
    filter.by.port.name: 'true'
  labels:
    app: liberty
  name: liberty
  spec:
  ports:
  - name: metrics
    targetPort: 5556
    port: 5556
    protocol: TCP
  - name: collector
    targetPort: 8443
    port: 8443
    protocol: TCP
  selector:
    app: liberty
  type: ClusterIP
  

• For applications that have a metrics endpoint with TLS enabled, you must use IBM Cloud Pak for Integration cert-manager to generate a secret and use it to configure the metrics endpoint.

  1. Use cert-manager to create a certificate resource for a workload.

     apiVersion: certmanager.k8s.io/v1alpha1
     kind: Certificate
     metadata:
       name: {{ .Release.Name }}-foo-certs
       namespace: {{ .Release.Namespace }}
     spec:
       secretName: {{ .Release.Name }}-foo-certs
       issuerRef:
         name: icp-ca-issuer
         kind: ClusterIssuer
       commonName: "foo"
       dnsNames:
           - "*.{{ .Release.Namespace }}.pod.cluster.local"
     

  2. Mount the secret to your pod. You can retrieve the cert/key from the mounted path. Under the mounted path, there are two files named tls.crt and tls.key. tls.crt includes a workload cert file and a ca cert file that must use to configure the application metrics endpoint.

     containers:
       - image: foo-image:latest
         name: foo
         volumeMounts:
           - mountPath: "/foo/certs"
             name: certs
     volumes:
       - name: certs
         secret:
           # secretName should be the same as the one defined in step 1.
         secretName: {{ .Release.Name }}-foo-certs
     

  3. Define annotations on workload service to allow Prometheus to use TLS to scrape metrics, prometheus.io/scrape and prometheus.io/scheme.

     apiVersion: v1
     kind: Service
     metadata:
       annotations:
       prometheus.io/scrape: 'true'
       prometheus.io/scheme: 'https'
     

• For applications that use collectd and depend on collectd-exporter to expose metrics, you update collectd configuration file within the application container. In this configuration file, you must add the network plug-in and point to collectd exporter. Add the following text to the configuration file:

  LoadPlugin network
  <Plugin network>
      Server "monitoring-prometheus-collectdexporter.kube-system" "25826"
  </Plugin>
  

Logs and metrics management for Prometheus

You can modify the time period for metric retention by updating the storage.tsdb.retention parameter in the config.yaml file. By default this value is set at 24h, which means that the metrics are kept for 24 hours and then purged. See Configuring the monitoring service.

However, if you need to manually remove this data from the system, you can use the rest API that is provided by the Prometheus component.

• To delete metrics data, see Delete Series .
• To remove the deleted data from the disk, and clean up the disk space, see Clean Tombstones .

The target URL must have the format:

https://<IP_address>:<Port>/prometheus

• <IP_address> is the IP address that is used to access the console.

• <Port> is the port that is used to access the console.

  • The command to delete metrics data resembles the following code:

    https://<IP_address>:<Port>/prometheus/api/v1/admin/tsdb/delete_series?*******
    

  • The command to remove deleted data and clean up the disk, resembles the following code:

    https://<IP_address>:<Port>/prometheus/api/v1/admin/tsdb/clean_tombstones
    

Accessing monitoring service APIs

You can access monitoring service APIs such as Prometheus and Grafana APIs. Before you can access the APIs, you must obtain authentication tokens to specify in your request headers. For information about obtaining authentication tokens, see Preparing to run component or management API commands.

After you obtain the authentication tokens, complete the following steps to access the Prometheus and Grafana APIs.

1. Access the Prometheus API at url, https://<Cluster Master Host>:<Cluster Master API Port>/prometheus/* and get boot times of all nodes.

   • $ACCESS_TOKEN is the variable that stores the authentication token for your cluster.

   • <Cluster Master Host> and <Cluster Master API Port> are defined in Master endpoints.

     curl -k -s -X GET -H "Authorization:Bearer $ACCESS_TOKEN" https://<Cluster Master Host>:<Cluster Master API Port>/prometheus/api/v1/query?query=node_boot_time_seconds
     

     For detailed information about Prometheus APIs, see Prometheus HTTP API .

2. Access the Grafana API at url, https://<Cluster Master Host>:<Cluster Master API Port>/grafana/* and obtain the sample dashboard.

   • $ACCESS_TOKEN is the variable that stores the authentication token for your cluster.

   • <Cluster Master Host> and <Cluster Master API Port> are defined in Master endpoints.

     curl -k -s -X GET -H "Authorization: Bearer $ACCESS_TOKEN” "https://<Cluster Master Host>:<Cluster Master API Port>/grafana/api/dashboards/db/sample"
     

     For detailed information about Grafana APIs, see Grafana HTTP API Reference .

Support for custom cluster access URL in monitoring service

You can customize the cluster access URL. For more information, see Customizing the cluster access URL. After you complete the customization, you must manually edit the Prometheus and Alertmanager resources and verify that all external links are correct.

kubectl edit prometheus monitoring-prometheus -n kube-system

externalUrl: https://<custom_host>:<custom_port>/prometheus

kubectl edit alertmanager monitoring-prometheus-alertmanager -n kube-system

externalUrl: https//:<custom_host>:<custom_port>/alertmanager