Configuring the Metric API as a Grafana data source

The Monitoring component provides an API that enables metrics to be accessed using a subset of the PromQL query language. This API provides access to all metrics that are stored using the Red Hat Advanced Cluster Management observability service. The API can be used as a Grafana data source. This enables you to create ad-hoc metric queries and custom reporting dashboards in Grafana based on the Monitoring metrics. More reference links are available here: Reference information.

This API does not perform filtering by tenant. It is fully compatible by design with the prometheus API, which enables it to be used as a Grafana data source. In response to metric queries, the API returns all metrics (regardless of tenant) that are stored in the Red Hat Advanced Cluster Management observability service. Any filtering that is needed must be performed at the promql query level by using metric labels. Metrics that are stored by IBM Cloud Pak® for Multicloud Management are labeled with the IAM account id (label name account_id). In addition, the Red Hat Advanced Cluster Management for Kubernetes Observability service adds a tenant id label (label name tenant_id) to each metric.

Before you begin

  1. To authenticate with the metrics promql endpoint, an IAM API Key is needed. There are different ways to get this API Key, for more information, see Create service ID and API key.

  2. To find the hostname of the metric's promql endpoint, run the following Openshift command:

    oc get route metric-promql -n management-monitoring

    The host name/port is under the HOST/PORT column. Here's an example of what the command output looks like:

NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
metric-promql metric-promql-management-monitoring.apps.murdoch.os.fyre.ibm.com /metric/1.0/promql monitoring-metric rest-api edge/Redirect None

Procedure

  1. In Grafana, configure a new Prometheus datasource with the following URL:
    a. Update the URL field with hostname obtained in step 2 in Before you begin

    https://<hostname>/metric/1.0/promql

    where hostname is the hostname obtained in step 2 in Before you begin section

  2. Create a new Custom HTTP header name X-API-KEY. Assign the API Key created in step 1 in Before you begin section.

  3. If the OpenShift cluster does not have a CA certified certificate, turn on the Skip TLS Verify switch.

    Note: Once you enter the X-API-KEY header Grafana caches it. If you click the delete button to delete it, it appears as deleted but if you refresh the page it comes back with the previous configured value. To remove the configured key, click the Reset and enter some other invalid value. If you delete the API Key, it can take up to 10 minutes to take effect.

Reference information

For more information about how to configure the Red Hat Advanced Cluster Management observability service, see Enabling the observability service in Red Hat Advanced Cluster Management

Some PromQL reference information is available here:
Querying Prometheus Opens in a new tab in the Prometheus documentation.

Introduction to PromQL, the Prometheus Query Language Opens in a new tab in the Grafana documentation.