Configuring Prometheus for health monitoring

Edit online

You can configure an external Prometheus instance with IBM® Guardium® Cryptography Manager to scrape metrics from IBM services.

Before you begin

Install Prometheus.
Ensure that you have access to the IBM services endpoint.
Ensure that you have valid OIDC credentials (client ID, client secret, username, password).
Ensure that you have network connectivity to the target services.

About this task

Complete the following steps to configure an external Prometheus with Guardium Cryptography Manager.

Note: Application types for Grafana setup

The following application types are available for setting up Grafana dashboards.

Spring Boot Applications: All services except Asset Inventory and Asset Discovery use Spring Boot Actuator metrics format
- User Management, Integration Manager, Audit Management, Notification Management
- Scheduler, Policy, Swagger, Policy Risk Evaluation
- Transparent Database Encryption (TDE) key management, Key Management Interoperability Protocol (KMIP) , System certificates, CLM, CPM
Go Applications: Asset Inventory and Asset Discovery are Go Applications with different metrics format
- Asset Inventory (/ibm/assetinventory/api/v1/metrics)
- Asset Discovery (/ibm/assetdiscovery/api/v1/metrics)

This distinction is critical when you design Grafana dashboards because Go Applications often use different metric naming conventions and provide a different set of metrics compared to Spring Boot Actuator endpoints.

Procedure

Obtain OIDC Access Token

Run the following curl command to obtain an OIDC access token.

curl --request POST \
  --url https://<KEYCLOAK_HOST>:<KEYCLOAK_PORT>/realms/<REALM_NAME>/protocol/openid-connect/token \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data client_id=<CLIENT_ID> \
  --data client_secret=<CLIENT_SECRET> \
  --data username=<USERNAME> \
  --data password=<PASSWORD> \
  --data grant_type=password

The response for step 1a contains an access_token field. Extract this token and save it to a file.
```
# Extract token from response and save to file
echo "<ACCESS_TOKEN>" > /etc/prometheus/tokens/my-metrics.jwt
```
Note: Ensure that the token directory exists and has appropriate permissions. The access token typically expires after a certain period. You need to implement a token refresh mechanism for use.
Get the appropriate CA certificate and save it to /etc/prometheus/certs/gcm-ca.pem.
- You can get CA certificate by the following command.
```
echo | openssl s_client -showcerts -connect <TARGET_HOST>:<TARGET_PORT> 2>/dev/null   | awk '/BEGIN CERTIFICATE/{flag=1} flag{print} /END CERTIFICATE/{exit}'   > /etc/prometheus/certs/gcm-ca.pem
```
- Ensure that the directories exist else create them by using mkdir /etc/prometheus/certs.

Configure Prometheus

Add the following scrape configurations to your prometheus.yml file.

scrape_configs:
  - job_name: 'user management'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/usermanagement/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'integration Manager'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/integrationmanager/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'audit management'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/auditmgmt/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'notification management'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/notificationmgmt/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'scheduler'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/scheduler/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'policy'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/gemimcpolicy/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'swagger'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/swagger/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'policy risk evaluation'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/gempolicyengine/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'tde'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/encryption/db/tde/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'kmip'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/encryption/db/tde/api/v1/kmip-actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'asset inventory' #go application
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/assetinventory/api/v1/metrics
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'asset discovery' #go application
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/assetdiscovery/api/v1/metrics
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'system certs'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/systemcerts/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'clm'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/clm/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

  - job_name: 'cpm'
    scheme: https
    bearer_token_file: /etc/prometheus/tokens/my-metrics.jwt
    metrics_path: /ibm/cpm/api/v1/actuator/prometheus
    tls_config:
      ca_file: /etc/prometheus/certs/gcm-ca.pem
    static_configs:
      - targets:
          - '<TARGET_HOST>:<TARGET_PORT>'

Where,

<TARGET_HOST>: Target service hostname or IP address
<TARGET_PORT>: Target service port (generally 31443)

Restart Prometheus after you update the configuration.

Verify Metrics collection

Go to the Prometheus web UI and check the targets page and check whether all configured jobs display as UP so that the configuration is correct.

You can test individual metrics endpoints by using the following curl commands.

#User Management
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/usermanagement/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#Integration Manager
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/integrationmanager/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#Audit Management
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/auditmgmt/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#Notification Management
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/notificationmgmt/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#Scheduler
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/scheduler/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#Policy
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/gemimcpolicy/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#Swagger
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/swagger/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#Policy Risk Evaluation
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/gempolicyengine/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#TDE
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/encryption/db/tde/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#KMIP
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/encryption/db/tde/api/v1/kmip-actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#Asset Inventory
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/assetinventory/api/v1/metrics \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#Asset Discovery
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/assetdiscovery/api/v1/metrics \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#System Certificates
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/systemcerts/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#CLM
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/clm/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

#CPM
curl --request GET \
  --url https://<TARGET_HOST>:<TARGET_PORT>/ibm/cpm/api/v1/actuator/prometheus \
  --header 'Authorization: Bearer <ACCESS_TOKEN>' \
  --header 'Content-Type: application/json'

Where,

<TARGET_HOST>: Target service hostname or IP address
<TARGET_PORT>: Target service port
<ACCESS_TOKEN>: OIDC access token obtained in Step 1.