Service monitoring with Prometheus and Grafana

IBM BAMOE Operator interaction with Prometheus and Grafana

IBM BAMOE provides a monitoring-prometheus-addon add-on that enables Prometheus metrics monitoring for IBM BAMOE services and generates Grafana dashboards that consume the default metrics exported by the add-on. The IBM BAMOE Operator uses the Prometheus Operator to expose the metrics from your IBM BAMOE project for Prometheus to scrape. Due to this dependency, the Prometheus Operator must be installed in the same namespace as your IBM BAMOE project.

When you deploy an IBM BAMOE service that uses the monitoring-prometheus-addon add-on and the Prometheus Operator is installed, the IBM BAMOE Operator creates a ServiceMonitor custom resource to expose the metrics for Prometheus, as shown in the following example:

Example ServiceMonitor resource for Prometheus

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: onboarding-service
  name: onboarding-service
  namespace: kogito
spec:
  endpoints:
  - path: /metrics
    targetPort: 8080
    scheme: http
  namespaceSelector:
    matchNames:
    - kogito
  selector:
    matchLabels:
      app: onboarding-service

You must manually configure your Prometheus custom resource that is managed by the Prometheus Operator to select the ServiceMonitor resource:

Example Prometheus resource

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      app: onboarding-service

After you configure your Prometheus resource with the ServiceMonitor resource, you can see the endpoint being scraped by Prometheus in the Targets page in the Prometheus web console:

Image of Kogito service targets view in Prometheus

Figure 1. Targets page in Prometheus web console

The metrics exposed by the IBM BAMOE service appear in the Graph view:

Image of Kogito service graph view in Prometheus

Figure 2. Graph view in Prometheus web console

The IBM BAMOE Operator also creates a GrafanaDashboard custom resource defined by the Grafana Operator for each of the Grafana dashboards generated by the add-on. The app label for the dashboards is the name of the deployed IBM BAMOE service. You must set the dashboardLabelSelector property of the Grafana custom resource according to the relevant IBM BAMOE service.

Example Grafana resource

apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: example-grafana
spec:
  ingress:
    enabled: true
  config:
    auth:
      disable_signout_menu: true
    auth.anonymous:
      enabled: true
    log:
      level: warn
      mode: console
    security:
      admin_password: secret
      admin_user: root
  dashboardLabelSelector:
    - matchExpressions:
        - key: app
          operator: In
          values:
            - my-kogito-application

Additional resources

Replacing IBM BAMOE services TrustStores

You can replace the default Java TrustStore that comes in IBM BAMOE images. Replacement of Java TrustStore is required when a given IBM BAMOE service makes HTTPS calls to services that require encrypted connections using a private Certificate Authority (CA).

Prerequisites

The Java TrustStore replaces the default one. You must add the trusted certificates to the default JKS. As an alternative, you can create a new TrustStore that consists of the required IBM BAMOE service certificates.

Note	Any IBM BAMOE service that is deployed in a given namespace can access the Java TrustStore if required.

To manipulate the Java TrustStore, you can use KeyExplorer or keytool that consists of JDK.

Procedure

Create a Kubernetes Secret based on the customized Java TrustStore:
Example Kuberbetes Secret
$ oc create secret generic kogito-truststore --from-file=cacerts=<path to your JKS> --from-literal=keyStorePassword=<TrustStore Password>
You must define the secret keys as shown in the previous command. cacerts is used for the TrustStore file and keyStorePassword is used for the TrustStore password.

Deploy the IBM BAMOE service that uses the TrustStore. For example:

Example IBM BAMOE service deployment

piVersion: app.kiegroup.org/v1beta1
kind: KogitoRuntime
metadata:
  name: my-service
spec:
  replicas: 1
  image: quay.io/my-namespace/my-service:latest
  trustStoreSecret: kogito-truststore

Enabling Prometheus metrics monitoring in IBM BAMOE

Prometheus is an open source systems monitoring toolkit that you can use with IBM BAMOE to collect and store metrics related to the execution of Business Process Model and Notation (BPMN) process models, business rules, and Decision Model and Notation (DMN) decision models. You can access the stored metrics through a REST API call to a configured application endpoint, through the Prometheus expression browser, or using a data-graphing tool such as Grafana.

Prerequisites

Prometheus is installed. For information about downloading and using Prometheus, see the Prometheus documentation page.

Procedure

In your IBM BAMOE project, depending on the framework you are using, add one of the following dependencies to the pom.xml file to enable the Prometheus add-on:
Add dependency for Prometheus Quarkus add-on
<dependency> <groupId>org.kie</groupId> <artifactId>kie-addons-quarkus-monitoring-prometheus</artifactId> <version>KOGITO_VERSION</version> </dependency>
By default, all stored metrics are exported to the configured monitoring module. If you want to disable certain metrics from being exported, you can set any of the following properties in the application.properties file of your IBM BAMOE project:
Enable or disable specific metrics from being exported
kogito.monitoring.rule.useDefault=false kogito.monitoring.process.useDefault=false kogito.monitoring.interceptor.useDefault=false
In the prometheus.yaml file of your Prometheus distribution, add the following settings in the scrape_configs section to configure Prometheus to scrape metrics from your IBM BAMOE service:
Example scrape configurations in prometheus.yaml file
scrape_configs: job_name: 'kogito-metrics' metrics_path: /metrics static_configs: - targets: ["localhost:8080"]
Replace the values according to your IBM BAMOE service settings.
In a command terminal, navigate to your IBM BAMOE project and run the project using your preferred run mode, such as development mode:
On Quarkus
mvn clean compile quarkus:dev
After you start your IBM BAMOE service, Prometheus begins collecting metrics and IBM BAMOE publishes the metrics to the configured REST API endpoint.

To verify the metrics configuration, use a REST client or curl utility to send a GET request to the configured /metrics endpoint, such as http://localhost:8080/metrics in this example:

Example curl command to return Prometheus metrics

curl -X GET http://localhost:8080/metrics

Example response

# HELP kogito_process_instance_completed_total Completed Process Instances
# TYPE kogito_process_instance_completed_total counter
# HELP kogito_process_instance_started_total Started Process Instances
# TYPE kogito_process_instance_started_total counter
kogito_process_instance_started_total{app_id="acme-travels",process_id="travels",} 1.0
# HELP kogito_work_item_duration_seconds Work Items Duration
# TYPE kogito_work_item_duration_seconds summary
# HELP drl_match_fired_nanosecond Drools Firing Time
# TYPE drl_match_fired_nanosecond histogram
drl_match_fired_nanosecond_bucket{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",le="1000000.0",} 1.0
drl_match_fired_nanosecond_bucket{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",le="2000000.0",} 1.0
drl_match_fired_nanosecond_bucket{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",le="3000000.0",} 1.0
drl_match_fired_nanosecond_bucket{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",le="4000000.0",} 1.0
drl_match_fired_nanosecond_bucket{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",le="5000000.0",} 1.0
drl_match_fired_nanosecond_bucket{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",le="6000000.0",} 1.0
drl_match_fired_nanosecond_bucket{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",le="7000000.0",} 1.0
drl_match_fired_nanosecond_bucket{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",le="8000000.0",} 1.0
drl_match_fired_nanosecond_bucket{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",le="9000000.0",} 1.0
drl_match_fired_nanosecond_bucket{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",le="+Inf",} 1.0
drl_match_fired_nanosecond_count{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",} 1.0
drl_match_fired_nanosecond_sum{identifier="acme-travels",rule_name="Brazilian citizens require visa to Australia",} 789941.0
# HELP kogito_process_instance_sla_violated_total Process Instances SLA Violated
# TYPE kogito_process_instance_sla_violated_total counter
# HELP kogito_process_instance_duration_seconds Process Instances Duration
# TYPE kogito_process_instance_duration_seconds summary
# HELP kogito_process_instance_running_total Running Process Instances
# TYPE kogito_process_instance_running_total gauge
kogito_process_instance_running_total{app_id="acme-travels",process_id="travels",} 1.0

If the metrics are not available at the defined endpoint, review and verify the IBM BAMOE and Prometheus configurations described in this section.

You can also interact with your collected metrics and application targets in the Prometheus expression browser at http://HOST:PORT/graph and http://HOST:PORT/targets, or integrate your Prometheus data source with a data-graphing tool such as Grafana:

Image of targets in Prometheus expression browser

Figure 3. Prometheus expression browser with IBM BAMOE service targets

Figure 4. Grafana dashboard with IBM BAMOE service metrics

Additional resources

Grafana dashboards for default metrics in IBM BAMOE

If any of the Prometheus monitoring modules are imported as dependencies in the pom.xml file of your IBM BAMOE project, some Grafana dashboards that use the default metrics are generated under the folder target/classes/META-INF/resources/monitoring/dashboards/ every time you compile your IBM BAMOE service.

Two types of dashboards are exported depending on the decision model used on the endpoints:

Operational dashboard: This dashboard is generated for DMN and DRL endpoints and contains the following metrics:
- Total number of requests on the endpoint
- Average number of requests per minute on the endpoint
- Quantiles on the elapsed time to evaluate the requests
- Exception details
  
  Figure 5. Generated operational dashboard example
Domain-specific dashboard: Currently this dashboard is exported only for DMN endpoints. The domain-specific dashboard contains a graph for each type of decision in the DMN model. Only the built-in types number, string, and boolean are currently supported.
- If the output of the decision is a number type, the graph contains the quantiles for that metric on a sliding window of 3 minutes.
- If the output is a boolean or a string type, the graph contains the number of occurrences for each output on a 10-minute average.
  
  Figure 6. Generated domain-specific dashboard example

Note	Generated dashboards for BPMN resources are currently not supported.

Custom Grafana dashboards in IBM BAMOE

You can add custom dashboards that are defined as json files in your project. For format specification and more information, see Official documentation page.

To add custom dashboards in your IBM BAMOE project, you must follow the following conventions:

Dashboard files must be stored in /src/main/resources/META-INF/dashboards directory
dashboard file names must start with domain-dashoboard (for domain specific dashboards) or operational-dashboard (for operational dashboards)
Dashboard file names must end with .json
Dashboard file names must not conflict with the auto-generated ones
The title attribute of custom dashboards must not conflict with the auto-generated attributes

Custom dashboards are available in the Grafana panel along with the auto-generated dashboards.

Disabling Grafana dashboards generation in IBM BAMOE

You can disable the generation of default dashboards in Prometheus metrics monitoring using the following properties:

Properties to disable default dashboards

kogito.grafana.disabled.operational.dashboards
kogito.grafana.disabled.domain.dashboards

Prerequisites

Prometheus is installed. For information about downloading and using Prometheus, see the Prometheus documentation page.

Procedure

To disable Grafana dashboard, add a comma-separated list of dashboard identifiers to the following properties:

Properties to disable Grafana dashboard

kogito.grafana.disabled.operational.dashboards
kogito.grafana.disabled.domain.dashboards

The kogito.grafana.disabled.operational.dashboards property disables the generation of operational dashboards and kogito.grafana.disabled.domain.dashboards property disables the generation of domain dashboards.

For example, a project containing the following resources uses the default configuration and generates the default dashboards:

Example project resources

Hello.drl
LoanEligibility.dmn
Traffic Violation.dmn

The following are the default generated dashboards in IBM BAMOE:

Table 1. Example default generated dashboards for a project
Dashboard type	Dashboard name
Operational	`Hello`
	`LoanEligibility`
	`Traffic Violation`
Domain	`Hello`
	`LoanEligibility`
	`Traffic Violation`

If you use the following configuration in the application.properties file, the generation of Hello and Traffic Violation operational dashboards, and LoanEligibility domain dashboard is avoided.

Configuration for disabling specific dashboards

kogito.grafana.disabled.operational.dashboards=Hello,Traffic Violation
kogito.grafana.disabled.domain.dashboards=LoanEligibility

Table 2. Customized generation of dashboards in IBM BAMOE
Dashboard type	Dashboard name
Operational	`LoanEligibility`
Domain	`Hello`
Domain	`Traffic Violation`

Important

The spaces between the dashboard identifiers are eliminated as follows:

Example dashboard identifiers with spaces

Hello, Traffic Violation, LoanEligibility

Example dashboard identifiers without spaces

"Hello"
"Traffic Violation"
"LoanEligibility"

However, the spaces within a dashboard identifier are maintained as follows:

Example dashboard identifiers with spaces

Traffic Violation, Traffic    Violation, Tra ffic Violation

Example dashboard identifiers without spaces

"Traffic Violation"
"Traffic    Violation"
"Tra ffic Violation"