Exporting Event Gateway metrics and traces with OpenTelemetry

Configure OpenTelemetry (OTEL) to export metrics from your Event Gateway.

The Event Gateway can be configured to send OTEL trace information from the gateway to IBM webMethods Hybrid Integration End-to-end Monitoring. This monitoring enables you to see how Kafka messages travel through the gateway and trace the messages as other applications within IBM webMethods Hybrid Integration see them.

To enable this monitoring feature, you must have access to the IBM webMethods Hybrid Integration End-to-end Monitoring administrator console, where you can generate an API Key to use IBM webMethods Hybrid Integration End-to-end Monitoring.

Generate an API key by using the IBM webMethods Hybrid Integration End-to-end Monitoring administrator console. To generate the API key:
1. In the IBM webMethods Hybrid Integration End-to-end Monitoring UI, click Manage > API key in the left pane.
2. Click Generate API Key.
On Kubernetes Deployment and operator-managed gateways, create a secret to store your API key:
```
kubectl -n <namespace> create secret generic otel-api-key --from-literal=apikeystring="Authorization=mcsp-key apikey=<api key>"
```
Replace <api key> with your API key.

Update your Event Gateway configuration.

On Kubernetes Deployment gateways, set egw.enable.otel.metrics="true" in your Event Gateway ConfigMap. Then set the spec.containers[egw].env variables as shown:

        # Optional env for enabling kafka record end to end monitoring
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "https://<otel endpoint>"
        - name: OTEL_LOGS_EXPORTER
          value: none
        - name: OTEL_METRICS_EXPORTER
          value: none
        - name: OTEL_SERVICE_NAME
          value: "<unique name>"
        - name: IBM_JAVA_OPTIONS
          value: "-javaagent:/opt/ibm/gateway/lib/opentelemetry-javaagent.jar
        - name: OTEL_EXPORTER_OTLP_HEADERS
          valueFrom:
            secretKeyRef:
              name: otel-api-key
              key: "apikeystring"

On Docker gateways, restart your the gateway instance with the following arguments configured:

-e OTEL_EXPORTER_OTLP_ENDPOINT="https://<otel endpoint>" \
-e OTEL_LOGS_EXPORTER="none" \
-e OTEL_METRICS_EXPORTER="none" \
-e OTEL_SERVICE_NAME="<unique name>" \
-e IBM_JAVA_OPTIONS="-javaagent:/opt/ibm/gateway/lib/opentelemetry-javaagent.jar" \
-e EGW_ENABLE_OTEL_METRICS="false" \
-e EGW_ENABLE_OTEL_KAFKA_RECORD_TRACING="true" \
-e OTEL_EXPORTER_OTLP_HEADERS="Authorization=mcsp-key apikey=<api key>" \

On operator-managed gateways, update the Event Gateway custom resource and set spec.openTelemetry as shown:

  # Optional config for enabling kafka record end to end monitoring
  openTelemetry:
    endpoint: "https://<otel endpoint>"
    metricsEnablement:
      - name: gateway
        enabled: false
    tracesEnablement:
      - name: kafkaRecord
        enabled: true

and set the spec.template.pod.spec.containers[egw].env variables as shown:

              # Optional env for enabling kafka record end to end monitoring
              - name: OTEL_EXPORTER_OTLP_HEADERS
                valueFrom:
                  secretKeyRef:
                    name: otel-api-key
                    key: "apikeystring"

Use the following tables to determine the values to use for the placeholders.

Table 1. Required OTEL properties
Variable name	Default value	Description
`OTEL_EXPORTER_OTLP_ENDPOINT`		The HTTP address where the OTEL data is sent. For example: `https://otel.example.com:4317`. The address can be found in the IBM webMethods Hybrid Integration End-to-end Monitoring UI.
`OTEL_EXPORTER_OTLP_HEADERS`		Must contain the API key as shown: `"Authorization=mcsp-key apikey=<api key>"` This property can also include other custom headers.
`EGW_ENABLE_OTEL_KAFKA_RECORD_TRACING`	`false`	Enable the export of Kafka record traces.
`EGW_ENABLE_OTEL_METRICS`	`true`	Enable the export of gateway metrics.
`OTEL_INSTRUMENTATION_<NAME>_ENABLED`		Use `OTEL_INSTRUMENTATION_<NAME>_ENABLED` to enable additional instrumentations. Replace `<NAME>` with the instrumentation that you want. For more information see: OTEL instrumentations.
`OTEL_SERVICE_NAME`		A string to uniquely identify the Event Gateway in the service traces.

Table 2. Optional OTEL properties
Variable name	Default value	Description
`OTEL_EXPORTER_OTLP_PROTOCOL`	`http/protobuf`	The communication protocol to use for communicating to the endpoint. Example values are `grpc` and `http/protobuf`.
`OTEL_METRIC_EXPORT_INTERVAL`	30000 ms	The interval in milliseconds between the start of two export attempts.
`OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE`		Path to the mounted file in the Event Gateway container that contains the OTEL client's trusted certificates.
`OTEL_EXPORTER_OTLP_CLIENT_KEY`		Path to the mounted file in the Event Gateway container that contains the mTLS client key.

Restart the Event Gateway.

OpenTelemetry metrics reference

The following table shows the OTEL metrics that are emitted by the Event Gateway:

Metric	Type	Description
login_success	LongCounter	Counts the number of successful client logins to the Event Gateway.
login_failed	LongCounter	Counts the number of failed client logins to the Event Gateway.
topic_authz_failed	LongCounter	Counts the number of topic authorization failures that are caused by clients that use the Event Gateway.
client_api_versions_gauge	LongGauge	Gauge that contains the API versions used per client ID for each Kafka API.
connected_clients	LongUpDownCounter	Total number of clients connected to this gateway at a specific point in time.
consumers_msgs	LongCounter	Counts the number of messages that clients consume through the Event Gateway.
consumers_bytes	LongCounter	Counts the number of bytes that clients consume through the Event Gateway.
producers_bytes	LongCounter	Counts the number of messages that clients produce through the Event Gateway.
producers_msgs	LongCounter	Counts the number of bytes that clients produce through the Event Gateway.
quota_delay	LongUpDownCounter	Available if the quota enforcement control is enabled. It provides the delays that are applied to ensure that quota limitations are maintained per client by using metadata attributes.