Integrating external observability stack

Edit online

Configure the OpenTelemetry Collector to export telemetry data to your enterprise observability stack while maintaining the IBM® Sovereign Core built-in observability features.

Before you begin

Before you integrate an external observability stack, ensure you have:

Access to your enterprise observability platform
Endpoint URLs and authentication credentials for the external platform
Network connectivity between the platform and external observability tools
Administrative access to the OpenTelemetry Collector configuration
Understanding of OpenTelemetry exporter configuration

About this task

External observability stack integration allows you to send telemetry data to your enterprise monitoring tools while maintaining the platform's native observability features. The integration uses OpenTelemetry standards to ensure compatibility with various observability vendors.

This documentation includes a sample configuration for an external observability stack. However, the same steps apply to any tool stack that follows OpenTelemetry standards.

Procedure

Metrics configuration for Prometheus remote write

Use Prometheus remote write to send metrics from your OpenShift cluster to external monitoring platforms.
Identify the external observability endpoints and ensure they are reachable from the system owner clusters.
Review the integration example provided by your observability vendor.
Verify that your external observability platform is accessible.
Ensure network connectivity and that the platform meets security and compliance requirements.
Collect the required connection information for your observability platform:
1. Endpoint URLs for metrics, logs, and traces.
2. API keys or authentication tokens.
3. Required headers or metadata.
4. Data format requirements (for example, OTLP or Prometheus).

To configure metrics, complete the following steps:

Create a Kubernetes secret containing your observability vendor license key in the openshift-user-workload-monitoring namespace:

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: rw-bearer-auth
  namespace: openshift-user-workload-monitoring
stringData:
  token: <VENDO_LICENSE_KEY>
type: Opaque
EOF

Replace <VENDOR_LICENSE_KEY> with the actual license key.

Check whether the user-workload-monitoring-config ConfigMap exists:

oc get cm/user-workload-monitoring-config -n openshift-user-workload-monitoring -o yaml

If the ConfigMap does not exist, create it. If it exists, update it to enable remote write:

cat <<EOF | oc apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    namespacesWithoutLabelEnforcement:
      - servicemonitors
    prometheus:
      remoteWrite:
      - url: <REMOTE WRITE URL>
        authorization:
          credentials:
            name: rw-bearer-auth
            key: token
        writeRelabelConfigs:
        - sourceLabels: [__name__]
          regex: 'up|container_.*|node_.*|kube_.*'
          action: keep
EOF

The value of url value must direct to your platform's Prometheus remote write endpoint. Replace the <REMOTE WRITE URL> from Prometheus remote write URL that is published by the platform documentation.
The writeRelabelConfigs section is optional and can be used to filter metrics and reduce data volume.

Verify that Prometheus is running without errors. You can review the user workload Prometheus logs:
```
oc logs -f prometheus-user-workload-0 -n openshift-user-workload-monitoring | grep -i error
```

Optional: To send system and cluster metrics, update the cluster-monitoring-config ConfigMap in the openshift-monitoring namespace:

cat <<EOF | oc apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      remoteWrite:
      - url: <REMOTE WRITE URL>
        authorization:
          credentials:
            name: rw-bearer-auth
            key: token
EOF

Ensure that the rw-bearer-auth secret also exists in the openshift-monitoring namespace.

To send Telemetry Data Using OpenTelemetry (OTLP), complete the following steps:

Create a Kubernetes secret to store the OTLP authentication credentials.

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: otlp-bearer-token
  namespace: openshift-logging
stringData:
  token: <VENDOR_LICENSE_KEY>
type: Opaque
EOF

Configure the ClusterLogForwarder resource to define an OTLP output and enable log pipelines for application, infrastructure, and audit logs.

cat <<EOF | oc apply -f -
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
  - name: otlp
    type: otlp
    url: https://<OTLP log endpoint>:4318
    secret:
      name: otlp-bearer-token
    otlp:
      # Use gRPC or HTTP protocol
      protocol: http
      # Compression (optional)
      compression: gzip
      # TLS configuration
      tls:
        insecureSkipVerify: false
  pipelines:
  - name: send-application-logs
    inputRefs:
    - application
    outputRefs:
    - otlp
  - name: send-infrastructure-logs
    inputRefs:
    - infrastructure
    outputRefs:
    - otlp
  - name: send-audit-logs
    inputRefs:
    - audit
    outputRefs:
    - otlp
EOF

Validate that logs are being collected and forwarded by reviewing the collector logs.
```
oc logs -f  -l app.kubernetes.io/component=collector  -n openshift-logging
```

To send traces using OTLP, complete the following steps:

Configure the OpenTelemetry Collector with receivers, processors, and a vendor‑specific OTLP exporter endpoint.

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector
  namespace: openshift-opentelemetry-operator
spec:
  mode: deployment
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    
    processors:
      batch:
        timeout: 10s
        send_batch_size: 1024
      
      # Delete unnecessary attributes to reduce data volume
      attributes/delete:
        actions:
        - key: http.url
          action: delete
        - key: http.target
          action: delete
      
      # Delete unnecessary resource attributes
      resource/delete:
        attributes:
        - key: process.command_line
          action: delete
    
    exporters:
      otlphttp:
        endpoint: https://{VENDOR_OTLP_ENDPOINT}:4318
        headers:
          api-key: ${VENDOR_LICENSE_KEY}
        compression: gzip
    
    service:
      pipelines:
        traces:
          receivers:
            - otlp
          processors:
            - batch
            - attributes/delete
            - resource/delete
          exporters:
            - otlphttp
  
  env:
  - name: VENDOR_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: vendor-license-key
        key: license-key

Confirm that the OpenTelemetry Collector pods are running.
```
oc get pods -n otel-collector
```

Review the collector logs to verify that traces are being received and exported.

oc logs -n otel-collector -l app.kubernetes.io/component=opentelemetry-collector -f

Check for successful trace export messages in the logs.

oc logs -n otel-collector -l app.kubernetes.io/component=opentelemetry-collector | grep "Traces sent"

Results

Your external observability stack is integrated, and telemetry data is exported to your enterprise monitoring tools while maintaining the platform's native observability capabilities.

What to do next

Configure dashboards in your external observability platform for logs and traces.
Set up alerting rules based on enterprise monitoring requirements.
Monitor OpenTelemetry Collector logs for export success rates, latency, and failures.
Periodically review processor settings to optimize data volume and signal quality.

Regularly review the integration configuration and performance. Update exporter settings as needed to optimize data export and ensure reliable delivery to your external platform.