Error accessing Grafana UI

You might see an error when you access the Grafana UI.

Symptom

You see a 400: Bad request error.

Cause

The client certificate for Grafana might be expired or invalid. When the client certificate was renewed by the certificate manager, it was not picked up by the management ingress. This type of situation occurs when the client certificate was configured in the annotation of the Grafana ingress.

oc get ingress grafana-ingress -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    icp.management.ibm.com/authz-type: rbac
    icp.management.ibm.com/rewrite-target: /
    icp.management.ibm.com/secure-backends: "true"
    icp.management.ibm.com/secure-client-ca-secret: ibm-monitoring-client-certs
    kubernetes.io/ingress.class: ibm-icp-management
...
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: ibm-monitoring-grafana
          servicePort: 3000
        path: /grafana

Resolving the problem

To resolve the issue, restart the management ingress pod.

oc -n ibm-common-services delete pod <management ingress pods>

Wait for the pod to restart.

You can access the Grafana UI after the pod restarts.