Refreshing bearer tokens for Prometheus authentication
Bearer tokens obtained from Cloud Pak for Data have a limited lifetime. Therefore, you must refresh the authentication token to keep Prometheus monitoring operational.
About this task
This is what happens when tokens expire:
- Prometheus receives 401 Unauthorized and 403 Forbidden responses as it tries to obtain metrics (called "scraping").
- The ServiceMonitor element reports a state of "Down"in the Prometheus UI.
- Metrics are not collected until the token is refreshed.
Procedure
To refresh your Prometheus authentication token:
What to do next
You can take a few actions to find out whether a bearer token secret update is necessary:
- Check whether metrics collections have failed. This is a good indicator for an expired bearer
token secret. Run the following Prometheus query for this purpose:
increase(datagate_table_sync_metrics_refresh_failures_total[10m]) > 0 - See if Prometheus user workload monitoring is enabled and
running:
oc get route -n openshift-user-workload-monitoring - Open the Prometheus UI and select on the sidebar. Check the target status that is displayed when you select and run the datagate-table-sync-metrics query.
In addition, you can take the following measures to automate bearer token secret updates:
- Create a cron job on the OpenShift server that refreshes the token and restarts the Prometheus pods.
- Use an external secrets operator for a centralized secrets management.
- Make the Data Gate pods use a sidecar container that manages the token lifecycle.