IBM Support

How to update the scrape_configs definitions for Prometheus on IBM Cloud Private 3.2.2

Question & Answer


I'm interested in updating the scrape_configs definitions, and this was previously done by customizing prometheus.yaml file.
In IBM Cloud Private 3.2.0 the prometheus.yaml could be customized as follows:
kubectl edit cm monitoring-prometheus
In IBM Cloud Private 3.2.2 this is no longer possible.
I have tried it as follows:
kubectl get secret prometheus-monitoring-prometheus -o yaml > prometheus-monitoring-prometheus.yaml
echo "<encoded-value>" | base64 -d > prometheus.yaml.gz
gunzip prometheus.yaml.gz
edit   prometheus.yaml
gzip prometheus.yaml
cat prometheus.yaml.gz | base64 -w 0
kubectl edit secret prometheus-monitoring-prometheus
But changes are reset to default by IBM Cloud Private and the customization is not applied.
Why does it happen?


Most of the Prometheus configuration is now under the control of the Prometheus operator.
As you can see from:
kind: Secret
generated: "true"
creationTimestamp: "2020-03-30T15:10:35Z"
managed-by: prometheus-operator
name: prometheus-monitoring-prometheus

Also this secret is managed by the Prometheus operator.
So any manual change is reverted to the settings provided by the Prometheus operator itself.
Theoretically, we should be able to edit prometheus.yml by editing the object prometheus called monitoring-prometheus, but
only a very small subset of parameters can be changed in this way.
If you are interested in updating the scrape_configs definitions, this can be done editing another secret that actually contains the scrape_targets.yml.
Secret name is  monitoring-prometheus-scrape-targets.
This secret is not managed by the Prometheus operator and can be used to provide custom targets.
The process is the same you used for the  prometheus-monitoring-prometheus secret:
1) kubectl get secret monitoring-prometheus-scrape-targets -o yaml > scrapetargets.yaml
2) echo "<encoded-value>" | base64 -d > additionalscrape.yml  (this is not a gzipped file)
3) modify the content of  additionalscrape.yml according to your need
4) cat additionalscrape.yml | base64 -w 0
5) kubectl edit secret monitoring-prometheus-scrape-targets (to change the scrape_targets.yml content with the string obtained in step 4.
The prometheus-config-reload container will recognize the changes and apply it after some seconds, you can then verify it directly into the Prometheus container into file /etc/prometheus/config_out/prometheus.env.yaml

[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSBS6K","label":"IBM Cloud Private"},"ARM Category":[{"code":"a8m0z000000cvtsAAA","label":"IBM Cloud Private->Monitoring"}],"ARM Case Number":"TS004147999","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]

Product Synonym

IBM Cloud Private;ICP

Document Information

Modified date:
13 October 2020