Question & Answer
Question
I'm interested in updating the scrape_configs definitions, and this was previously done by customizing prometheus.yaml file.
In IBM Cloud Private 3.2.0 the prometheus.yaml could be customized as follows:
kubectl edit cm monitoring-prometheus
In IBM Cloud Private 3.2.2 this is no longer possible.
I have tried it as follows:
kubectl get secret prometheus-monitoring-prometheus -o yaml > prometheus-monitoring-prometheus.yaml
echo "<encoded-value>" | base64 -d > prometheus.yaml.gz
gunzip prometheus.yaml.gz
edit prometheus.yaml
gzip prometheus.yaml
cat prometheus.yaml.gz | base64 -w 0
kubectl edit secret prometheus-monitoring-prometheus
But changes are reset to default by IBM Cloud Private and the customization is not applied.
Why does it happen?
Answer
Most of the Prometheus configuration is now under the control of the Prometheus operator.
As you can see from:
As you can see from:
kind: Secret
metadata:
annotations:
generated: "true"
creationTimestamp: "2020-03-30T15:10:35Z"
labels:
managed-by: prometheus-operator
name: prometheus-monitoring-prometheus
Also this secret is managed by the Prometheus operator.
So any manual change is reverted to the settings provided by the Prometheus operator itself.
Theoretically, we should be able to edit prometheus.yml by editing the object prometheus called monitoring-prometheus, but
only a very small subset of parameters can be changed in this way.
So any manual change is reverted to the settings provided by the Prometheus operator itself.
Theoretically, we should be able to edit prometheus.yml by editing the object prometheus called monitoring-prometheus, but
only a very small subset of parameters can be changed in this way.
If you are interested in updating the scrape_configs definitions, this can be done editing another secret that actually contains the scrape_targets.yml.
Secret name is monitoring-prometheus-scrape-targets.
This secret is not managed by the Prometheus operator and can be used to provide custom targets.
The process is the same you used for the prometheus-monitoring-prometheus secret:
Secret name is monitoring-prometheus-scrape-targets.
This secret is not managed by the Prometheus operator and can be used to provide custom targets.
The process is the same you used for the prometheus-monitoring-prometheus secret:
1) kubectl get secret monitoring-prometheus-scrape-targets -o yaml > scrapetargets.yaml
2) echo "<encoded-value>" | base64 -d > additionalscrape.yml (this is not a gzipped file)
3) modify the content of additionalscrape.yml according to your need
4) cat additionalscrape.yml | base64 -w 0
5) kubectl edit secret monitoring-prometheus-scrape-targets (to change the scrape_targets.yml content with the string obtained in step 4.
2) echo "<encoded-value>" | base64 -d > additionalscrape.yml (this is not a gzipped file)
3) modify the content of additionalscrape.yml according to your need
4) cat additionalscrape.yml | base64 -w 0
5) kubectl edit secret monitoring-prometheus-scrape-targets (to change the scrape_targets.yml content with the string obtained in step 4.
The prometheus-config-reload container will recognize the changes and apply it after some seconds, you can then verify it directly into the Prometheus container into file /etc/prometheus/config_out/prometheus.env.yaml
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSBS6K","label":"IBM Cloud Private"},"ARM Category":[{"code":"a8m0z000000cvtsAAA","label":"IBM Cloud Private->Monitoring"}],"ARM Case Number":"TS004147999","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]
Product Synonym
IBM Cloud Private;ICP
Was this topic helpful?
Document Information
Modified date:
13 October 2020
UID
ibm16346844