Troubleshooting
Problem
How can the time interval that alerts are received be changed? Trying to avoid receiving alerts every 30 seconds
Symptom
How can the time interval that alerts are received be changed? Trying to avoid receiving alerts every 30 seconds
Cause
Alerts are received every 30 seconds until resolved
Environment
WSL 1.2.2 on Linux with Microsoft Azure
Diagnosing The Problem
Alerts are received every 30 seconds. Need to change the interval in the config map for prometheus alert rules
Resolving The Problem
Steps necessary for changing the interval:
1. Run the command: kubectl -n sysibm-adm get cm -o yaml prometheus-alertmanager-cm it will return something like:
apiVersion: v1
data:
alertmanager.yml: |-
global:
resolve_timeout: 5m
data:
alertmanager.yml: |-
global:
resolve_timeout: 5m
route:
receiver: 'webhook'
group_by: ['alertname']
group_wait: 30s
group_interval: 30s
repeat_interval: 1h
receiver: 'webhook'
group_by: ['alertname']
group_wait: 30s
group_interval: 30s
repeat_interval: 1h
receivers:
- name: 'webhook'
webhook_configs:
- url: 'http://dash-back-svc:3010/alert/metrics-alert'
send_resolved: true
kind: ConfigMap
metadata:
creationTimestamp: 2018-10-08T14:41:37Z
name: prometheus-alertmanager-cm
namespace: sysibm-adm
resourceVersion: "2088"
selfLink: /api/v1/namespaces/sysibm-adm/configmaps/prometheus-alertmanager-cm
uid: 43011bec-cb08-11e8-a250-00163e01ec23
- name: 'webhook'
webhook_configs:
- url: 'http://dash-back-svc:3010/alert/metrics-alert'
send_resolved: true
kind: ConfigMap
metadata:
creationTimestamp: 2018-10-08T14:41:37Z
name: prometheus-alertmanager-cm
namespace: sysibm-adm
resourceVersion: "2088"
selfLink: /api/v1/namespaces/sysibm-adm/configmaps/prometheus-alertmanager-cm
uid: 43011bec-cb08-11e8-a250-00163e01ec23
2. Change the group_interval and the repeat_interval to the values of your choice with the command: kubectl -n sysibm-adm edit cm prometheus-alertmanager-cm
Then save the file to keep the changes.
Document Location
Worldwide
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSHGWL","label":"IBM Watson Studio Local"},"Component":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"1.2.2","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]
Was this topic helpful?
Document Information
Modified date:
30 December 2019
UID
ibm10886217