The following is an example watch definition for
IBM Cloud Private environments where the watch is triggered every 5
minutes to load the Logstash logs that were written in the last 5 minutes and contain any of the
following keywords:
failed,
error, or
warning.
The watcher posts the payload for such logs to
event management using the webhook
URL.
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"indices": [
"logstash-2018*"
],
"body": {
"query": {
"bool": {
"must_not": {
"match": {
"kubernetes.container_name": "custom-metrics-adapter"
}
},
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-5m"
}
}
},
{
"terms": {
"log": [
"failed",
"error",
"warning"
]
}
}
]
}
}
}
}
}
},
"actions": {
"my_webhook": {
"webhook": {
"method": "POST",
"headers": {
"Content-Type": "application/json"
},
"url": "<CEM WEBHOOK>",
"body": "{{#toJson}}ctx.payload{{/toJson}}"
}
}
}
}
Important: Ensure you set the trigger for the watch to a frequency that suits your
requirements for monitoring the logs. Consider the load on the system when setting frequency. In the
previous example, the watch is triggered every 5 minutes to load the logs that were written in the
last 5 minutes using the "schedule": {"interval": "5m"} and "@timestamp":
{"gte": "now-5m" } settings. If you set interval to less than 5 minutes in
this case, then the same logs are sent to event management more than once, repeating event data in
the correlated incidents.
Restriction: The "terms": {"log": []} section in the
watch definition determines the mapping to the event severity levels in event management. The default values are "failed",
"error", and "warning", and are mapped to "critical", "major", and "minor" severity levels. If you
use any other value, the event severity is mapped to "indeterminate" in event management.
Attention: In
IBM Cloud Private environments ensure you exclude
"kubernetes.container_name":
"custom-metrics-adapter" from your watch definition using the following
setting:
"must_not": {
"match": {
"kubernetes.container_name": "custom-metrics-adapter"
}
The
size of the
custom-metric-adapter logs can be large and overload the
event management processing. In addition, the log format
is unreadable to users.
Copy to add the generated webhook URL to the clipboard. Ensure you save the
generated webhook to make it available later in the configuration process. For example, you can save
it to a file.