Example of natural language log anomaly detection

This topic provides an examples of log anomalies generated by the natural language log anomaly detection algorithm and when they are likely to be found.

Example: unexpected log pattern

You recently installed an application, which is made up of multiple microservices. Most of the microservices that make up this application generate their own log files.

Note For the natural language log anomaly detection algorithm to work, the following minimum data requirements must be met. At least 75% of the component microservices must have at least 2000 data points, where a data point is usually a log line.

The example proceeds as follows:

  1. Data from the microservices is trained, as described in Workflow of natural language log anomaly detection.

  2. During training, log patterns statistics are gathered and stored in the initial version of the model, as described in Workflow of natural language log anomaly detection. One of the patterns identified in the model is the following pattern within the message keyword.

    incident euid=<NUM> epid=<NUM> dsteuid=<NUM> dstepid=<NUM> <*>type=undefined subtype=undefined level=notice

    This pattern occurs on average roughly 2.5 times in every 10 second time slot. See Clean log file lines for an example of what a log like this might look like.

  3. When the model is applied to the live data stream as described in Workflow of natural language log anomaly detection, at a certain point the log pattern described in the previous step starts occurring in a number of the 10 second time slots. See Error log file lines for an example of what a log like this might look like.

  4. Log anomaly alerts are generated for each log anomaly and sent to the in-cluster ObjectServer. These alerts report that a pattern has been detected with higher frequency than expected. See Log anomaly alert format for an example of what a log anomaly alert might look like.

Reference information

The following information is referenced in this task.

Clean log file lines

These example JSON log snippets display log lines for one of the microservices within the application.

Clean log file lines

The pattern of interest is:

incident euid=<NUM> epid=<NUM> dsteuid=<NUM> dstepid=<NUM> <*>type=undefined subtype=undefined level=notice

You can find this pattern in the message parameter within the log lines below by searching for the invariant code type=undefined subtype=undefined level=notice. This will show you that the pattern of interest only occurs once, in the first line.

In a real life example, the number of lines and the number of occurrences of the pattern in a 10 second time slot would be consistent with the two or three times that is expected based on the values in the model.

[
    {"_index":"filebeat-7.12.1-2021.08.20-000001","_type":"_doc","_id":"zpprensBxt31inA6cZbF","_score":null,"_source":{"@timestamp":"2021-08-24T23:08:00.283Z","stream":"stdout","input":{"type":"container"},"container":{"runtime":"docker","id":"bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53","image":{"name":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024"}},"ecs":{"version":"1.8.0"},"log":{"offset":640306,"file":{"path":"/var/log/containers/qotd-ratings-5444ddd79c-gp7kb_qotd_qotd-rating-bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53.log"}},"message":"incident euid=2570 epid=7324 dsteuid=2411 dstepid=7148 vd=Jarret10 type=undefined subtype=undefined level=notice","kubernetes":{"container":{"name":"qotd-rating","image":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024"},"deployment":{"name":"qotd-ratings"},"node":{"uid":"193183f9-01bd-11ec-926b-0617a6095287","labels":{"node-role_kubernetes_io/master":"true","beta_kubernetes_io/arch":"amd64","beta_kubernetes_io/os":"linux","kubernetes_io/hostname":"padme-okd.coc-ibm.com","node-role_kubernetes_io/compute":"true","node-role_kubernetes_io/infra":"true"},"hostname":"padme-okd.coc-ibm.com","name":"padme-okd.coc-ibm.com"},"namespace_uid":"dcc7d8e8-01bf-11ec-9802-0617a6095287","pod":{"name":"qotd-ratings-5444ddd79c-gp7kb","uid":"d9b9549f-0529-11ec-9802-0617a6095287"},"namespace":"qotd","replicaset":{"name":"qotd-ratings-5444ddd79c"},"labels":{"pod-template-hash":"1000888357","app":"qotd-rating"}},"agent":{"hostname":"padme-okd.coc-ibm.com","ephemeral_id":"39a50623-e967-451f-b30d-f06fd421d0cf","id":"456adf88-c87d-49d2-8aa2-9b93d39ba057","name":"padme-okd.coc-ibm.com","type":"filebeat","version":"7.12.1"},"host":{"name":"padme-okd.coc-ibm.com"}},"sort":[1629846480283,640306]}

    {"_index":"filebeat-7.12.1-2021.08.20-000001","_type":"_doc","_id":"z5prensBxt31inA6d5Z9","_score":null,"_source":{"@timestamp":"2021-08-24T23:08:02.065Z","agent":{"version":"7.12.1","hostname":"padme-okd.coc-ibm.com","ephemeral_id":"39a50623-e967-451f-b30d-f06fd421d0cf","id":"456adf88-c87d-49d2-8aa2-9b93d39ba057","name":"padme-okd.coc-ibm.com","type":"filebeat"},"log":{"offset":11804878,"file":{"path":"/var/log/containers/qotd-web-5cbb5d9f64-srmc4_qotd_qotd-web-f6c3fb360307ff0759831d56124674a15af7d14e135a19928f4b3641aeee605c.log"}},"stream":"stdout","message":"Starting new request token: 783499 for IP: 10.128.0.41","input":{"type":"container"},"kubernetes":{"container":{"name":"qotd-web","image":"registry.gitlab.com/quote-of-the-day/qotd-web:latest"},"deployment":{"name":"qotd-web"},"node":{"labels":{"kubernetes_io/hostname":"padme-okd.coc-ibm.com","node-role_kubernetes_io/compute":"true","node-role_kubernetes_io/infra":"true","node-role_kubernetes_io/master":"true","beta_kubernetes_io/arch":"amd64","beta_kubernetes_io/os":"linux"},"hostname":"padme-okd.coc-ibm.com","name":"padme-okd.coc-ibm.com","uid":"193183f9-01bd-11ec-926b-0617a6095287"},"namespace_uid":"dcc7d8e8-01bf-11ec-9802-0617a6095287","pod":{"uid":"efbf8ddd-04fc-11ec-9802-0617a6095287","name":"qotd-web-5cbb5d9f64-srmc4"},"namespace":"qotd","replicaset":{"name":"qotd-web-5cbb5d9f64"},"labels":{"app":"qotd-web","pod-template-hash":"1766185920"}},"container":{"runtime":"docker","id":"f6c3fb360307ff0759831d56124674a15af7d14e135a19928f4b3641aeee605c","image":{"name":"registry.gitlab.com/quote-of-the-day/qotd-web:latest"}},"ecs":{"version":"1.8.0"},"host":{"name":"padme-okd.coc-ibm.com"}},"sort":[1629846482065,11804878]}

    {"_index":"filebeat-7.12.1-2021.08.20-000001","_type":"_doc","_id":"0JprensBxt31inA6d5Z9","_score":null,"_source":{"@timestamp":"2021-08-24T23:08:02.066Z","host":{"name":"padme-okd.coc-ibm.com"},"ecs":{"version":"1.8.0"},"stream":"stdout","message":"[783499] Web request: /.","input":{"type":"container"},"kubernetes":{"pod":{"uid":"efbf8ddd-04fc-11ec-9802-0617a6095287","name":"qotd-web-5cbb5d9f64-srmc4"},"namespace":"qotd","replicaset":{"name":"qotd-web-5cbb5d9f64"},"labels":{"pod-template-hash":"1766185920","app":"qotd-web"},"container":{"name":"qotd-web","image":"registry.gitlab.com/quote-of-the-day/qotd-web:latest"},"deployment":{"name":"qotd-web"},"node":{"hostname":"padme-okd.coc-ibm.com","name":"padme-okd.coc-ibm.com","uid":"193183f9-01bd-11ec-926b-0617a6095287","labels":{"kubernetes_io/hostname":"padme-okd.coc-ibm.com","node-role_kubernetes_io/compute":"true","node-role_kubernetes_io/infra":"true","node-role_kubernetes_io/master":"true","beta_kubernetes_io/arch":"amd64","beta_kubernetes_io/os":"linux"}},"namespace_uid":"dcc7d8e8-01bf-11ec-9802-0617a6095287"},"container":{"runtime":"docker","id":"f6c3fb360307ff0759831d56124674a15af7d14e135a19928f4b3641aeee605c","image":{"name":"registry.gitlab.com/quote-of-the-day/qotd-web:latest"}},"log":{"offset":11805003,"file":{"path":"/var/log/containers/qotd-web-5cbb5d9f64-srmc4_qotd_qotd-web-f6c3fb360307ff0759831d56124674a15af7d14e135a19928f4b3641aeee605c.log"}},"agent":{"name":"padme-okd.coc-ibm.com","type":"filebeat","version":"7.12.1","hostname":"padme-okd.coc-ibm.com","ephemeral_id":"39a50623-e967-451f-b30d-f06fd421d0cf","id":"456adf88-c87d-49d2-8aa2-9b93d39ba057"}},"sort":[1629846482066,11805003]}

    {"_index":"filebeat-7.12.1-2021.08.20-000001","_type":"_doc","_id":"0ZprensBxt31inA6d5Z9","_score":null,"_source":{"@timestamp":"2021-08-24T23:08:02.182Z","input":{"type":"container"},"kubernetes":{"deployment":{"name":"qotd-web"},"node":{"name":"padme-okd.coc-ibm.com","uid":"193183f9-01bd-11ec-926b-0617a6095287","labels":{"node-role_kubernetes_io/compute":"true","node-role_kubernetes_io/infra":"true","node-role_kubernetes_io/master":"true","beta_kubernetes_io/arch":"amd64","beta_kubernetes_io/os":"linux","kubernetes_io/hostname":"padme-okd.coc-ibm.com"},"hostname":"padme-okd.coc-ibm.com"},"namespace_uid":"dcc7d8e8-01bf-11ec-9802-0617a6095287","pod":{"uid":"efbf8ddd-04fc-11ec-9802-0617a6095287","name":"qotd-web-5cbb5d9f64-srmc4"},"namespace":"qotd","replicaset":{"name":"qotd-web-5cbb5d9f64"},"labels":{"app":"qotd-web","pod-template-hash":"1766185920"},"container":{"name":"qotd-web","image":"registry.gitlab.com/quote-of-the-day/qotd-web:latest"}},"host":{"name":"padme-okd.coc-ibm.com"},"agent":{"id":"456adf88-c87d-49d2-8aa2-9b93d39ba057","name":"padme-okd.coc-ibm.com","type":"filebeat","version":"7.12.1","hostname":"padme-okd.coc-ibm.com","ephemeral_id":"39a50623-e967-451f-b30d-f06fd421d0cf"},"stream":"stdout","message":"[783499] Getting daily quote.","log":{"file":{"path":"/var/log/containers/qotd-web-5cbb5d9f64-srmc4_qotd_qotd-web-f6c3fb360307ff0759831d56124674a15af7d14e135a19928f4b3641aeee605c.log"},"offset":11805098},"container":{"image":{"name":"registry.gitlab.com/quote-of-the-day/qotd-web:latest"},"runtime":"docker","id":"f6c3fb360307ff0759831d56124674a15af7d14e135a19928f4b3641aeee605c"},"ecs":{"version":"1.8.0"}},"sort":[1629846482182,11805098]}

    {"_index":"filebeat-7.12.1-2021.08.20-000001","_type":"_doc","_id":"3pprensBxt31inA6h5Yj","_score":null,"_source":{"@timestamp":"2021-08-24T23:08:02.195Z","kubernetes":{"namespace":"qotd","replicaset":{"name":"qotd-quote-585979769"},"labels":{"app":"qotd-quote","pod-template-hash":"141535325"},"container":{"name":"qotd-quote","image":"registry.gitlab.com/quote-of-the-day/quote-service:latest"},"deployment":{"name":"qotd-quote"},"node":{"uid":"193183f9-01bd-11ec-926b-0617a6095287","labels":{"node-role_kubernetes_io/compute":"true","node-role_kubernetes_io/infra":"true","node-role_kubernetes_io/master":"true","beta_kubernetes_io/arch":"amd64","beta_kubernetes_io/os":"linux","kubernetes_io/hostname":"padme-okd.coc-ibm.com"},"hostname":"padme-okd.coc-ibm.com","name":"padme-okd.coc-ibm.com"},"namespace_uid":"dcc7d8e8-01bf-11ec-9802-0617a6095287","pod":{"name":"qotd-quote-585979769-wpg25","uid":"e810e7e8-04fc-11ec-9802-0617a6095287"}},"container":{"image":{"name":"registry.gitlab.com/quote-of-the-day/quote-service:latest"},"runtime":"docker","id":"ef6bf31678a9d80b5ac95a27dbb72f68a2f855aa09874e6eda222e4ff1c3184e"},"log":{"file":{"path":"/var/log/containers/qotd-quote-585979769-wpg25_qotd_qotd-quote-ef6bf31678a9d80b5ac95a27dbb72f68a2f855aa09874e6eda222e4ff1c3184e.log"},"offset":6052045},"input":{"type":"container"},"agent":{"id":"456adf88-c87d-49d2-8aa2-9b93d39ba057","name":"padme-okd.coc-ibm.com","type":"filebeat","version":"7.12.1","hostname":"padme-okd.coc-ibm.com","ephemeral_id":"39a50623-e967-451f-b30d-f06fd421d0cf"},"ecs":{"version":"1.8.0"},"host":{"name":"padme-okd.coc-ibm.com"},"stream":"stdout","message":"[783499] Quote request: /daily."},"sort":[1629846482195,6052045]}
]

Error log file lines

These example JSON log snippets display log lines for one of the microservices within the application. You can see that the pattern of interest occurs in every line in this log snippet, in all five times.

The pattern of interest is:

incident euid=<NUM> epid=<NUM> dsteuid=<NUM> dstepid=<NUM> <*>type=undefined subtype=undefined level=notice

You can find this pattern in the message parameter within the log lines below by searching for the invariant code type=undefined subtype=undefined level=notice. This will show you which lines of code instantiate this pattern.

In a real life example, the number of lines and the number of occurrences of the pattern in a 10 second time slot would be significantly larger. You might have 100 or so log lines in 10 seconds and the pattern might occur on the order of 20 times, which is a significant deviation from the two or three times that is expected based on the values in the model. Based on this, log anomaly alerts would be generated.

Error log file lines
[
    {"_index":"filebeat-7.12.1-2021.08.20-000001","_type":"_doc","_id":"pppqensBxt31inA6h5Qa","_score":null,"_source":{"@timestamp":"2021-08-24T23:07:00.053Z","input":      {"type":"container"},"container":          {"id":"bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53","image":{"name":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024"},"runtime":"docker"},"host":{"name":"padme-okd.coc-ibm.com"},"log":{"file":{"path":"/var/log/containers/qotd-ratings-5444ddd79c-gp7kb_qotd_qotd-rating-bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53.log"},"offset":610747},"stream":"stdout","message":"incident euid=5840 epid=9800 dsteuid=3612 dstepid=7788 vd=Corbin_Gerlach59 type=undefined subtype=undefined level=notice","kubernetes":{"namespace_uid":"dcc7d8e8-01bf-11ec-9802-0617a6095287","pod":{"name":"qotd-ratings-5444ddd79c-gp7kb","uid":"d9b9549f-0529-11ec-9802-0617a6095287"}, "namespace":"qotd","replicaset":{"name":"qotd-ratings-5444ddd79c"},"labels":{"app":"qotd-rating","pod-template-hash":"1000888357"},"container":{"name":"qotd-rating","image":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024"},"deployment":{"name":"qotd-ratings"},"node":{"hostname":"padme-okd.coc-ibm.com","name":"padme-okd.coc-ibm.com","uid":"193183f9-01bd-11ec-926b-0617a6095287","labels":   {"beta_kubernetes_io/arch":"amd64","beta_kubernetes_io/os":"linux","kubernetes_io/hostname":"padme-okd.coc-ibm.com","node-role_kubernetes_io/compute":"true","node-role_kubernetes_io/infra":"true","node-role_kubernetes_io/master":"true"}}},"ecs":{"version":"1.8.0"},"agent":{"name":"padme-okd.coc-ibm.com","type":"filebeat",        "version":"7.12.1","hostname":"padme-okd.coc-ibm.com","ephemeral_id":"39a50623-e967-451f-b30d-f06fd421d0cf","id":"456adf88-c87d-49d2-8aa2-9b93d39ba057"}}
    ,"sort":[1629846420053,610747]},

    {"_index":"filebeat-7.12.1-2021.08.20-000001","_type":"_doc","_id":"qppqensBxt31inA6h5Qa","_score":null,"_source":{"@timestamp":"2021-08-24T23:07:00.578Z","stream":"stdout","message":"incident euid=5896 epid=8870 dsteuid=4250 dstepid=5530 vd=Sunny39 type=undefined subtype=undefined level=notice","kubernetes":{"pod":{"name":"qotd-ratings-5444ddd79c-gp7kb","uid":"d9b9549f-0529-11ec-9802-0617a6095287"},"namespace":"qotd","replicaset":{"name":"qotd-ratings-5444ddd79c"},"labels":{"app":"qotd-rating","pod-template-hash":"1000888357"},"container":{"image":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024","name":"qotd-rating"},"deployment":{"name":"qotd-ratings"},"node":{"name":"padme-okd.coc-ibm.com","uid":"193183f9-01bd-11ec-926b-0617a6095287","labels":{"beta_kubernetes_io/os":"linux","kubernetes_io/hostname":"padme-okd.coc-ibm.com","node-role_kubernetes_io/compute":"true","node-role_kubernetes_io/infra":"true","node-role_kubernetes_io/master":"true","beta_kubernetes_io/arch":"amd64"},"hostname":"padme-okd.coc-ibm.com"},"namespace_uid":"dcc7d8e8-01bf-11ec-9802-0617a6095287"},"ecs":{"version":"1.8.0"},"host":{"name":"padme-okd.coc-ibm.com"},"log":{"offset":611253,"file":{"path":"/var/log/containers/qotd-ratings-5444ddd79c-gp7kb_qotd_qotd-rating-bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53.log"}},"input":{"type":"container"},"container":{"image":{"name":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024"},"runtime":"docker","id":"bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53"},"agent":{"id":"456adf88-c87d-49d2-8aa2-9b93d39ba057","name":"padme-okd.coc-ibm.com","type":"filebeat","version":"7.12.1","hostname":"padme-okd.coc-ibm.com","ephemeral_id":"39a50623-e967-451f-b30d-f06fd421d0cf"}},"sort":[1629846420578,611253]},

    {"_index":"filebeat-7.12.1-2021.08.20-000001","_type":"_doc","_id":"vJpqensBxt31inA6i5QD","_score":null,"_source":{"@timestamp":"2021-08-24T23:07:01.057Z","input":{"type":"container"},"kubernetes":{"labels":{"app":"qotd-rating","pod-template-hash":"1000888357"},"container":{"name":"qotd-rating","image":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024"},"deployment":{"name":"qotd-ratings"},"node":{"uid":"193183f9-01bd-11ec-926b-0617a6095287","labels":{"node-role_kubernetes_io/compute":"true","node-role_kubernetes_io/infra":"true","node-role_kubernetes_io/master":"true","beta_kubernetes_io/arch":"amd64","beta_kubernetes_io/os":"linux","kubernetes_io/hostname":"padme-okd.coc-ibm.com"},"hostname":"padme-okd.coc-ibm.com","name":"padme-okd.coc-ibm.com"},"namespace_uid":"dcc7d8e8-01bf-11ec-9802-0617a6095287","pod":{"name":"qotd-ratings-5444ddd79c-gp7kb","uid":"d9b9549f-0529-11ec-9802-0617a6095287"},"namespace":"qotd","replicaset":{"name":"qotd-ratings-5444ddd79c"}},"container":{"id":"bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53","image":{"name":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024"},"runtime":"docker"},"host":{"name":"padme-okd.coc-ibm.com"},"log":{"file":{"path":"/var/log/containers/qotd-ratings-5444ddd79c-gp7kb_qotd_qotd-rating-bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53.log"},"offset":611542},"stream":"stdout","message":"incident euid=2210 epid=3789 dsteuid=3713 dstepid=4212 vd=Garrick40 type=undefined subtype=undefined level=notice","agent":{"name":"padme-okd.coc-ibm.com","type":"filebeat","version":"7.12.1","hostname":"padme-okd.coc-ibm.com","ephemeral_id":"39a50623-e967-451f-b30d-f06fd421d0cf","id":"456adf88-c87d-49d2-8aa2-9b93d39ba057"},"ecs":{"version":"1.8.0"}},"sort":[1629846421057,611542]},

    {"_index":"filebeat-7.12.1-2021.08.20-000001","_type":"_doc","_id":"vZpqensBxt31inA6i5QD","_score":null,"_source":{"@timestamp":"2021-08-24T23:07:01.584Z","log":{"offset":611726,"file":{"path":"/var/log/containers/qotd-ratings-5444ddd79c-gp7kb_qotd_qotd-rating-bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53.log"}},"input":{"type":"container"},"container":{"image":{"name":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024"},"runtime":"docker","id":"bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53"},"message":"incident euid=8003 epid=5254 dsteuid=2568 dstepid=6719 vd=Sylvan.Romaguera type=undefined subtype=undefined level=notice","stream":"stdout","kubernetes":{"pod":{"uid":"d9b9549f-0529-11ec-9802-0617a6095287","name":"qotd-ratings-5444ddd79c-gp7kb"},"namespace":"qotd","replicaset":{"name":"qotd-ratings-5444ddd79c"},"labels":{"app":"qotd-rating","pod-template-hash":"1000888357"},"container":{"name":"qotd-rating","image":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024"},"deployment":{"name":"qotd-ratings"},"node":{"name":"padme-okd.coc-ibm.com","uid":"193183f9-01bd-11ec-926b-0617a6095287","labels":{"node-role_kubernetes_io/compute":"true","node-role_kubernetes_io/infra":"true","node-role_kubernetes_io/master":"true","beta_kubernetes_io/arch":"amd64","beta_kubernetes_io/os":"linux","kubernetes_io/hostname":"padme-okd.coc-ibm.com"},"hostname":"padme-okd.coc-ibm.com"},"namespace_uid":"dcc7d8e8-01bf-11ec-9802-0617a6095287"},"host":{"name":"padme-okd.coc-ibm.com"},"agent":{"ephemeral_id":"39a50623-e967-451f-b30d-f06fd421d0cf","id":"456adf88-c87d-49d2-8aa2-9b93d39ba057","name":"padme-okd.coc-ibm.com","type":"filebeat","version":"7.12.1","hostname":"padme-okd.coc-ibm.com"},"ecs":{"version":"1.8.0"}},"sort":[1629846421584,611726]},
    {"_index":"filebeat-7.12.1-2021.08.20-000001","_type":"_doc","_id":"0ppqensBxt31inA6jpTq","_score":null,"_source":{"@timestamp":"2021-08-24T23:07:02.092Z","stream":"stdout","input":{"type":"container"},"kubernetes":{"node":{"name":"padme-okd.coc-ibm.com","uid":"193183f9-01bd-11ec-926b-0617a6095287","labels":{"node-role_kubernetes_io/infra":"true","node-role_kubernetes_io/master":"true","beta_kubernetes_io/arch":"amd64","beta_kubernetes_io/os":"linux","kubernetes_io/hostname":"padme-okd.coc-ibm.com","node-role_kubernetes_io/compute":"true"},"hostname":"padme-okd.coc-ibm.com"},"namespace_uid":"dcc7d8e8-01bf-11ec-9802-0617a6095287","pod":{"uid":"d9b9549f-0529-11ec-9802-0617a6095287","name":"qotd-ratings-5444ddd79c-gp7kb"},"namespace":"qotd","replicaset":{"name":"qotd-ratings-5444ddd79c"},"labels":{"pod-template-hash":"1000888357","app":"qotd-rating"},"container":{"name":"qotd-rating","image":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024"},"deployment":{"name":"qotd-ratings"}},"ecs":{"version":"1.8.0"},"agent":{"version":"7.12.1","hostname":"padme-okd.coc-ibm.com","ephemeral_id":"39a50623-e967-451f-b30d-f06fd421d0cf","id":"456adf88-c87d-49d2-8aa2-9b93d39ba057","name":"padme-okd.coc-ibm.com","type":"filebeat"},"log":{"file":{"path":"/var/log/containers/qotd-ratings-5444ddd79c-gp7kb_qotd_qotd-rating-bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53.log"},"offset":612021},"message":"incident euid=2850 epid=7575 dsteuid=3306 dstepid=2473 vd=Keshaun_Berge96 type=undefined subtype=undefined level=notice","container":{"id":"bc5dd98411a160669c3b0cf8af424f51d0a504cc27f7026a682318389f5ffe53","image":{"name":"registry.gitlab.com/quote-of-the-day/qotd-ratings-service:build-354180024"},"runtime":"docker"},"host":{"name":"padme-okd.coc-ibm.com"}},"sort":[1629846422092,612021]}
]

Log anomaly alert example

This example alert incorporates data from the identified log anomalies into an ObjectServer alert structure. Note that based on the values in the details.log_anomaly_detector.causality.template_list field, three different templates have been identified as occurring with significant deviations from the model and have been grouped together within this anomaly.

[
    {"id": "a21bd7c9-b3f6-3684-b900-12057634cb66",
    "occurrenceTime": "2021-08-24T23:07:00.000Z",
    "sender": {"name": "Log Anomaly"},
    "event_link": "https://api.us-south.logging.cloud.ibm.com/v1/export?to=1629846430000&from=1629846420000",
    "summary": "Abnormal behavior found in component: qotd-rating",
    "description": "[<NUM>] Ratings request, id: <NUM>: 3\n[<NUM>] The monkey's dart hit the <NUM>: 4\nHeartbeat status: CPU=\"OK\" Memory=\"OK\": 2\nincident euid=<NUM> epid=<NUM> dsteuid=<NUM> dstepid=<NUM> <*>type=undefined subtype=undefined level=notice: 1",
    "severity": 4,
    "type":
        {"eventType": "problem",
        "classification": "log anomaly",
        "condition": "0d33c050-101f-11ec-881b-acde48001122"},
    "resource":
        {"name": "qotd-rating",
        "type": "application",
        "application": "qotd-rating",
        "entity": "qotd-rating"},
    "expirySeconds": 900,
    "details":
        {"log_anomaly_detector":
          {"start_timestamp": 1629846420000,
          "end_timestamp": 1629846430000,
          "original_group_id": "0d33c050-101f-11ec-881b-acde48001122",
          "causality":
            {"service": ["qotd-rating"]},
            "detected_at": 1632252299670.0,
            "source_application_id": "qotd-rating",
            "log_anomaly_confidence": 0.9152,
            "log_anomaly_model": ["PCA"],
            "prediction_error":
              {"pca_prediction_error": 26.884115203263626,
              "embedding_prediction_error": 0.02826894412475822},
            "error_templates": [],
            "template_list": ["Heartbeat status: <*><*><*>", "Heartbeat status: <*><*><*><*><*>", "incident euid=<NUM> epid=<NUM> dsteuid=<NUM> dstepid=<NUM> <*>type=undefined subtype=undefined level=notice", "Heartbeat status: CPU=\"OK\" Memory=\"OK\"", "[<NUM>] Ratings request, id: <NUM>", "[<NUM>] That doesn't seem fair.", "[<NUM>] The monkey's dart hit the <NUM>", "Heartbeat status: <*><*><*><*><*><*>", "Heartbeat status: <*><*><*><*>", "Unknown_normal", "Unknown_error"],
            "count_vector": [0, 0, 21, 0, 5, 0, 5, 2, 0, 0, 0],
            "text_dict":
              {"template_ids": ["7eddc431", "b59d9df4", "7cd5f6b4"],
              "template_list": ["Heartbeat status: CPU=\"OK\" Memory=\"OK\"", "Heartbeat status: <*><*><*><*><*><*>", "incident euid=<NUM> epid=<NUM> dsteuid=<NUM> dstepid=<NUM> <*>type=undefined subtype=undefined level=notice"],
              "count_vector": [0, 2, 21],
              "expected_count_vector": [14.24, 14.91, 15.13],
              "expected_count_lower_bound": [5.34, 6.45, 10.46],
              "expected_count_upper_bound": [23.14, 23.36, 20.02]},
              "application_group_id": "1000",
              "application_id": "1000",
              "model_version": "v4",
              "severity_from_model": 0,
              "description": "Heartbeat status: <*><*><*><*><*><*>: 2\nincident euid=<NUM> epid=<NUM> dsteuid=<NUM> dstepid=<NUM> <*>type=undefined subtype=undefined level=notice: 21\n[<NUM>] Ratings request, id: <NUM>: 5\n[<NUM>] The monkey's dart hit the <NUM>: 5"}},
              "timestamp": 1632252299670.0}
  ]