Determining which algorithm generated a log anomaly

There are two log anomaly detection AI algorithms, each of which can run independently of the other: natural language and statistical baseline. If both algorithms are enabled, then any log anomalies discovered by both will be reconciled, so that only one alert is generated. Use the information in this topic to work out which log anomaly detection algorithms contributed to the detection of the anomaly.

About this task

There are two ways to check which log anomaly detection AI algorithms generated a given log anomaly:

  • By viewing the details of the alert in the Alert Viewer.
  • By monitoring the Kafka topic cp4waiops-cartridge.lifecycle.input.events kafkatopics, and parsing out the values in the log_anomaly_model field.

Viewing details in the Alert Viewer

  1. In the Alert Viewer, double click the log anomaly alert of interest. The Alert Details panel opens on the right, displaying the properties of the alert.
  2. In the Properties table, locate the Description field.
  3. Within the Description field, locate the log_anomaly_model key. The nested values within this key identify which log anomaly detection algorithm contributed to the detection of this anomaly. Possible nested values are listed in Possible values within the log_anomaly_model JSON key below.

Alert details
Figure. The Alert details panel

Monitoring the relevant Kafka topic

  1. Monitor the Kafka topic cp4waiops-cartridge.lifecycle.input.events kafkatopics. This topic provides a Kafka stream of alert instances.
  2. Look for a sender key where name attribute is "Log Anomaly" to identify those alert instances that were generated by the log anomaly detection AI algorithms.
  3. Within that section of the JSON, locate the log_anomaly_model key. The nested values within this key identify which log anomaly detection algorithm contributed to the detection of this anomaly. Possible nested values are listed in Possible values within the log_anomaly_model JSON key below.

Reference information

The following information is referenced in this task.

Possible values within the log_anomaly_model JSON key

Value Algorithm Meaning
PCA_Stats Natural language The time window in which this alert occurred is statstically different to the training data.
PCA_error_or_unknown_template Natural language There are error templates or unknown templates in this time window which the algorithm has identified as anomalous.
PCA_low_frequency_error_template Natural language There are error templates which occur with low frequency in this time window and the algorithm has identified this as anomalous.
RSM_entities Statistical baseline The algorithm has determined that there are significantly different error entities in this time window.
RSM_embeddings Statistical baseline The algorithm has found word embeddings for this time window that are distant from the embedding vectors for normal data.
RSM_websphere Statistical baseline This value is specific to WebSphere logs. The algorithm has found significantly different error message identifiers or log levels for this time window.