Viewing alerts in ChatOps
The Alerts area in ChatOps provides details and traceable evidence about alerts. View details of potentially relevant alerts and log anomalies to help you better understand the incident, determine the cause for the alerts, and confirm a diagnosis. Alerts are grouped based on three algorithms: Scope-based, Temporal, and Topological.
- View basic Alerts information
- View detailed information on Alerts
- Investigating log anomalies
- Domain-specific log anomalies
View basic Alerts information
- Open ChatOps in Slack or Microsoft Teams.
- Select an incident from your incident ( proactive ) channel, then scroll down to the Alerts section of the Incident. This entity presents alerts that are associated with the incident, including their severity level. Click View in Alert Viewer in ChatOps to access the alerts in IBM Cloud Pak® for AIOps Platform UI.
View detailed information on Alerts
In the Alerts section of your incident, click View alerts. The View alerts modal opens.
- Use the two dropdown menus to filter alerts by type of alert, severity, status and rank.
- If necessary, scroll down to find all alerts associated with the incident.
- An alert listing can have a number and icon label that is attached to it.
These labels indicate probable cause ranking, severity of alert and if the alert is a trigger alert - indicated by a lightning icon. Note: These labels are present only if the alert is a trigger alert and/or if it is a probable cause of the incident - or both.
- You can hide any listed alerts for a specific incident entity by clicking Hide for that alert. Click Done when you are done hiding alerts. Note: To see alerts that you hid, select the entity, and click Unhide.
- For more information on an alert, click Show more on the View alerts modal to view the alert’s Status, Source, First occurrence, Last occurrence and Resource, Ranking and Severity.
Investigating log anomalies
Based on an analysis of your log data performed during AI model training, templates are built that model each type of log message identified. Live log data is analyzed with respect to these templates and any surprises are reported as log anomaly alerts. Here are some examples of log anomaly alerts:
- A log message was expected but is not present in the log data within a time range.
- A log message was not expected but is present in the log data within a time range.
- A log message occurred with a higher or lower frequency than expected within a time range.
Select Log anomalies from the dropdown menu in the View alerts modal to view the templates that are used to identify anomalous behavior in your log files.
To view the entire log of a log anomaly (the log for all of the templates), click Attach template logs. This action attaches the entire log to your Slack channel where you and your collaborators can view the log in its entirety.
This window also reports surprises associated with individual log anomaly templates. For each surprise, the following information is provided.
-
Log anomaly Template or label. If this log message has never been seen before, one of the following labels will appear:
Unknown_error: the log message has not been seen before and it contains error information.Unknown_normal: the log message has not been seen before but it does not contain error information.
-
Actual: The number of times this log message actually occurs in the log data within a time range.
-
Expected: The number of times this log message is expected to occur in the log data within a time range.
-
Preview log: Click this button to view the actual occurrences in your log. The Log preview window shows a maximum of 3000 characters.
Note: In Microsoft Teams, logs are saved to Microsoft Sharepoint and can be shared from there.
Example 1
| Template or label | Actual | Expected |
|---|---|---|
stdout F {“name”:“@pod75wd5abd11”,“__in”:<NUM>, “hostname”:“backend-456b7373c9efgh”,“pid”:<NUM>,“module”: “announceCycle/agentHostLookup”,“level”:<NUM>,“msg”:“Extremely unusual message for system |
0 | 3.12 |
In this example, three log messages matching the template were expected in a given timeframe, but there were no log messages present in the log data. You need to investigate why the expected log messages did not appear. Note Expected_count is shown as a float with two digits after the decimal point. This value is calculated based on the statistical analysis described in Workflow of natural language log anomaly detection and Workflow of statistical baseline log anomaly detection.
Example 2
| Template or label | Actual | Expected |
|---|---|---|
Unknown_error |
2 | 0 |
In this example, a log message that has never been seen before was present in the live log data, twice within the time range. This unexpected log message also contains error information so should be investigated with urgency.
Domain-specific log anomalies
If an alert is identified as a log anomaly from a specific domain, such as IBM MQ or WebSphere, you can scroll to the (View Alerts > Show more) modal to find more details about the anomaly. View Message Code, Explanation, Categories, Frequency of related message codes and Recommended actions, which can contain documentation links that might help resolve the alert.
For more information, see Domain-specific log anomaly alert details.