Metric-based machine learning overview

The metric-based machine learning system in IBM Z® Anomaly Analytics ensemble detects anomalies in the metric data from z/OS® System Management Facilities (SMF) record types and in the log data from IBM® IMS log record types.

Data flow

The following steps describe the data flow among the components of the metric-based machine learning system. The step numbers correspond to the numbers that are shown in Figure 1.
  1. Historical or streaming SMF and IMS data is collected by the Z Common Data Provider and forwarded to Apache Kafka.
  2. The metric-based machine learning system subscribes to the Apache Kafka topic and processes the SMF and IMS data.
  3. If a metric data model is available, the summarized data is compared to the model to identify abnormal behavior.
  4. The score results are written to the Apache Kafka message broker and are stored in a NoSQL long-term data store.
  5. The rules engine polls Apache Kafka for anomalous events that exceed a configured threshold. Highly anomalous events are forwarded to the Apache Kafka Ensemble-Event topic.
  6. Ensemble subscribes to the Ensemble-Event topic, and it groups these anomalous events, together with anomalous events from the log-based machine learning system, into event groups by resource (system or subsystem). Ensemble calculates the severity and confidence score of each event group. The results are stored in a NoSQL long-term data store, and they are forwarded to the Apache Kafka Ensemble-EventGroupNotification topic.
  7. When a user accesses the ensemble GUI, the score results are retrieved from the NoSQL data store and rendered in the GUI.
Figure 1. Metric-based machine learning overview
The illustration shows the flow of data among the primary components, as described in the text.