Multi-class classifier model for incidents

Edit online

Starting in 9.2, the multi-class classifier (MCC) model is an artificial intelligence model that provides suggestions for categorizing incidents based on historical data. The model analyzes incident descriptions to suggest appropriate incident types and categories.

The MCC model uses machine learning algorithms to identify patterns in your organization's incident data. By analyzing the text in incident summaries and details, the model learns to associate specific words, phrases, and contexts with particular incident types and categories.

Model training

The MCC model is trained on your organization's historical incident database. The training data must include a sufficient number of properly categorized incidents to serve as the source of truth for the model. The quality and quantity of training data directly affect the accuracy of the suggestions that the model provides.

During training, the model learns the relationships between incident descriptions and their assigned categories. The model identifies patterns such as:

Common terminology used for specific incident types
Keywords that indicate particular incident categories
Contextual information that helps distinguish between similar incident types
Patterns in how incidents are described across different locations or departments

Training process

To train the MCC model, an administrator must complete the following steps:

Configure training and inference filters to determine which incidents are used for training and which incidents display AI suggestions
Select a sufficient number of properly categorized historical incidents for training
Initiate the training process through the AI configuration application
Monitor training progress and review model accuracy scores

Training can take several hours depending on the volume of data. After training is complete, the model must be prepared for inferencing before suggestions become available to users.

Model inference

After the model is trained, it can provide suggestions for new incident records. When you enter text in the Summary or Details field of an incident, the model analyzes the text and generates suggestions for the Incident Type and Incident Category fields.

The model provides multiple suggestions that are ranked by confidence level. You can review the suggestions and select the one that best matches the incident. The model continues to learn from the selections that you make, which improves its accuracy over time.

Training and inference filters

The MCC model uses two types of filters to manage data:

Training filter: Determines which historical incidents are used to train the model. The training filter should include incidents with accurate categorizations that represent the types of incidents your organization typically handles.
Inference filter: Determines which incidents display AI suggestions to users. Only incidents that meet the inference filter criteria show the AI suggestion feature in the user interface.

Administrators can customize these filters to match organizational needs by modifying the query definitions in the object structure configuration.

Suggestion confidence

Each suggestion includes a confidence score that indicates how certain the model is about the recommendation. Higher confidence scores suggest that the model found strong patterns in the training data that matches the current incident description. Lower confidence scores indicate that the incident description might be ambiguous or that the model has insufficient training data for similar incidents.

Review all suggestions carefully, especially those suggestions with lower confidence scores, to help ensure that the selected category accurately reflects the incident.

Model accuracy and monitoring

After training, the model provides an accuracy score that measures how well it performs on the training data. The accuracy score represents the percentage of suggestions that the model considers reasonable based on the training data.

A higher accuracy score (closer to 1.0 or 100%) indicates better model performance. However, accuracy depends on the quality and diversity of training data. A high accuracy score with incomplete or non-diverse training data might not produce useful suggestions in practice.

Monitor model performance over time by reviewing:

User acceptance rates of AI suggestions
Frequency of manual overrides
Consistency of categorizations across similar incidents

Retrain the model when accuracy decreases or when organizational incident patterns change significantly.

Data requirements

To use the MCC model effectively, your organization must have sufficient historical incident data with the following characteristics:

Minimum data volume

At least 20 properly categorized incidents for each incident type and category combination. For optimal model performance, include 20-50 incidents per category.

Data quality

Use unique incidents. The training process filters out duplicate incidents.
Use diverse incidents that include a range of descriptions and address various problems.
Ensure incidents have accurate incident type and category assignments.
Verify that incident type and category values have accurate descriptions in the system.
Ensure incident summaries and details contain meaningful, descriptive information about the events.

Consistent categorization

Historical data must demonstrate consistent categorization practices. Inconsistent or inaccurate historical categorizations reduce model accuracy.

Regular updates

Update training data regularly to reflect current incident patterns and organizational changes. Retrain the model when significant changes occur in incident types or categorization practices.

Similarity detection

In addition to the MCC model, the similarity AI model can be applied to identify similar incident records. This feature helps you detect trends, identify recurring incidents, and recognize patterns in incident types, locations, or equipment involved. By analyzing similar incidents, you can take corrective action to prevent future occurrences.