Events not being matched to deployed pattern

A pattern has been deployed but incoming events in the live events stream, which are expected to match the pattern, are not being grouped in the Event Viewer.

Draft comment: gb030961
Event Analytics Defect 70855

PROBLEM

A pattern has been deployed but incoming events in the live events stream, which are expected to match the pattern, are not being grouped in the Event Viewer.

CAUSE

There can be any number of different reasons why live events are not being matched. For example, one or more of the Netcool®/Impact services that is essential for processing events into groups might be switched off. Another cause might be unusual characters within event fields not being escaped correctly and therefore preventing a synthetic event from being inserted into the ObjectServer.

In order to more fully understand the troubleshooting process, it is important to first understand the following:
  • The process by which live events are matched to deployed groups and patterns.
  • Netcool/Impact processes and services involved in this process

Event matching process

There are two stages to the process of matching events to deployed groups and patterns:
1. Try to match live events to deployed groups
Live events arriving in the ObjectServer are first read by the Netcool/Impact ProcessRelatedEvents reader service to see if they match any deployed related event groups. The match is tested against deployed groups first as they have a stricter matching criteria compared to deployed patterns. This is because the match is done using the Event Identity field. After the ProcessRelatedEvents service has completed processing for a given live event, the IBMProcessingStage field in that event's alerts.status table is updated from 0 to 500.
2. For those live events that could not be matched to deployed groups, try to match them to deployed patterns
This stage is made up of two phases:
2.1 Identify the relevant event type fields for each live event
If any live events are not matched by the ProcessRelatedEvents reader service to a deployed group, then the IBMExtractedType field in the alerts.status table for each of those events is set by the ProcessRelatedEventTypes service. This is done by iterating through the additional and default event types, and applying the event type filter to the live events to determine which field to use for the Event Type value.
Note: The event types set in the Event Analytics configuration wizard at the time of the configuration run are the ones that are used. The order of additional types is important. For more information on the order in which the event types are processed, see Configuring event pattern processing.
2.2 Try to match these live events to active event patterns
Finally the ProcessRelatedEventPatterns reader service retrieves the events whose IBMExtractedType field has been set and attempts to match them to active patterns. The list of active patterns is stored in memory using the LoadRelatedEventPatterns service. After the ProcessRelatedEventPatterns reader service completes, it sets the IBMProcessingStage field in the alerts.status table for each processed event to 1001. If the event was matched then the ParentIdentifier field for that event is also set to the value of the Identifier field of the parent event.

Policies and services

The following Netcool/Impact policies and services are involved in the process of matching events to deployed groups and patterns:
Policy Service Descriptions Phase of event matching process

RE_PROCESSEVENT

ProcessRelatedEvents

Inspects live events arriving in the ObjectServer to see if they match any deployed related event groups.

1

PG_PROCESSTYPES

ProcessRelatedEventTypes

Sets the IBMExtractedType field for any events that were not matched to a related event group.

2.1

PG_PROCESSPATTERNS

ProcessRelatedEventPatterns

Retrieve events whose IBMExtractedType field has been set and tries to match them to active event patterns. Triggers Java™ code that does some of the matching of events to patterns.

2.2

PG_PATTERN_LOADINTOMEMORY

LoadRelatedEventPatterns

Stores active event patterns in memory.

All

RESOLUTION

The troubleshooting and resolution activities to resolve this issue can be summarized as follows. These activities are listed in increasing order of complexity.
  • Check that relevant policies and services are running
  • Check the values of relevant live event fields
  • Analyze policy logs

Check policies and services

Check that the following Netcool/Impact services are running:
  • ProcessRelatedEvents
  • ProcessRelatedEventTypes
  • ProcessRelatedEventPatterns
  • LoadRelatedEventPatterns

Check values of event fields

Check the values of the event fields that are modified by the event matching process.
Event field Description Possible values

IBMExtractedType

Indicates whether event types have been extracted for events that are to be matched to active patterns.
Note: This event type is only set for those events that could not be matched to deployed groups.
0: No event types were extracted for this event. This could mean one of the following:
  • The event was already matched to a deployed group, and no pattern matching will be done
  • The system is trying to match this event to a pattern but no event type has been assigned to this event.

1: Event types were extracted for this event. This means that the system is trying to match this event to a pattern and event type(s) have been assigned to this event.

IBMProcessingStage

Specifies the overall processing stage.

0: Processing has not started for this event.

500: The event was matched to a deployed group.

1001: The event was matched to an active event pattern.

ParentIdentifier

Stores the Identifier of the parent event. If there is a value in this field, then it means that this event has been grouped either my matching it to a deployed group, or to an event pattern, where that pattern uses a most important event for the parent event, and not a synthetic event.

Analyze policy logs

Turn on verbose logging and review the log output.
Turn on verbose logging
Turn on verbose logging for the PG_PROCESSPATTERNS policy and associated Java code, by performing the following steps:
  1. In the PG_PROCESSPATTERNS policy, change the verboseLogging variable from false to true.
  2. In the Services tab of the Netcool/Impact GUI, do the following:
    • Set the log level for the PG_PROCESSPATTERNS policy logger to 2.
    • Select Append Policy Name to Log File Name. This ensures that the names of all log files generated by the PG_PROCESSPATTERNS policy include the name of the PG_PROCESSPATTERNS policy.
    • Select Append Thread Name to Log File Name. Each event is processed using a different thread. Selecting this option ensures that a separate log file is generated for each thread. This enables you to review the processing of each event separately.
    For more information on the Policy logger service, see Netcool/Impact documentation: Policy logger service.
  3. Edit the etc/impactserver.log4j.properties file and set log4j to TRACE for the following classes:
    log4j.appender.NETCOOL.threshold=TRACE
    log4j.category.com.micromuse.response.common.parser.NOIProcessRelatedEventPatternsRequest=TRACE
Review log output
You must review the output of both the PG_PROCESSPATTERNS policy log and the log output for Java code triggered by this policy. Log output for the Java code can be reviewed by inspecting the impactserver.log. The following table provides guidance to support the review of the log output.
Log message example 1
15 Jul 2021 07:48:03,625: [PG_PROCESSPATTERNS] [MessageProcessor-Dog#9]Parser log: 
############### No matching patterns have been found for this event! 111 ###############
Meaning
No matching patterns by event type were found for any events with an Identifier of 111.
Next steps
  • Check that the LoadRelatedEventPatterns service is running.
  • Check if the event type field in the live event matches the value for the event type for deployed patterns.
Log message example 2
FindMatchingOpenInstance: SelectPG_Patterns_Size: 4
Found Matching Pattern Results PATTERNID = 1590
Found Matching Pattern Results PATTERNID = 1644
Found Matching Pattern Results PATTERNID = 1665
Found Matching Pattern Results PATTERNID = 1666
Meaning
The first message shows how many patterns match an event. The second message lists the identifiers of the patterns that match that event.
Next steps
This message indicates that patterns are being matched as expected, so there are no next troubleshooting steps for this message.
Log message example 3
24 Jan 2020 10:56:44,449: [PG_PROCESSPATTERNS][MessageProcessor-Dog#21]Parser log: Started Creating Synthetic Parent Event....
24 Jan 2020 10:56:44,452: [PG_PROCESSPATTERNS][MessageProcessor-Dog#21]Parser log: Started to Group Event under Synthetic Event....
24 Jan 2020 10:56:44,453: [PG_PROCESSPATTERNS][MessageProcessor-Dog#21]Parser log: Finished to Group Event under Synthetic Parent Event.....
24 Jan 2020 10:56:44,453: [PG_PROCESSPATTERNS][MessageProcessor-Dog#21]Parser log: Get all Stored Events for the pattern...
24 Jan 2020 10:56:44,462: [PG_PROCESSPATTERNS][MessageProcessor-Dog#21]Parser log: Number of stored events waiting for this actionable event: 0
24 Jan 2020 10:56:44,462: [PG_PROCESSPATTERNS][MessageProcessor-Dog#21]Parser log: Finished Creating Synthetic Parent Event....
Meaning
Indicates that a new synthetic event has been created and child events are being grouped under it.
Next steps
This message indicates that patterns are being matched as expected, so there are no next troubleshooting steps for this message.