Netcool/Impact Missing Events

Body

Some events are apparently not being processed by Impact

or seem to be overlooked by the EventReader

 

Over many years of supporting the Netcool/Impact product there are several topics that occur repeatedly. This blog is intended to enable you to undertake initial investigations into the potential causes of this particular topic.

 

In the majority of cases raised under this very general title, Impact is not actually missing events. However, you need to determine if this is actually the case and this blog is seeking to assist in this determination. This will enable you to perform some initial diagnostics and connect you with other resources that provide further details on specific subjects.

 

EventReader Logging.

 

One of the first steps is to ensure the various components are reporting sufficient details to enable you to investigate such behaviour. Firstly, ensure that the EventReader is enabled to write to file - for example:

 

 

This will produce the output of the format:

 

[DateTimeStamp]: Query: select top 1000 * from alerts.status where StateChange >= 1234567890 AND Class != 10500 AND ([Filter]) order by StateChange ;

 

Where [DateTimeStamp] is the date and time the log message was recorded in the format:

 

day month year hours:minutes:seconds,milliseconds

 

and [Filter] is the conglomerate of all the active Filter conditions defined in the EventReader. The "select top 1000" limitation is designed to prevent the EventReader from overloading the EventQueue in any one poll. The "Class != 10500" condition prevents the EventReader from acquiring by default the Status events created by Impact SelfMonitoring - for further information please visit the TechNote "EventReader filters out Class 10500 event by default".

 

The "StateChange >= 1234567890" condition is the tool used in combination with the Filters to ensure only updated events are captured enabled by ticking the "Actions: [ ] Get updated events" option - for further information please visit the TechNote "OMNIbusEventReader use of Serial or StateChange".

 

For further information on the data output to this log file please visit the TechNote "Interpreting the EventReader log file messages".

 

Once we are able to see the SQL Impact is using to capture the target events we can then test the exact query within an nco_sql session or via the OMNIbus Configuration Client SQL interface. If the SQL is valid and does capture the target events we then need to confirm that the EventReader does actually capture the events.

 

EventReader Debug.

 

Setting up EventReader debug enables one to confirm if the Service does indeed capture expected target events by logging the Serial and StateChange value of every event it captures.

 

For further information please visit the TechNote "Enabling EventReader debug".

 

One can then repeat the test and search for the Serial value of the expected target events to confirm if the EventReader does capture the events. If the EventReader does indeed capture the events we then need to confirm that the target Policies do actually process the events.

 

EventProcessor Debug.

 

This requires an additional property in the ImpactServer properties file:

 

$NCHOME/impact/etc/[ImpactServerName]_server.props

 

to increase the amount of information provided in the EventProcessor log file:

 

impact.server.eventprocessing.debug=true

 

For further information please visit the TechNote "Enabling EventProcesser debug".

 

Structured Logging.

 

Using two custom variables, such as ID and WP, is useful for setting locater values to be reported with each Log() statement - for example:

 

ID="ID_"+ @Serial +"_"+ @StateChange +": ";

WP="Start";

Log(1,ID+WP);

 

ID - a unique record of the event currently undergoing Policy processing and, for our purposes here, Serial and StateChange are most appropriate to tie up the PolicyLogger output with that of the EventReader log file in debug.

 

WP – WayPoint where one names the section of Policy code pertinent to an action or goal of a series of actions, which also doubles up as a micro-comment to keep actual lines of comment to a minimum.

Combined with the PolicyLogger log level capability; these give us the ability to finely track the progress and path of an event through the Policy.

 

Using the combined Serial + StateChange prefix values one can repeat the test and search for the Serial value of the expected target events to confirm if they are processed by the Policy or Policies and confirm if they follow the expected path and acquire the expected values and responses in doing so. If the Policy does indeed process the events as expected we then need to confirm that the Policy does then output the expected SQL to update the target events in the ObjectServer.

 

For further information please visit the TechNote "Structured Logging in Policies". This is also expanded upon in a YouTube presentation "Netcool Impact Structured Logging".

 

 

PolicyLogger SQL logging.

 

SQL logging in the PolicyLogger output will enable one to view the actual statements being produced by the Policies and test the exact statements within and nco_sql session or via the OMNIbus Configuration Client SQL interface.

 

For further information please visit the TechNote "PolicyLogger SQL logging".

 

If this issue proves to be interference from reprocessing of the event we can prevent this through the use of flag field values and other measures detailed below.

 

Avoiding unwanted event reprocessing.

 

The main method to achieve this is to use a flag field to signify when an event is suitable for processing and has been processed. An integer field is most suitable for this task with the default value (zero) being used to indicate that an event is not suitable for processing. It then requires an active decision to flag an event for processing - be that in Probe Rules files or ObjectServer Triggers or Tools or by some other external action.

 

For further information please visit the TechNote "Avoiding unwanted event reprocessing".

 

Event Flagging.

 

The use of a field entitled ImpactFlag has been so promoted over the years that it is now becoming included in developments of the product. In later versions this is a default field in Impact and the ObjectServer and is used by the OMNIbusEventReader to identify events that have been processed.

 

For further information please visit the TechNote "Event Flagging". This is also expanded upon in a YouTube presentation "Event Flagging".

 

 

The subject of this blog is also discussed in the YouTube presentations "Netcool Impact Missing Events Part 1 of 2" and "Netcool Impact Missing Events Part 2 of 2".

 

 

 

 

 

 

Subscribe and follow us for all the latest information directly on your social feeds:

 

 

  

Check out all our other posts and updates:
Academy Blogs:	https://goo.gl/eZjStB
Academy Videos:	https://goo.gl/BMc9hg
Academy Google+:	https://goo.gl/ukxTcP
Academy Twitter :	https://goo.gl/GsVecH