An Explanation of the ITM triggers for OMNIbus

Background

OMNIbus customers normally rely on the out-of-box automations (generic triggers) for the automatic de-duplication and clearing of events flowing into the OMNIbus object servers. The generic de-dup trigger is designed to implicitly correlate update events with insert events using the alerts.status table’s ‘Identifier’ primary key value as the correlation. The result of de-duplication is that the user always sees only the most recent status of an event; the Tally Count ensures that the user doesn’t lose knowledge of how many times the event has occurred. De-duplication is always the replacement of a single event by another event with the same Identifier.

The generic clear trigger is designed to correlate Resolution events with one or more Problem events using the alerts.status table’s ‘AlertKey’, ‘AlertGroup’, ‘Node’, and ‘Manager’ column values and a Type of ‘2’ as the correlation. The result of clear trigger processing is that event(s) are removed from the Object Server when they are resolved, ensuring that the user is only focusing on active problems and obviating the need to open tickets for events that are quickly resolved. Clearing is less performant than de-duplication because there may be multiple events involved and the processing is more complex.

The default triggers were designed to work with events and event sources that behave in a certain way and meet the content requirements of the Tivoli Netcool/OMNIbus Integration Best Practices. For architectural reasons, ITM’s events either cannot or do not meet some of those behavioral and content requirements, and this is what drives the need for custom triggers. Below I describe how the ITM custom triggers provide the same behavior for ITM events in OMNIbus that the generic triggers provide for other event sources.

ITM de-duplication trigger

The generic de-duplication trigger is designed to process  event state changes that occur only as a result of a change at the monitored resource or from an action taken at the OMNIbus console. However, in ITM there are several additional ways ITM event state can be changed that must be accounted for. ITM has a feature called the Situation Event Console (SEC) that shows the list of currently active situations. The SEC allows ITM users to Acknowledge (ITMStatus=A), Re-surface (ITMStatus=E), and in the case of Pure situations, Close events (ITMStatus=N).  ITM also allows administrators to specify an UNTIL clause for a Sampled or Pure situation that will result in a Close event after some specified period of time. ITM also has a SOAP interface that allows external programs to change event state using SOAP messages.  Finally, ITM has interfaces that allow administrators to Stop (ITMStatus=P) and Delete (ITMStatus=D) situations. All of these affect a situation’s state and hence cause an event to be forwarded for the situation.

Another source of event state changes unique to ITM is “loopback” events. ITM can be configured to send events to multiple event receivers. When an ITM event’s state is changed in one OMNIbus console, ITM must also notify the other OMNIbus servers that the event’s state has changed to keep its data consistent.  It does this by issuing an event (termed ‘loopback’ event) with the updated fields to all of its configured listeners. When the event reaches the OMNIbus server that happened to originate the change, special processing must be in place to avoid incrementing the Tally or undoing a Clear operation.

The generic de-duplication trigger does not have logic for processing Acknowledgements that are not initiated from the OMNIbus console, nor does it have logic for ignoring an event state change from ITM for which its object server was the originator. Instead, this logic is provided by the custom ITM de-duplication trigger.

There’s one other feature  the custom de-duplication trigger provides: clearing of an open event when an update event with a matching Identifier has ITMStatus=N. Per OMNIbus best practices ((http://publib.boulder.ibm.com/infocenter/tivihelp/v8r1/topic/com.ibm.netcool_OMNIbus.doc_7.3.1/omnibus/wip/admin/reference/omn_adm_per_bestpracticestriggers.html), ITM’s de-dup trigger clears Problem Events (ITMStatus=Y) with Resolution Events (ITMStatus=N) having the same Identifier because it’s more performant to do that as a de-dup operation than a clear operation.  There is one exception to this behavior. If a HUB TEMS is recycled, meaning there was a gap in situation state for an unknown amount of time, the TEMS baselines situation status after it is back up since there is no longer a basis for correlating the current state of a situation with a previous state. Then to ensure consistency, ITM effectively baselines its state in OMNIbus by sending an event with ITMStatus=N and AlertKey=**.  The custom ITM clear trigger recognizes that a reset is occurring and contains logic to close all open ITM events from the HUB that recycled. Note that the reset processing does not clear open Pure events because we recognize there is no opportunity to re-sample their status.

The de-duplication trigger provided by the IBM Tivoli Monitoring event synchronization component and the generic  de-duplication triggers provided with OMNIbus are in conflict with each other. The generic de-duplication triggers modify the Severity and Summary attributes that results in those attributes having incorrect values for IBM Tivoli Monitoring events for both unidirectional and bidirectional event synchronization. For information on how to prevent the generic de-duplication trigger from processing ITM events, follow this link: http://pic.dhe.ibm.com/infocenter/tivihelp/v15r1/topic/com.ibm.itm.doc_6.2.3fp1/itm623FP1_install617.htm?path=3_0_3_0_9_1_2_4_4#chng_dedup_trig.

ITM clear trigger

The generic clear trigger is designed to process alerts that are either Problems or Resolutions, are distinguished by their Type value, and whose Resolution types have a severity of Indeterminate. ITM does support the notion of Problem events and Resolution events. Problem events are Sampled (ITMSitType=S) or Pure  (ITMSitType=P) situations with ITMStatus=Y, and Resolution events are Sampled  or Pure situations with ITMStatus=N. The severity of an ITM alert is independent of whether it is a Problem or a Resolution. The resolution event has the same severity as the original problem.

However, ITM situations also support additional states as explained earlier.  Specifically, ITM has interfaces that allow administrators to Stop (ITMStatus=P) and Delete (ITMStatus=D) situations, both of which affect a situation’s state and hence cause an event to be published.  

The generic clear trigger does not have logic to close events for situations that have been stopped or deleted. Instead, this is provided by the custom ITM clear trigger. The trigger is designed such  that a Stop event event has  the effect of clearing any problem (Type 20) events with the same ITMThrunode (RTEMS),  ITMHostname (Hub TEMS hostname), and AlertKey (situation name) values. Also, a Delete event should have the effect of clearing any problem (Type = 20) events with the same ITMHostname (Hub TEMS hostname) and AlertKey (situation name) values. To enforce this behavior, the ITM probe rules set events with ITMStatus=D or ITMStatus=P to a Type of 21 (Resolution).

Conclusion

Hopefully, this article has helped you to better understand the motivation and design of the ITM de-duplication and clear triggers for OMNIbus. For information on ITM and OMNIbus integration, follow this link: http://pic.dhe.ibm.com/infocenter/tivihelp/v15r1/topic/com.ibm.itm.doc_6.2.3fp1/itm623FP1_install601.htm?path=3_0_3_0_9_1#omnibus_install .