System z Monitoring and Application Management
Home > Best Practices > Forwarding alerts to third-party event managers in IBM Tivoli Monitoring and IBM Tivoli OMEGAMON XE products
Forwarding alerts to third-party event managers in IBM Tivoli Monitoring and IBM Tivoli OMEGAMON XE products
This article describes the use of policy automation as a mechanism to forward alerts, as well as closing events, to a third-party event manager.
Consolidated event management continues to be a top priority IT initiative for today's enterprise. Many organizations invest in event management products to help them correlate data from multiple collection sources. IBM® and other vendors offer this type of software solution. Event management products typically harvest data made available by performance monitoring tools. When the event management tooling and the performance monitoring products are part of a single vendor solution, propagation of alerts to the highest level management console is made simple. This is the case when IBM Tivoli® Monitoring and IBM Tivoli OMEGAMON® XE products collect performance data, and Tivoli Business Systems Manager and Omnibus are the recipients of the alert information. Alert propagation is easily achieved by selecting a check box on the monitoring situation definition. Forwarding alerts from OMEGAMON XE products to other third party event managers requires a somewhat more sophisticated approach.
Creating alerts to send
Threshold-based alerts in Tivoli OMEGAMON XE products are defined by using situations. A situation can evaluate any of the performance metrics collected by the OMEGAMON XE software. When the evaluation is true, an alert is posted to the Tivoli Enterprise Portal navigation tree. The alerts indicate severity through the assignment of color to augment an icon. Situations evaluate on user-specified intervals, which are the intervals determined by the type of data being evaluated.
Situations can invoke an associated executable file, using a feature known as a Take Action command. These commands are delivered to the target platform, as if the user were entering commands into a console. The action passes parameters for execution, with symbolic variables being resolved as part of the results set that the situation evaluated. These actions can be used to send alerts to an event manager if the event manager has supplied a command-line interface that can be invoked to convert alerts into a suitable format. One drawback of this approach is that there is no means to automatically reset the alert that was forwarded to the third-party event manager. Alerts posted to the Tivoli Enterprise Portal navigator will automatically be reset when the alerting situation no longer evaluates to true. However, there is no capability within situations to close alerts on a third-party event console. Events that have been forwarded to Omnibus or the Tivoli Business Systems Manager products can be automatically cleared, using a simple check box on the situation definition.
Using policies to forward and reset alerts
An advanced form of automation, known as policy automation, is the recommended mechanism for communication to third-party event managers. Policies can execute multiple actions, waiting for any number of conditions to exist prior to the execution of the next action. This feature enables resetting events to be sent to the third-party event managers when OMEGAMON XE product situations are no longer true.
Policies are defined using the Workflow Editor. The Workflow Editor uses a drag and drop interface, placing evaluations and actions on a blank window. These objects are wired together using the Connect 2 activities tool. The result is a thread of execution containing a situation that indicates a problem condition, a series of actions to be performed while the situation is true, and a subsequent action to be taken when the situation is no longer true.
Building the policy
The first step in any policy is to wait on a situation. The nature of the situation is not germane to this discussion. However, in this example, a situation is used that monitors the IBM MQSeries® product. This situation example emphasizes the ability to isolate events when multiple managed systems exist on a single endpoint. It is somewhat common to find multiple queue managers executing under a single system image, and accordingly, the Tivoli Monitoring event forwarding mechanism must be able to accommodate this architecture.
The situation monitors multiple application queues, alerting when the current depth of messages on any of the specified queues exceeds a prescribed threshold. A distinct alert is created for each queue that exceeds the assigned threshold. This feature, known as a display item, is honored by policy automation. Thus, a separate thread of execution of the policy exists for each queue currently experiencing a high depth condition. If the display item is not coded on the situation, a single alert is used to represent all queues with high depth.
Correlation mode
Correlation mode establishes policy behavior when evaluating results returned from situations. The possible correlation modes are:
- None - consider the results of situations embedded within the policy, regardless of what managed system the situations executed on.
- Hostname - consider the results of situations embedded in the policy, in conjunction with other embedded situations when the hostnames of the systems on which the situations execute are identical.
- Host address - consider the results of situations embedded in the policy, in conjunction with other embedded situations when the host address (IP address) of the system on which the situations execute are identical.
- Managed system - consider the results of situations embedded in the policy, in conjunction with other embedded situations when the managed system name of the systems on which the situations execute are identical.
Correlation mode is a critical setting for the alert forwarding policy. This setting establishes the granularity required to match raised and reset alert conditions. For example, an alert raised against queue manager A cannot be offset by a situation evaluating false against queue manager B. The correct setting to achieve this is to correlate by managed system.
Correlate by managed system is available in the Tivoli Enterprise Portal at the following versions and maintenance levels:
- IBM Tivoli Monitoring 610 FP7 IF4
- IBM Tivoli Monitoring 620 FP3
- IBM Tivoli Monitoring 621 GA IF4
These Tivoli Monitoring versions and maintenance levels result in the addition of the Managed System choice to the following Correlate by drop-down menu.
Accessing the workflow editor
The workflow editor is accessed from the Create new icon + + on the action bar of the Tivoli Enterprise Portal (portal). When you are in the workflow editor, create a new policy using the Create new icon. When created, the policy is displayed in a list of defined policies. A column approximately in the middle of the report is labeled Correlate by, and is where you select the appropriate mode. As stated previously, you need this policy to correlate situation results at the most granular level, which is the Correlate by Managed System mode.
The next step is to embed situations within the policy, which is done by clicking the workflow components palette on the left side of the workspace. With the general activities selected, find the Wait until a situation is True object. Then position the cursor on the window and click again, dropping the object in place. At this point you assign the MQSeries monitoring situation, selecting it from the available situations list.
The next step is to add the action to the policy. Select the Take action or Write message object, and then click in the appropriate location on the window.
Here you specify the action required to send the alert to the third-party event manager. This example shows the BMC BEM product. The specifics of coding this mechanism to work with other vendor products are left to the reader.
On the Action Settings panel, select System Command as an action type. Then provide the specific action to be performed in the System Command input area. In this example, a script called tivolimsender.bat is run. Input to the script is provided through symbolic parameters and constants describing the nature of the alert.
Full Text System Command:
F:\IBM\ITM\scripts\tivolimsender.bat "MAJOR" "&{Queue_Statistics.Host_Name}" "MQ queue &{Queue_Statistics.Queue_Name} on &{Queue_Statistics.MQ_Manager_Name} is alerting with &{Queue_Statistics.Current_Depth} messages in the queue. Alerting threshold is [10] messages. &{Queue_Statistics.Queue_Description_U}"
The action must execute on a system containing the script. This specification is made by clicking on the More Options button, and assigning a managed system for execution.
These two objects can now be wired together. Select the Connect 2 activities tool from the general activities window, and then click on the embedded situation object. Wire a connection between the situation object and the action object, specifying a condition of Situation is True. You are now configured to send alerts over to BEM.
Resetting the alert
Now you want to instruct the policy to reset the alert on BEM when the condition is no longer True. On the Workflow Components window, select the Extensions tab. Here you find a Wait until situation is False object. Select the situation used previously and assign it to the Wait until situation False object. Next you add the action required to reset the alert on BEM. As before, a Take Action object from the General Activities window is used to initiate the command. Specify System Command and enter the text of the script to be executed. Here is the example:
Full Text Clear Event:
F:\IBM\ITM\scripts\tivolimsender.bat "OK" "&{Queue_Statistics.Host_Name}" "MQ queue &{Queue_Statistics.Queue_Name} on &{Queue_Statistics.MQ_Manager_Name} is no longer alerting on queue depth.
The final step in defining the policy is to wire the objects together, except this time you create a connection between the Wait until situation is False object and the Take Action object. This connection is the Situation is false type.
Policy distribution
Situations are distributed to managed systems, either individually or through managed system lists. This distribution controls where the situation will execute. Policies are also distributed to managed systems. However, in the case of policies, the term represents a different behavior. Policies will only consider for input, results of situations that have executed on a managed system to which the policy has been distributed.
Vendor specific executables
The policy defined previously runs the tivolimsender.bat script. This script leverages a vendor-provided utility to pass data to BEM. See the following contents of the script, provided here as a sample:
set MESSAGE_SEVERITY=%1
set TIVOLI_ALERT_HOST=%2
set TIVOLI_MESSAGE=%3
set TIVOLI_INST=sys095
set BEM_IP=10.1.3.122
set BEM_PORT=1828
set BEM_KEY=yourBEMkey
set EVENT_TYPE=ITM_MQ
set WORKING_DIR=%dir1%
echo %TIVOLI_MESSAGE%
F:\IBM\ITM\scripts\msend -q -n @%BEM_IP%:%BEM_PORT%#%BEM_KEY% -a EVENT -o %EVENT_TYPE% -r %MESSAGE_SEVERITY% -b "mc_host=%TIVOLI_ALERT_HOST%" -m %TIVOLI_MESSAGE%
Summary
In this article, OMEGAMON XE alerts were forwarded to a third-party event manager. Policy automation was used to initiate a custom script containing a vendor-supplied command-line utility. Key data was supplied to the script as parameters, obtained from the results returned from an OMEGAMON XE situation. The script was invoked first when the situation turned True, and a second time when the situation turned False. Alerts opened and closed in this manner facilitate synchronization between the Tivoli Enterprise Portal and the third-party event console.
About the author
Wayne Bucek is a Certified Consulting IT Specialist for IBM's Tivoli System z products. He has 20 years experience in the systems management arena, focused on helping organizations monitor their mission critical applications. In his current role, he assists customers with the implementation and usage of multiple OMEGAMON / Tivoli Monitoring products. Prior to joining IBM, he had ten years experience in the areas of systems programming and performance management.
Copyright
®
© Copyright IBM Corporation 2009
IBM United States of America
Produced in the United States of America
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of In-ternational Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other compa-nies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information
" at www.ibm.com/legal/copytrade.shtml.
Other company, product and service names may be trademarks or service marks of others.
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PAPER "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.
Information in this paper as to the availability of products (including portlets) was believed accurate as of the time of publication. IBM cannot guarantee that identified products (including portlets) will continue to be made available by their suppliers.
This information could include technical inaccuracies or typographical errors. Changes may be made periodically to the information herein; these changes may be incorporated in subsequent versions of the paper. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this paper at any time without notice.
Any references in this document to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.
IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to:
IBM Director of Licensing
IBM Corporation
4205 South Miami Boulevard
Research Triangle Park, NC 27709 U.S.A.
