![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
The IBM Tivoli Netcool/OMNIbus Mttrapd probe is the IBM Tivoli Netcool/OMNIbus Simple Network Management Protocol (SNMP) trap listener. The probe listens on the network for SNMP traps from various devices/systems/applications (herein referred to as a "device"), converting them into events based upon a rule-set defined in the probe's rules file and sends them to an IBM Tivoli Netcool/OMNIbus instance, known as the ObjectServer. The rules file is created based upon the device's Message Information Base (MIB) definition, usually available from the device supplier as a text file. The rules file can be processed by the Mttrapd probe either as a standalone file, or as part of the IBM Tivoli Netcool/OMNIbus Netcool Knowledge Library (KcKL).
The NcKL is a set of rules files that is released by IBM to enhance the integrity between third party devices and the Mttrapd probe. It contains predefined rules files for hundreds of popular devices that are created by subject matter experts with an in-depth knowledge of the inner-workings of the particular device. Moreover, the rules files are created with knowledge of the alarms/events the device can raise, and their meaning. For this reason, IBM Tivoli Netcool administrators should use the NcKL whenever possible in order to ensure the correct processing of traps.
There exists, however, a large number of devices for which NcKL has no rules file. In this case, the IBM provided MIB2Rules utility can be used to convert any given MIB file into an NcKL supported rules file for quicker deployment. The MIB2Rules utility has grown beyond its original purpose of MIB to rules conversion, making it seem difficult to use, when in fact it's very simple.
Notably, SNMP has no default concept of trap severity, or problem/resolution correlation. It can define functionality available to the SNMP manager (in this case the Mtrapd probe), but cannot describe the purpose of the device or its components. For these and other reasons, it is virtually impossible to create a tool that can automate the creation of a rules file that will be sufficient, without further modification.
The remainder of this article will outline the tasks required to convert a MIB file into a rules file that can leverage the powerful correlation capabilities of IBM Tivoli Netcool/OMNIbus. The article will make use of the MIB defined in the SYMTRAP-MIB.mib file, located in the downloads section. Note: This MIB file is not guaranteed to be a correct version of the MIB, as it is simply provided as an example. Please contact the MIB owner for the most up-to-date version of the MIB.
This article assumes that the KcKL is already installed and configured. It is beyond the scope of the article to include such information. For more information on the KcKL, refer to the information centre.
The first step in the process is to determine and locate the MIB file containing the definition of the traps. Traps for which there exists no rules will manifest as events in the ObjectServer with a Summary value of: Unknown Specific Trap Number (specific-trap) Received for Enterprise enterprise, where specific-trap is the trap number and enterprise is the enterprise object identifier allocated to the device manufacturer.
Using the MIB file supplied as an example, a trap for "Symbios SCSI Agent is down" would appear in the ObjectServer Summary field as follows:
Unknown Specific Trap Number (102) Received for Enterprise .1.3.6.1.4.1.2.1123 |
Obviously, this Summary value does not provide information to determine how to handle the event. Therefore, the MIB file must be located and converted into a rules file.
The location of the rules file will depend on the supplier of the device generating the trap. In most cases, the supplier provides the MIB in a digital format, either on-line or in product media. Moreover, there are likely to be references to the MIB in appropriate product documentation.
Note that most MIB files will have a number of dependant MIBs, e.g., RFC1212.mib. Most of the standard dependent MIB files are supplied with the MIB2Rules utility. If the MIB is not supplied, it's likely the MIB can be found on the internet.
For illustration purposes, this articles supplies the example MIB defined in the SYMTRAP-MIB.mib file, and can be found in the downloads section.
Once the MIB has been located, it can now be converted into a rules file. The following section provides further information on the conversion process.
Once the correct MIB file(s) has been identified and located, it can now be converted into a rules file using the MIB2Rule utility. The conversion process involves two simple steps:
- Import the MIB into MIB2Rules
- Export the Rules from MIB2Rules
Each step is outlined in more detail below.
Perform the following steps to import the MIB(s) into the MIB2Rules utility:
- Copy the MIB (e.g., SYMTRAP-MIB.mib) and its non-base dependencies into the $MIB2Rules_Dir/mibs/downloads directory.
- Start MIB2Rules using the operating system appropriate method
- Click the Import button, or select File - Import... from the menu
- In the Import window, locate and select the $MIB2Rules_Dir/mibs/downloads directory
- (Optional) If the MIB files are located in multi-level directory structure, ensure you select the Traverse Subdirectories option
- Click the Import button
After following the steps above, the Status window will appear as MIB2Rules processes each of the MIB files located in the import directory. Once it's complete, assuming no errors, you will receive a prompt to Allow MIB Module Upload?; either option (Yes/No) is suitable. Notification of errors will be shown in the Status window. The errors must be resolved before the MIB can be exported. Typically, errors will be related to dependent MIBs not being available.
Upon import completion, the Status window will display the modules that it has processed. Review the details and click OK when complete. Figure 1 below shows the results of importing the SYMTRAP-MIB file, supplied with this article.
Figure 1. Importing SYMTRAP-MIB
MIB2Rules will now display the imported MIB in the main MIB2Rules window using a tree control. The trap(s) identified in the previous section (e.g., those under .1.3.6.1.4.1.2.1123) can be located by filtering the display to Traps/Notifications by selecting View - Traps/Notifications. Locate the appropriate trap(s) by traversing the tree control, or by entering an appropriate search term (e.g., "symbios") in the Search field and clicking Find. If the trap is listed, the MIB has successfully been imported and is ready to be exported as a rules file.
Figure 2 below shows the imported MIB in the main window.
Figure 2. Imported SYMTRAP-MIB
Perform the following steps to create rules files for the imported MIB(s):
- Locate the uppermost OID (e.g., .1.3.6.1.4.1.2.1123.3.1.2.2.0).
- Click the Export button.
- Select the appropriate directory to export the rules file. For example, when exporting to the KcKL, select $MIB2Rules_Dir/export/rules/NCKL. Note that the name of the export directory is not relevant to the intended usage of the rules file.
- Select the File type: of Netcool Knowledge Library.
- The Scope option can be ignored, as only traps will be exported for a KcKL file type.
- Click the Export button.
The rules file(s) will be created in the selected directory. If you are using the example mib file provided, you will likely find 9 files in the export directory; the most relevant being those beginning with symbios- (Total of three) - the other files can be ignored. The files that result from an export of the example MIB are provided in the downloads section.
The next section will discuss how to modify the newly created files to be suitable for an existing KcKL installation.
This section describes how to modify the new rules files to be suitable for inclusion in the KcKL. This process includes the following activities:
- Assign trap severity, type and expiration;
- Determine related traps for event correlation; and,
- Determine generic changes.
Each activity is described below in detail.
Assigning trap severity, type and expiration
The process of assigning the trap severity, type and expiration requires some knowledge of each trap. If you are not responsible for the creation of the MIB, it's likely you will be required to rely on third-party information to determine the purpose of each trap contained in the MIB. However, any well written MIB will include relevant information in the DESCRIPTION field that will at least give some purpose for each trap. More information may also be provided in comments (lines beginning with "--"). Examine the MIB file supplied, specifically lines 160 - 164 for an example.
156: symSCSI1 TRAP-TYPE 157: ENTERPRISE symTrap 158: DESCRIPTION 159: "Symbios SCSI Agent is up." 160: --#SUMMARY "Symbios SCSI Agent is up." 161: --#SEVERITY INFORMATIONAL 162: --#STATE OPERATIONAL 163: --#HELP "symscsi.hlp" 164: --#HELPTAG 101 165: ::= 101 |
Each trap is listed in the exported rules file with extension ".sev.snmptrap.lookup". Using our example, the file is symbios-SYMTRAP-MIB.sev.snmptrap.lookup. This lookup file contains three columns: severity (default: 1), type (default: 0), and expiration (default: 0). Examine the contents of the MIB for each trap to determine the appropriate values of these columns.
Severity maps to the Severity field of the event in the ObjectServer to which the probe communicates. Valid values will depend on your implementation, with defaults between 0 (clear) and 5 (critical). Alternatively, the value 'd' can be used to simply discard the event without passing it to the ObjectServer.
Type can be assigned a value of either 1 (problem) or 2 (resolution), depending on the trap, and maps to the Type field of the event in the ObjectServer. Note that resolution traps must be assigned a severity of 1. Further discussion on the impact of assigning the type field can be found in defining correlation.
Expiration determines the period for which the event should reside in the ObjectServer before being cleared. This value is specified in seconds, with a default value of 0 (never expire). Again, the value to use will depend on the trap and the relative importance of the trap to your environment.
For example, SYMTRAP-MIB defines the trap called symSCSI5. This trap has a DESCRIPTION of "The SCSI Controller# %d has Failed",
with further comments suggesting the trap has a severity of CRITICAL. Therefore, the recommended values for this trap are:
SNMPTRAP-symbios-SYMTRAP-MIB-symSCSI5 5 1 0, meaning the trap will be assigned a severity of
5 (critical), type of 1 (problem), and expiration of 0 (never expire). Note: Ignore the '%d' in the description for now - further
discussion about this place-holder can be found in determining generic changes below.
It must be noted that in some cases, the severity, type and expiration values will depend on complex conditions. For example, the severity might be determined by a value contained in the trap. In such cases, the severity lookup file (the file ending in ".sev.snmptrap.lookup") cannot be used to assign the severity, type or expiration. Instead, assign the value in the appropriate rules file (i.e. the file ending in ".include.snmptrap.rules"). Additionally, remove, or comment-out, the corresponding line the severity lookup file.
For example, you may wish to assign the severity for symSCSI9 traps depending on the specific SCSI Controller device number
(whose value is assigned in line 171: $scsiController = $1). Modify the rules file according to your logic
then assign the severity, type and expiration using the fields, $DEFAULT_Severity, $DEFAULT_Type and
$DEFAULT_ExpireTime. Finally, comment-out line 48 in symbios-SYMTRAP-MIB.sev.snmptrap.lookup by adding the '#' character to
the beginning of the line, as follows: SNMPTRAP-symbios-SYMTRAP-MIB-symSCSI9 1 0 0.
In many cases, problem traps have a corresponding resolution trap. For example, using the example MIB, the symSCSI6 trap is a resolution trap to the problem trap symSCSI5, with meaning "SCSI Controller# %d recovered". Therefore, we want our ObjectServer to correlate the events generated by these traps.
By default, the ObjectServer will correlate events using the generic_clear trigger (defined in the ObjectServer Automations). The generic_clear trigger standardizes correlation by looking at the contents of the Type field instead of looking for specific resolutions (Link Up, Port Restored) and correlating them with specific problems (Link Down, Port Fail).
For efficiency, the actual code that does the correlation looks complicated, but the principle is simple: Find all resolution (Type 2) events, and match them with corresponding problem (Type 1) events that have identical values in certain fields (by default: Manager, Node, AlertGroup, AlertKey) and have the correct time relationship (the problem occurred before the resolution) and clear them.
Given the generic_clear trigger functionality, we need to ensure that the exported rules files correctly assigns values to the fields
used for correlation. Examining the exported rules file for the example MIB, note the value assigned to the AlertGroup field for trap
symSCSI5 (line 91): @AlertGroup = "symSCSI5". This value does not match the value assigned to
the corresponding symSCSI6 trap, which has the following value (line 109): @AlertGroup = "symSCSI6".
Changing these values to be the same (e.g., @AlertGroup = "SCSIController") will ensure the ObjectServer events
are correlated correctly (in this case the other correlation related fields are the same and do not require modifications).
MIB2Rules makes a best effort attempt at assigning values to specific ObjectServer fields based on the information available to it. However, sometimes the assigned values are not sufficient. Therefore, the rules file should be examined to determine insufficient assignments and corrected accordingly.
Using the example rules file, refer to line 53 (shown here as two lines):
@Summary = "symSCSI3: The SCSI Controller# %d with ControllerName %s and ManagerName Id %s has been discovered" |
Note the use of "%d" and "%s" - these values will be literally assigned within the Summary field, which is obviously incorrect. Therefore, the line should be modified as follows:
@Summary = "The SCSI Controller# " + $scsiController + " with ControllerName " + $controllerName + " and ManagerName Id " + $managerName + " has been discovered" |
Another field that typically requires close examination is @Identifier. Be wary of @Identifier including variables that are a date+time (usually the date and time the trap occurred) - doing so will break the ObjectServer correlation (specifically, de-duplication) functionality.
The modified rules are now ready to be merged into a KcKL installation.
Merging the new rules files into the KcKL is a simple process that includes the following tasks:
- Copy the new files to the $NC_RULES_HOME/include-snmptrap directory
- Modify $NC_RULES_HOME/snmptrap.rules to:
- Embed the new rules files (e.g symbios-SYMTRAP-MIB.include.snmptrap.rules and symbios-SYMTRAP-MIB.adv.include.snmptrap.rules). To do so, locate the includes section and add the following line:
include "$NC_RULES_HOME/include-snmptrap/symbios-SYMTRAP-MIB.include.snmptrap.rules |
- Include the new severity lookup table (e.g symbios.SYMTRAP-MIB.sev.snmptrap.lookup). To do so, locate the tables section and add the following lines (Note: The first two lines should be entered as a single line):
table symbios-SYMTRAP-MIB_sev =
"$NC_RULES_HOME/include-snmptrap/symbios-SYMTRAP-MIB.sev.snmptrap.lookup"
default = {"Unknown","Unknown","Unknown"}
|
- Force the Mttrapd probe to re-read the rules file, e.g.,
kill -HUP pid(where pid is the PID) - Perform a probe syntax check, e.g.,
nco_p_syntax -rulesfile $NC_RULES_HOME/snmptrap.rules. Note: This task requires the nco_p_syntax probe, which can be obtained from your IBM representative.
The merged rules are now ready for testing.
Testing the new rules files simply involved forcing the generation of a trap. Sometimes, this is easier said than done; however, most devices allow for the generation of a test trap, whose definition should be included in the MIB.
Once the trap has been received and an event is generated in the ObjectServer, confirm the correctness of relevant fields; including, but not limited to: Summary, AlertGroup, AlertKey, Type and Identifier.
If your environment allows, test for event correlation by sending a problem event followed by a resolution event. Ensure the problem event is cleared once the resolution event is generated. If the events are not correlated, confirm the correlation fields are matching, specifically the AlertGroup field.
Finally, ensure that multiple traps (with the same values) generate only one event, with the event's tally field incremented for each trap. If a new event is generated for each unique trap, check the assignment of the Identifier field in the rules file.
The extensibility and flexibility of IBM Tivoli Netcool OMNIbus allows for a wide range of devices to be supported with minimal effort in your Event Management solution. Using the IBM Tivoli Netcool/OMNIbus Knowledge library in conjunction with the MIB2Rules utility extends the device support of the Mttrapd probe with little effort and no down-time. Notably, the solution is not limited to network devices but can be extended to any system that communicates via the SNMP, including applications and other monitoring solutions. This positions IBM Tivoli Netcool/OMNIbus as the ideal candidate for the role of 'Manager of Managers'.
| Description | Name | Size | Download method |
|---|---|---|---|
| Example MIB1 | SYMTRAP-MIB.mib | 17KB | HTTP |
| Resulting exported files using example MIB | Exported-Files.rar | 6KB | HTTP |
Information about download methods
Note
- The latest version of the MIB should be obtained from Engenio Information Technologies, Inc.
Learn
-
Refer to
IBM Tivoli Netcool/OMNIbus documentation for more information about Netcool/OMNibus.
-
Refer to
IBM Tivoli Netcool/OMNIbus Knowledge Library for more information about Netcool/OMNibus Knowledge Library.
-
Go to the
Integrated Service Management Library
website to download the MIB2Rules (now MIB Manager) utility.
Discuss
- Check out
developerWorks® blogs and get involved in the
developerWorks community.
Peter Tuton is a Software Engineer in the Tivoli Security Integration Factory Team based on the Gold Coast, Australia. In this role, Peter is the Technical Lead for the team developing integration solutions for Tivoli Access Manager and Tivoli Federated Identity Manager. Peter has been involved in the certification of a number of IBM integration solutions with leading software companies, including Siebel, PeopleSoft and SAP.
