Creating DNS Lookup Monitors and Situations to Send Alerts to Omnibus

This guide shows you how to create DNS (Domain Name Server) monitors and situations to alert you when the DNS monitor returns a "Bad" result. Normally, when a DNS monitor returns a "Bad" result, it means that the DNS is not working correctly.

We are going to start by showing how the DNS monitor is similar to the familiar nslookup command, how lookup information is presented in ISM workspaces, and then how you create a situation to generate custom Omnibus alerts when the DNS monitor returns a "Bad" result.

Comparison to nslookup Command

The DNS monitor function is similar to the results that the nslookup command returns.

In the simplest case, the nslookup command takes a single parameter: either the hostname when you want to know the IP address for that host, or the IP address when you want to find out which hostname is associated with that address.

For example, here we use the hostname in an nslookup command. The nslookup command tells us the default DNS used to resolve that hostname and the IP address for that DNS host. Here, the DNS host is IP 9.42.106.2, and its hostname is ns01.rtp.raleigh.ibm.com. The DNS looked up the hostname argument and returned IP 9.37.226.16. This is the IP address for host gsagents.rtp.raleigh.ibm.com.                                                                                                                                                                                               

   C:\IBM\ITM>nslookup gsagents.rtp.raleigh.ibm.com
   Server:  ns01.rtp.raleigh.ibm.com
   Address:  9.42.106.2

   Name:    gsagents.rtp.raleigh.ibm.com
   Address:  9.37.226.16

Here is a variation of the nslookup command.  The command takes two parameters: the hostname or IP address you want to lookup and the IP address of the DNS you want to use to resolve the hostname or IP address.

   nslookup <hostname or IP to lookup> <dns IP address>

Example:

Here, we are asking the DNS on host 9.42.106.2 to do a reverse lookup on a hostname. The DNS returns its hostname and IP address, and the IP address for the hostname we provided.

   C:\IBM\ITM>nslookup gsagents.rtp.raleigh.ibm.com  9.42.106.2
   Server:  ns01.rtp.raleigh.ibm.com
   Address:  9.42.106.2

   Name:    gsagents.rtp.raleigh.ibm.com
   Address:  9.37.226.16

DNS Lookup  Terminology: When you give a hostname and receive an IP address, the request is called a "forward" lookup. In contrast, when you provide an IP address and receive a hostname, the request is called a "reverse" lookup.

How to Interpret DNS Monitor Information in ISM Workspaces

Here, we are going to work backwards, starting with the ISM workspaces. It is helpful to understand how DNS monitor information is presented first -- so you will be able to construct the DNS monitors to post custom information.

Understand that the ISM Hosts workspace presents metrics for different kinds of ISM monitors. For a DNS monitor, the ‘Host’ column identifies a DNS. Not all of the addresses or hostnames in the Hosts column are DNS servers. If you are monitoring a number of DNS servers, one approach is to only assign DNS monitors to a single ISM agent. Then, the Hosts workspace for that agent will only present DNS status.

Here, the DNS server is 9.42.106.2. In this example, the remaining hosts are not DNS servers.

In this example, we happen to know that IP 9.42.106.2 is the address of a DNS host. When you drill down thru the link at the beginning of this row, you see the status of the DNS monitors. For example:
 

In this screenshot, the Host column identifies the DNS server (the resolver), the MonLoc column tells us where the DNS monitor is running, and the Description column tells us the hostname or IP address we asked the DNS to resolve. When you create DNS monitors, it is important to construct the Description as unique text that identifies whether this is a forward lookup or a reverse lookup, and what hostname or IP address you were trying to resolve.

Creating DNS Monitors in ISM Configuration Editor

In the ISM config editor, the server column identifies the IP address of the DNS server (the name resolver) and the host column identifies the hostname or IP address you would like the DNS to resolve.
 

​

Creating Situations to Raise Alerts on "Bad" DNS Lookups

You don't want to inspect the DNS monitor status in the TEP constantly to discover when a monitor is receiving a "Bad" result from a DNS.

Instead, you want a situation to raise an alert and send an email and/or raise an alert on Omnibus so the network people can resolve the issue.

In the TEP Navigator Tree, right-click ‘Internet Service Monitors’ and select ‘Situations’ from the popup menu.

The situation editor loads. Make sure that you can see all of the available ISM situations.

Note: You may need to configure the EIF event classes for ITCAM for Transactions. Select an existing situation. Review the following technote to verify that the event classes are available for the ITCAM for Transactions agents. If the event classes are not available, use the technote to guide you through the process of configuring the event classes. If you need to configure the event classes, make sure you are using an appropriate ITM Tools CD image for your ITM installation.

See ‘Preparing ITM Event Integration Facility to forward events to Omnibus’

https://www-01.ibm.com/support/docview.wss?uid=swg21515472

Create a new ISM situation. Please do not edit product-provided situations. However, you can make a copy of an existing situation.

TIP:  Use your initials at the beginning of the situation name. That way, people will know who created the situations.

Browse the attribute groups and attributes. Review the ITCAM for Transactions V7.4.0.1 User’s Guide for attribute definitions. In this example, in the situation formula, we use the attributes ServiceLevel and Service from the KIS SERVICE INSTANCE STATISTICS ATTRIBUTE GROUP.

Pick one attribute from the list. Click OK to create the situation with a single attribute in the situation formula. Then, click the ‘Add conditions…’ button to add the second attribute.

IMPORTANT: Do not mix attributes from different attribute groups.

IMPORTANT: Do not enclose the Service name in single quotes (apostrophes). Allow the situation editor to enclose the string for you. The situation will not work correctly if you enclose the string in quotes.

TIP: In the situation formula, include Service == ‘DNS’ so the situation will only fire on DNS monitor events.

Now set the sampling interval. The sampling interval should match the DNS monitor sampling interval. The default is 5 minutes. Remember to tic the ‘Run at startup’ checkbox to start the situation when the ISM agent restarts.

Then, click the ‘Advanced’ button, and click the ‘Display item’ tab. Set the Display item to use the ‘Description’ attribute in the KIS SERVICE INSTANCE STATISTICS attribute group. The Display item allows the situation to recognize DNS monitor events uniquely – based on the value of the Description you entered in the monitor definition when you created the DNS monitor.

Click the ‘Action’ tab at the top of the situation editor screen. Set the radio buttons so that the situation runs on the ISM agent. You can enter a system command that will be executed on the ISM agent host. Use the ‘Attribute Substitution’ button to add attributes to the string. Use a similar approach to send email.

In this example, we write a string of attributes to a flat file on the ISM agent host. This approach is often useful for problem determination. The system command writes the string every time the situation fires. Note that you create the system command as a single line without line breaks.

echo DNS_Resolution_Failed: &{KIS_SERVICE_INSTANCE_STATISTICS.Timestamp} '&{KIS_SERVICE_INSTANCE_STATISTICS.Description}' ISM Agent &{KIS_SERVICE_INSTANCE_STATISTICS.MonLoc} >> C:\Temp\gsDNS_Resolution_Failed.txt

Click the ‘EIF’ tab at the top of the screen. Then, click the ‘EIF Slot Customization’ button at the bottom of the screen.

In the ‘Event class name’, select class ITM_KIS_SERVICE_INSTANCE_STATISTICS. Note that you need to use the same attribute group here that you used for the situation formula. For the ‘msg’ slot, enter the custom message that will be eent to Omnibus.

TIP: The format of this message is slightly different than the format of the system command you defined on the Action tab.

In this example, we defined the following custom message. Note that the message must not contain embedded line breaks.

gsDNS_Resolution_Failed : $KIS_SERVICE_INSTANCE_STATISTICS.Timestamp$ $KIS_SERVICE_INSTANCE_STATISTICS.Description$ ISM Agent $KIS_SERVICE_INSTANCE_STATISTICS.MonLoc$

In this example, this string evaluates to the following message:

Click Apply and OK buttons to apply your changes and exit the situation editor. Make sure your situation has started.

IMPORTANT: EIF messages are only sent to Omnibus the first time when a situation goes from false to true.