Consolidated views of IBM Tivoli Directory Server components using IBM Tivoli Monitoring

Document options Document options requiring JavaScript are not displayed Rate this page Help us improve this content

Level: Intermediate

Amit Bhate (abhate@in.ibm.com), Staff Software Engineer, IBM India
Ramakrishna Gorthi (rjgorthi@in.ibm.com), Staff Software Engineer, IBM India

01 Nov 2007

IBM® Tivoli® Monitoring monitors and manages system and network applications on a variety of platforms and keeps track of the availability and performance of all parts of your enterprise. This article shows how IBM Tivoli Monitoring can be used to do a consolidated monitoring of specific components of the IBM Tivoli Directory Server.

Introduction

Any successful deployment of a directory solution requires ongoing monitoring to ensure that system performance and availability meets organization's goals. This article documents a procedure to do a consolidated monitoring of specific components of the Tivoli Directory Server using the Tivoli Monitoring Universal Agent. The article helps organizations to have single view statistics of the entire replication and the distributed directory topologies.

This article mostly focusses on monitoring Tivoli Directory Server version 6.1, but it can be extended to higher versions of Tivoli Directory Server by following the documented examples.

Some of the key questions that this article addresses that are important to a Tivoli Directory Server deployment are:

• What is the status of all the queues in the replication topology?
• Is any of the servers in the replication topology out of sync?
• How good is the proxy server serving requests?
• Is there any overloaded back-end server in the distributed directory topology?

IBM Tivoli Directory Server

The IBM Tivoli Directory Server implements the Internet Engineering Task Force (IETF) LDAP V3 specifications. It also includes enhancements added by IBM in functional and performance areas. It uses IBM DB2® as the backing store to provide per LDAP operation transaction integrity, high performance operations, and on-line backup and restore capability. The IBM Tivoli Directory Server interoperates with the IETF LDAP V3 based clients.

IBM Tivoli Monitoring

• Visualize real-time monitoring data from your environment.
• Monitor resources in your environment for certain conditions, such as high CPU or an unavailable application.
• Establish performance thresholds and raise alerts when thresholds are exceeded or values are matched.
• Trace the causes leading up to an alert.
• Create and send commands to systems in your managed enterprise by means of the Take Action feature.
• Use integrating reporting to create comprehensive reports about system conditions.
• Monitor conditions of particular interest by defining custom queries using the attributes from an installed agent or from an ODBC-compliant data source.

Software pre-requisites

The article is written with the following software requirements:

• IBM Tivoli Monitoring Version 6.1.0
• IBM Tivoli Universal Agent Version 6.1.0
• IBM Tivoli Directory Server Version 6.1

The information about installing these products is available through the links in the Resources section below.

This article will cover the information on setting up the Universal Agent. For information about configuring/setting-up the rest of the products, see the documents in the Resources section below.

Back to top

Distributed directory and Replication setup

As mentioned in the introduction section, this article addresses some key questions pertaining the directory server. The answers are in the form of consolidated views of the following two directory server components:

• Distributed Directory
• Replication

The basic server used to setup all the above instances is assumed to be lbmw.in.ibm.com and the servers are running on the following ports:

The entire topology shown in Figure 3 above is used for monitoring the consolidated distributed directory statistics.

The replication topology for Partition 1 is used for monitoring the consolidated replication statistics.

Back to top

Universal Agent Essentials

Tivoli Enterprise Monitoring Agents are installed on the systems or subsystems whose applications and resources are to be monitored. The agent collects monitoring data from the managed system and passes it to the monitoring server to which it is connected. The client gathers the current values of the attributes and produces reports formatted into tables and charts. It can also test the values against a threshold and display an alert icon when that threshold is exceeded or a value is matched. These tests are called situations.

The IBM Tivoli Universal Agent is a generic agent of IBM Tivoli Monitoring. You can configure the IBM Tivoli Universal Agent to monitor any data you collect. You can view the data in real-time and historical workspaces on the Tivoli Enterprise Portal and manage with Tivoli Enterprise Portal monitoring situations and automation policies, the same as data from other Tivoli Enterprise Monitoring Agents.

The Universal Agent is thoroughly documented in the IBM Tivoli Universal Agent User's Guide, but a few details are worth mentioning here.

Starting and stopping the Universal Agent

On Linux, CANDLEHOME is typically set to /opt/IBM/ITM, if the default installation path is followed.

Another environment variable that needs to be set for the Universal Agent to work is the UAHOME. On Linux, the UAHOME is set to /opt/IBM/ITM/li6243/um. These settings may change with the operating system and with the path of installation.

If you prefer GUI for the settings, you can right-click on the Universal Agent in the Navigator view and choose Start, Stop or Restart.

Defining data in the Universal Agent

An IBM Tivoli Universal Agent application consists of one or more attribute groups, each group consisting of one or more attributes. The application to be monitored is defined in a data definition file, called a metafile, which is imported by the IBM Tivoli Universal Agent. The command to import the metafile is um_console.

The metafile is a plain text file. It contains the following control statements in the order shown (if present):

Choosing a data collection approach

The Universal Agent supports several different ways of providing data to the monitoring server, including log file parsing and script execution, all of which will be used in this article. The Universal Agent can also receive events via SNMP and API calls, as well as monitor Web servers via URL polling, and databases via ODBC and receive data over sockets; however, this article does not make use of those capabilities.

The best choice of data provider is going to be influenced by the source of the data and how you want to monitor it, that is, polled, continuously, or event-driven. Since any data manipulation (for example, calculating operations outstanding, operation completion rate) must be done by the data provider before the TEMS receives it and passes it on to the portal, the choice of data provider becomes important.

When the data of interest is already in a log file and just needs to be parsed and provided to the monitoring server as it is written to the log file, the log file adapter is the most useful tool to use. If only simple calculations (add, subtract, divide or multiply integers) need to be performed on the data before sending it to the monitoring server, the calculations may be embedded in the metafile data descriptions and performed by the logfile adapter.

If more advanced processing needs to be done (e.g. calculating response times from timestamps) the best approach is the socket provider, which can receive data from a program or script that reads the log file, performs the processing and writes the data to a socket connected to the Universal Agent.

When the data needs to be retrieved periodically from some utility (e.g. using ldapsearch to fetch the cn=monitor statistics) then an executable invoked by the script data provider is the best approach.

Viewing Tivoli Monitoring data

This section provides information as to how the Tivoli Monitoring graphs/reports can be viewed. These are the graphs/reports that the portal generates in response to the metafile and scripts shown in the article. There are a couple of ways for viewing the graphs/reports:

• Launch the GUI for managing the Tivoli Enterprise Monitoring Services using the command:
  
  

  $CANDLEHOME/bin/itmcmd manage

  
  
  In the above GUI, right click on the Tivoli Enterprise Portal Desktop Client and hit on Start.
• Launch the Tivoli Enterprise Portal using the link:
  
  

  http://host:1920///cnp/kdb/lib/cnp.html

  
  
  where,
  host is the name of the system where the Tivoli Enterprise Portal Server is running.

Once the Tivoli Enterprise Portal is launched, in the navigator on the left, you would find the hostname of the monitoring server listed under the node of the corresponding operating system. Under the Universal Agent select the application host:TDSxx to list down the attribute groups you have in your mdl file.

Describing the data to be monitored

All of the definitions developed in the rest of this paper will be contained in the metafile for the TDS application. The file is named tds.mdl, which starts with declaring the name of the application like this:

This declares the name of the application as TDS, which will show up in the Portal underneath the node name as an application named TDSnn, where nn is the version number of the metafile, starting initially at 00 and incrementing each time you make significant changes to it. The IBM Tivoli Universal Agent User's Guide lists the types of metafile updates that result in major or minor version changes.

After the name of the application, the data description is provided to the Universal Agent. The data is described to the Universal agent in the format as below:

In the above example, the data description starts with the //NAME line, naming the attribute group ProxyMonitorStats, giving it a type of P (polled) with a 60 second lifetime, and have the Universal Agent add a timestamp each time the file is read. The //SOURCE line declares the source of the data is a log file named proxy.dat, which copies recently polled data in the internal buffer with the "COPY" clause. The "COPY" clause indicates that the file be handled in the block mode. The data is saved to an internal buffer name PxyMonStats2 for further processing by another attribute group. The data is separated by semicolons.

After the above data description, the type of each attribute is specified. Here are some examples of the different attribute types:

Display strings, such as the APP_version string, are declared with a D followed by the maximum length of the string. Integer values, such as the TotalConnections counter, are declared with a C followed by the maximum value of the integer. Place holders for attributes which can be skipped are declared with the type K. Integer attributes like TotalResultsSent which need to store negative values are declared with type G.

Back to top

Consolidated Distributed Directory Monitoring

This section provides details as to how monitor searches can help in getting the consolidated results of the distributed directory topology.

Proxy Monitor Search Analysis

The statistics of the Proxy Server at any given instant of time can be gathered using the monitor search. It's a base-scoped search against "cn=proxy, cn=monitor". The interaction between the proxy server and the corresponding back-end servers is tracked using a base-scoped search against "cn=backends, cn=proxy, cn=monitor".

Both the above searches are supported only by the Proxy Server. These search results need to be consolidated in a single view to provide the status of the entire distributed directory. The consolidation can be done by means of a C program.

The following assumptions are made for explaining the program details:

• The LDAP URL of the proxy server is passed to the program.
• The program dumps the consolidated monitoring statistics to proxy.dat
• The program name is proxyMonitor.c and the corresponding binary is named proxyMonitor.

1. When the program is invoked, transfer the contents of proxy.dat to proxyOld.dat. The basic intent of this step is to store the data from a previous run of the program in a different file i.e. proxyOld.dat
2. Ensure that proxy.dat is blank and open it in append mode so that formatted data can be dumped to it as and when the data is ready.
3. Read the contents of proxyOld.dat in memory. This would be required to calculate the completion rate of different operations.
4. Perform a proxy front-end monitor search on the Proxy Server:
   
   

   ldapsearch -h <ProxyHostName> -p <ProxyPort> -D <bindDN> -w <bindPW> -b "cn=proxy, cn=monitor" -s base objectclass=*

   
   
   
5. Collect the output of the above command in a file and then read the same in memory.
6. Parse this data to form a semicolon(;) separated list of all the attribute values.
7. Calculate the rate of change of these attribute values with respect to the data in proxyOld.dat.
   e.g. (OperationsCompleted from Step 5 - OperationsCompleted from proxyOld.dat)/60 would give the OperationCompletionRate.
   Note that in this calculation it is assumed that the proxy statistics are collected in intervals of 60 seconds.
8. Append the compiled data to proxy.dat in the format APP_Version;Proxy;Server:Port;Data
   where,
   APP_Version is the string "IBM Tivoli Directory, 6.1"
   Proxy is the string "Proxy"
   Server:Port is the Hostname:port OR HostIP:port of the system on which the proxy server is running.
   Data is the data collected and calculated in the previous step.
   
9. Perform a proxy back-end monitor search on the Proxy Server.
   
   

   ldapsearch -h <ProxyHostName> -p <ProxyPort> -D <bindDN> -w <bindPW> -b "cn=backends, cn=proxy, cn=monitor" -s base objectclass=*

   
   
   
10. Collect the output of the above command in a file and then read the same in memory.
11. Parse this data to form a semicolon(;) separated list of all the attribute values.
12. Calculate the rate of change of these attribute values with respect to the data in proxyOld.dat. This procedure is similar to the one performed in Step 7 above.
    The only difference between the calculations of the proxy back-end statistics and the proxy front-end statistics is that in the case of proxy back-end statistics, a set of rows is to be handled. The number of rows depends upon the number of back-end servers configured for the proxy server. The statistics for each back-end server are stored on a newline.
13. Append the compiled data to proxy.dat in the format APP_Version;Backend[1..n];Server:Port;Data
    where,
    APP_Version is the string "IBM Tivoli Directory, 6.1"
    Backend[1..n] is the string "Backend1" for the 1st. back-end, "Backend2" for the 2nd. back-end and so on till the nth back-end.
    Server:Port is the Hostname:port OR HostIP:port of the system on which the back-end server is running.
    Data is the data collected and calculated in the previous step.
    
14. The proxy back-end monitor searches don't have as many attributes as are displayed by the proxy front-end monitor searches. While entering data in proxy.dat enter -1 as the value against such attributes.

Here's how the proxy.dat file will look like after the above procedure:

The mdl file needs to be updated to display the attributes gathered out of the monitor search. The mdl file (tds.mdl) will look like:

As seen earlier, proxy.dat would be populated by proxyMonitor. However, there needs to be a mechanism in place which would invoke proxyMonitor periodically. This can be achieved by means of the following attribute group:

The attribute group used to specify the binary for fetching the proxy monitor data is named ~ProxyMonitorStatFetch. The ~ signifies that this is an invisible attribute group and hence won't be displayed in the console. It is just used for fetching the data and populating the proxy.dat file. The actual work of displaying the data is done by the ProxyMonitorStats and the ProxyStats attribute groups. ~ProxyMonitorStatFetch is invoked in intervals of 60 seconds and the LDAP URL of the proxy server is passed as an argument to proxyMonitor.

Graphs on Proxy Monitor Search results

Launch the TEPC using the steps shown in section Viewing Tivoli Monitoring data.

Using the data collected through the attribute group ProxyMonitorStats, a set of graphs can be drawn.

• Open the report for the ProxyMonitorStats attribute group, by clicking on the same in the navigator on the left.
• Click the Bar Chart tool icon and click on the Table view which contains the data to be plotted.
• A dialog box will pop-up providing you the option of selecting the attributes that can be plotted in the bar chart.
• Select OpsInitiated and click OK.
  
  Figure 6: Select attributes for plotting bar chart
  
  
  

Proxy Performance statistics

A lot of the data returned by "cn=proxy, cn=monitor" and "cn=backends, cn=proxy, cn=monitor" searches is in the form of counters, e.g. OpsInitiated and OpsCompleted. By doing some simple arithmetic we can derive useful results, such as current operations in progress, which is (OpsInitiated - OpsCompleted). The Universal Agent can do this arithmetic for us through derived variables in the metafile.

proxyMonitor would compute the completion rate for operations as well. The polling interval is passed to proxyMonitor to do these computations. The Universal Agent can perform these calculations if there is just one single row to work upon. For multiple rows in the output, the numbers picked up to calculate the difference aren't the way they are supposed to be. So, it is required that the completion rates be computed in the C code (proxyMonitor.c) itself.

The proxy.dat generated by proxyMonitor is used to view the "Proxy Performance Statistics".

As far as the Universal Agent goes, we start with the "//NAME" line, naming the attribute group ProxyStats, giving it a type of P (polled) with a 60 second lifetime, and have the Universal Agent add a timestamp each time the file is read. The "//SOURCE" line declares the source of the data is a log file named proxy.dat which copies recently polled data in the internal buffer with the "COPY" clause. The "COPY" clause basically indicates that the file be handled in the block mode. The data is saved to an internal buffer named PxyMonStats for further processing by another attribute group, and the data is declared to be separated by semicolons:

Display strings, such as the APP_version string, are declared with a D followed by the maximum length of the string. Integer values, such as the OpsInitiated counter, are declared with a C followed by the maximum value of the integer. In order to display both positive and negative integers, variables are declared with a G followed by the maximum value of the integer. Some fields need not be displayed as part of the ProxyStats Attribute group. Such attributes are declared with a K, which indicates that the relevant fields are to be skipped.

As seen in the previous section, proxy.dat would be populated by the attribute group ~ProxyMonitorStatFetch.

Graphs on Proxy Performance

Launch the TEPC using the steps shown in section Viewing Tivoli Monitoring data.

Using the data collected through the attribute group ProxyStats, a set of graphs can be drawn.

• Open the report for the ProxyStats attribute group, by clicking on the same in the navigator on the left.
• Click the Bar Chart tool icon and click on the Table view which contains the data to be plotted.
• A dialog box will pop-up providing you the option of selecting the attributes that can be plotted in the bar chart.
• Select the attributes of the type *Rate and click OK.

For getting the above view, click on the Plot Chart tool  and then click on the report where you want to pick up fields for plotting the chart. Once that's done a popup will appear to select the set of attributes to be plotted. Select the attributes and click on OK to see the above graph. Refreshing the screen at a given point of time provides the plot of the chart till that point of time.

Back to top

Consolidated Replication Monitoring

The statistics on the replication topology at any given instant of time can be gathered using a set of operational attributes. The requisite operational attributes are passed to searches against the replication agreements.

As seen in the output above, we are getting the change count, replication status and the replica URL only for those agreements for which the server on port 2389 is the supplier (the server on which the above search was fired). For the rest of the agreements no data is returned. We would need to compile a list of all the replication agreements in the topology and fire the above query on the corresponding suppliers.

This task is achieved by a C program. The program will parse the output of a search like the one shown above, gather the change count and status and arrange them in a single line. The data pertaining to a given agreement will appear on a separate line. The replica URL would be used to fire recursive searches and get the pending change count and replication status for the other agreements in the topolgoy. On each line the supplier and the consumer would also be noted. All the fields would be separated by a semicolon (;). A single log file (repl.dat) would be used to dump the formatted results of the searches against all the suppliers in the topology. The Universal Agent would pick up the data from repl.dat.

The following assumptions are made for explaining the program details:

• The LDAP URL of one of the suppliers in the replication topology is passed to the program.
• The program dumps the consolidated replication statistics to repl.dat.
• The program name is replTopology.c and the corresponding binary is named replTopology.

1. Get the LDAP URL of the supplier. This is the one that the user passed in.
2. Check if the program has already processed the LDAP URL. If YES, go to Step 8.
3. Search for entries of the type ibm-replicationAgreement in the directory (under the given replication context) and from such entries fetch the attributes ibm-replicationPendingChangeCount, ibm-replicationState and ibm-replicaURL.
4. From the search results pick the first entry.
5. If the entry contains the requisite replication attributes, process the entry as given:
   • Parse the attribute values and append the same to repl.dat. The format of appending the results would be: "ReplContext;SupplierHostName:SupplierPort;ConsumerHostName:ConsumerPort;PendingChangeCount;QueueStatus"
     where,
     ReplContext is the replication context for which the data is being collected.
     SupplierHostName:SupplierPort is the host:port detail of the supplier.
     ConsumerHostName:ConsumerPort is the host:port detail of the consumer.
     PendingChangeCount is the number of changes pending between the above supplier and consumer.
     QueueStatus is the status of the replication queue.
   • Store the ibm-replicaURL in a data structure like a two dimensional array, since later this URL can be used to fire the next set of searches. Assume that the name of this data structure is urlList.
6. If the entry doesn't contain the requisite replication attributes ignore the entry.
7. If there are any further entries obtained in Step 3 which are yet to be processed go to Step 5.
8. Get the next LDAP URL to be processed from urlList.
9. If there is an LDAP URL to be processed, go to Step 2.
10. If all the given LDAP URLs are processed, repl.dat should have the requisite information of the entire replication topology.

Here's how the repl.dat file will look like after the above procedure:

Now the data is described to the Universal Agent. We start with the "//NAME" line, naming the attribute group ReplicationStats, giving it a type of P (polled) with a 90 second lifetime, and have the Universal Agent add a timestamp each time the file is read. The "//SOURCE" line declares the source of the data is a log file named repl.dat which copies recently polled data in the internal buffer with the "COPY" clause. The "COPY" clause basically indicates that the file be handled in the block mode. The data would be stored to an internal buffer named ReplStatus for further processing by another attribute group. The data is declared to be separated by semicolons:

Display strings, such as the ReplContext string, are declared with a D followed by the maximum length of the string. Integer values, such as the PendingChangeCount, are declared with a C followed by the maximum value of the integer.

As seen earlier, repl.dat would be populated by replTopology. However, there needs to be a mechanism in place which would invoke replTopology periodically. This can be achieved by means of the following attribute group:

The attribute group used to do the actual search against the server is ~ReplStatusFetch. The ~ signifies that this is an invisible attribute group. It is just used for fetching the data and populating the repl.dat file. The actual work of displaying the data is done by the ReplicationStats attribute group. ~ReplStatusFetch is invoked in intervals of 60 seconds and the LDAP URL of any of the suppliers in the replication topology is passed as an argument to replTopology. The data fetched from that supplier is used to fetch the data pertaining to the other suppliers in the topology.

Graphs on Replication Statistics

Launch the TEPC using the steps shown in section "Viewing Tivoli Monitoring data".

Using the data that's collected through the attribute group ReplicationStats, a set of graphs can be drawn.

The above report shows that everything is normal. Customer deployments can be complicated with 100s of supplier-consumer pairs. Having a single view of all the replication queues in the topology makes it easier for administrators to locate the problematic queue and consequently the problematic server.

In the above report, lbmw.in.ibm.com:389 is not able to replicate contents to the server on port 3389. The current replication status for that queue is Connecting. Same is the status of the agreement from port