Tivoli Directory Server monitoring using IBM Tivoli Monitoring

Document options Document options requiring JavaScript are not displayed Rate this page Help us improve this content

Level: Intermediate

Dave Bachmann (bachmann@us.ibm.com), Senior Software Engineer, IBM US
Ramakrishna Gorthi (rjgorthi@in.ibm.com), Software Engineer, IBM Global Services, India
Amit Bhate (abhate@in.ibm.com), Staff Software Engineer, IBM India

14 Sep 2007

IBM® Tivoli® Monitoring monitors and manages system and network applications on a variety of platforms and keeps track of the availability and performance of all parts of your enterprise. This article details how IBM Tivoli Monitoring can be used to monitor the performance of the IBM Tivoli Directory Server.

Introduction

Any successful deployment of a directory solution requires ongoing monitoring to ensure that system performance and availability meet an organization's goals. This article documents a procedure to monitor the Tivoli Directory Server using the Tivoli Monitoring Universal Agent. The article is based upon a set of best practices for monitoring Tivoli Directory Server and its component parts. A parameter list that has been formulated over years of research into successful deployments of the directory server is being monitored through this article. The article also provides insights into setting situations to catch problems that administrators should be concerned about.

This article focuses on monitoring Tivoli Directory Server version 5.2 and 6.0 servers, but it can be extended to other versions of Tivoli Directory Server by following the documented examples.

Some of the key questions that this article addresses, which are important to any Tivoli Directory Server deployment, are

• Is the server available?
• How busy is the server?
• Is the server about to run out of memory?
• Is the server backlogged?
• How responsive is the server?
• Is the database about to run out of space?
• Are there any errors being logged?

Back to top

IBM Tivoli Directory Server

The IBM Tivoli Directory Server (TDS) implements the Internet Engineering Task Force (IETF) LDAP V3 specifications. It also includes functional and performance enhancements added by IBM.. TDS uses IBM DB2® as the backing store to provide per LDAP operation transaction integrity, high performance operations, and on-line backup and restore capability. The IBM Tivoli Directory Server interoperates with the IETF LDAP V3-based clients.

Back to top

IBM Tivoli Monitoring

• Visualize real-time monitoring data from your environment.
• Monitor resources in your environment for certain conditions, such as high CPU or an unavailable application.
• Establish performance thresholds and raise alerts when thresholds are exceeded or values are matched.
• Trace the causes leading to an alert.
• Create and send commands to systems in your managed enterprise by means of the Take Action feature.
• Use integrated reporting to create comprehensive reports about system conditions.
• Monitor conditions of particular interest by defining custom queries using the attributes from an installed agent or from an ODBC-compliant data source.

Back to top

Software pre-requisites

The article is written with the following software requirements:

• IBM Tivoli Monitoring version 6.1.0
• IBM Tivoli Universal Agent version 6.1.0
• IBM Tivoli Directory Server version 5.2 or 6.0

The installation information for these products is available through the links in the Resources section below.

This article covers the information on setting up the Universal Agent. For information about configuring and setting up the rest of the products, see the documents in the Resources section below.

Back to top

Universal Agent essentials

Tivoli Enterprise Monitoring Agents are installed on the systems or subsystems whose applications and resources that you want to monitor. The agent collects monitoring data from the managed system and passes it to the monitoring server to which it is connected. The client gathers the current values of the attributes and produces reports formatted into tables and charts. It can also test the values against a threshold and display an alert icon when that threshold is exceeded or a value is matched. These tests are called "situations".

The IBM Tivoli Universal Agent is a generic agent of IBM Tivoli Monitoring. You can configure the IBM Tivoli Universal Agent to monitor any data you collect. You can view the data in real-time and historical workspaces on the Tivoli Enterprise Portal and manage with Tivoli Enterprise Portal monitoring situations and automation policies, the same as data from other Tivoli Enterprise Monitoring Agents.

The Universal Agent is thoroughly documented in the IBM Tivoli Universal Agent User's Guide , but a few details are worth mentioning here.

Starting and stopping the Universal Agent

On Solaris, CANDLEHOME is typically set to /opt/IBM/ITM, if the default installation path is followed.

Another environment variable that needs to be set for the Universal Agent to work is the UAHOME variable. On Solaris, the UAHOME variable is set to /opt/IBM/ITM/sol286/um. These settings might change with the operating system and with the path of installation.

If you prefer the GUI for the settings, you can right-click on the Universal Agent in the Navigator view and choose Start, Stop or Restart.

Defining data in the Universal Agent

An IBM Tivoli Universal Agent application consists of one or more attribute groups, each group consisting of one or more attributes. The application to be monitored is defined in a data definition file, called a “metafile”, which is imported by the IBM Tivoli Universal Agent. The command to import the metafile is the um_console command. The command to import the metafile is um_console.

The metafile is a plain text file. It contains the following control statements in the order shown (if present):
SNMP:
For SNMP Data Providers only, introduces the data definition for IBM Tivoli Monitoring provided SNMP MIB applications. SNMP TEXT introduces the data definition for user-defined SNMP applications.
APPL:
Specifies the name that IBM Tivoli Monitoring uses for the application.
NAME:
Defines the name of an attribute group, the type of data being collected, and the period for which the data is valid.
INTERNAL:
Provides for data redirection between attribute groups as a way to perform additional processing.
SOURCE:
Defines the location of the data you are collecting.
RECORDSET:
For File Data Providers only, defines the set of records from which the data provider extracts data.
CONFIRM:
For Socket Data Providers only, specifies the requirements for data acknowledgment.
SQL:
For ODBC Data Providers only, defines the Select statement or stored procedure to use for collecting relational data.
SUMMARY:
Defines the requirements for gathering the frequency of data input during monitoring.
ATTRIBUTES:
Introduces the attribute definitions and specifies the attribute delimiters in the data string. Below the ATTRIBUTES control statement, list the individual attribute definition statements.

Choosing a data collection approach

The Universal Agent supports several different ways of providing data to the monitoring server, including log file parsing, script execution, and receiving data over a socket, all of which will be used in this article. The Universal Agent can also receive events via SNMP and API calls, as well as monitor Web servers via URL polling, and databases via ODBC; however, this article does not make use of those capabilities.

The best choice of data provider is going to be influenced by the source of the data and how you want to monitor it, that is: polled, continuously, or event-driven. Because data manipulation (for example, calculating response times, hit ratios, or percentages) must be done by the data provider before the TEMS receives it and passes it on to the portal, the choice of data provider becomes important.

When the data of interest is already in a log file and just needs to be parsed and provided to the monitoring server as it is written to the log file, the “log file adapter” is the most useful tool to use. If only simple calculations (add, subtract, divide or multiply integers) need to be performed on the data before sending it to the monitoring server, the calculations can be embedded in the metafile data descriptions and performed by the log file adapter.

If more advanced processing needs to be done (for example, calculating response times from timestamps), the best approach is the “socket provider”, which can receive data from a program or script that reads the log file, performs the processing, and writes the data to a socket connected to the Universal Agent.

When the data needs to be retrieved periodically from some utility (for example, using ldapsearch to fetch the cn=monitor statistics), then a shell script invoked by the script data provider is the best approach.

Describing the data to be monitored

All of the definitions developed in the rest of this paper will be contained in the metafile for the TDS application. The file is named tds.mdl, which starts with declaring the name of the application as shown below.

This declares the name of the application as TDS, which will show up in the Portal underneath the node name as an application named TDSnn, where the nn is the version number of the metafile, starting initially at 00 and incrementing each time you make significant changes to it. The IBM Tivoli Universal Agent User's Guide lists the types of metafile updates that result in major or minor version changes.

After the name of the application, the data description is provided to the Universal Agent. The data is described to the Universal agent in the format as shown below.

In the above example, the data description starts with the //NAME line, naming the attribute group LDAP_Monitor, giving it a type of P (polled) with a 90 second lifetime, and have the Universal Agent add a timestamp each time the script is run. The //SOURCE line declares the source of the data is a script named TDSMon.sh, with environment variable set from the TDS.env file, to run every 60 seconds. The data would be saved to an internal buffer named LdapMonitor for further processing by another attribute group. Declare that the data is separated by semicolons.

After the data description shown above, the type of each attribute is specified. Here are some examples of the different attribute types:

Display strings, such as the APP_version string, are declared with a “D” followed by the maximum length of the string. Integer values, such as the TotalConnections counter, are declared with a “C” followed by the maximum value of the integer. Floating point values, such as ElapsedTimeSeconds, can also have the Scale and Precision specified.

Back to top

Important Tivoli Directory Server metrics

Some of the questions that a monitoring deployment should be able to answer about Tivoli Directory Server are as follows:

• Is the server available?
• How busy is the server?
• Is the server about to run out of memory?
• Is the server backlogged?
• How responsive is the server?
• Is the database about to run out of space?
• Are there any errors being logged?

The data that the set of monitors, described in this article, collects for the Tivoli Directory Server answers those questions by reporting the following:

• Server availability
• Server process activity
• Size of the server process
• Workflow queue sizes
• LDAP request response times
• Tablespace sizes and utilization
• Errors logged to the Tivoli Directory Server message log

Back to top

Tivoli Directory Server monitoring

This section provides details as to how the monitor search results can be analyzed to derive the directory server metrics mentioned in the previous section.

Tivoli Directory Server availability and workload

The easiest way of checking the availability of the Tivoli Directory Server is by sending it a search request. If a monitor search request is used to check the server availability, it can also help us get workload statistics. A base-level search against the base cn=monitor for "objectclass=*" will return the following information:

The above output needs to be formatted into a single line of data separated by semicolons for the Universal Agent's consumption. This can be done with a simple awk script:

The output of monitor search above can be passed onto the awk script to get the data in the desired format. TDSMon.sh is the script where, the ldapsearch and the awk construct are put together. The contents of the TDSMon.sh script would be:

Now the mdl file needs to be updated to display the attributes gathered out of the monitor search. The associated mdl file (tds.mdl) will look like the following:

The contents of TDS.env file, which is used as an environment to the above script, are:

Where,
LDAPSERVER should point to the hostname or the IP of the directory server to be monitored.
LDAPDN should match the bind DN.
LDAPPW should match the bind PW.
It's assumed that the directory server runs on the port 389.

Tivoli Directory Server workload rates

A lot of the data returned by cn=monitor is in the form of counters, for example, OpsInitiated and OpsCompleted. By doing some simple arithmetic, useful results can be derived, such as current operations in progress, which are OpsInitiated - OpsCompleted. The Universal Agent can do this arithmetic for us when derived variables are declared in the metafile. For the example of the operations in progress, the OpsOutstanding counter is declared like this:

One can also get rates, such as OpsPerSecond, using the "?" type declaration, like this:

Earlier the data from the LDAP_Monitor attribute group was saved to an internal buffer named LdapMonitor. Here an attribute group is declared that uses that buffer as a source of data.

In the LDAPRates attribute group, the attributes that aren't useful for calculations are skipped, for example, APP_Version. The attributes to be skipped are prefixed with "-". Here is the entire attribute group:

Note here we’ve used "%" to divide and multiply by 100 (very useful for calculating percentages). Also note that attributes like FilterCacheAttempts attribute are an example of intermediate derived values, used to provide input to the calculation of attributes like FilterCacheHitRatio and not actually displayed.

Viewing Tivoli monitoring data

This section provides screenshots of the graphs and reports that the portal provides, in response to the metafile and scripts shown in the earlier section. There are a couple of ways for viewing the graphs and reports:

• Launch the GUI for managing the Tivoli Enterprise Monitoring Services using the command:

  
  
  

  $CANDLEHOME/bin/itmcmd manage

  
  
  
  In the above GUI, right-click on the Tivoli Enterprise Portal Desktop Client and click Start.

• Launch the Tivoli Enterprise Portal using the link:

  
  
  

  http://host:1920///cnp/kdb/lib/cnp.html

  
  
  

  Where,
  host is the name of the system where the Tivoli Enterprise Portal Server is running.

When the Tivoli Enterprise Portal is launched, in the navigator on the left, you will see the hostname of the monitoring server listed under the node of the corresponding operating system. Under the Universal Agent select the application host:TDSxx to list down the attribute groups you have in your mdl file.

The attribute groups describing the monitor statistics are the LDAP_Monitor and LDAPRates. As explained earlier, the LDAPRates attribute group contains a set of derived fields for each type of operation showing the number of operations currently being worked on by the server. A bar chart can be plotted with all the "XXX_Outstanding" counters, replacing the default Table view at the top of the screen.

• Open the report for the LDAPRates attribute group by clicking on the same in the navigator on the left.
• Click the Bar Chart tool icon and click on the Table view which contains the data to be plotted.
• A dialog box will pop-up providing you the option of selecting the attributes that can be plotted in the bar chart.
• Select all of the "...Outstanding" items (use the Ctrl key to make multiple selections) and click OK.

To change the label for the chart above:

• Right-click on the new chart and choose Properties....
• Click on the Style tab and change the text field from Bar Chart to Current Workload.
• Click on the picture of the legend on the right and click on the Legend Label tab and you can take Oustanding out of all the labels.
• Click on OK and you will see the updated chart.

To save all this work, click on any another attribute group, and answer Yes when asked if you want to save your changes to the LDAPRates attribute group.

The LDAPRates attribute group also has several attributes related to the cache utilizations and hit ratios. Figure 1 provides the report on cache utilizations and hit ratios.

Back to top

Tivoli Directory Server audit log

This section details the procedure on analyzing the audit log to calculate the response times of operations.

Enabling the Tivoli Directory Server audit log

In order to get Tivoli Directory Server response times, the audit log needs to be enabled. This can be done either via the Tivoli Directory Server Web Administration GUI or via an ldapmodify command. See the Tivoli Directory Server Administration Guide for information on enabling the audit log via the GUI. You can easily turn on and off auditing for different operations via an ldapmodify operation against the cn=audit,cn=localhost object. You can also query the current state of auditing via an ldapsearch against the cn=audit,cn=localhost object. For example, on Tivoli Directory Server v5.2:

For TDS v6.0 the audit log information is stored under cn=Audit,cn=Log Management,cn=Configuration. Hence, if you want to enable or disable audit log for TDS v6.0, use that as the base. For the sake of this example, TDS v5.2 is being assumed.

By default auditing is off (ibm-audit=false). You'll want to set ibm-audit=true as well as ibm-auditFailedOPonly=false and ibm-auditOpXXX=true for each operation you are interested in monitoring. Create the following LDIF file (ldapaudit.on.52) to be used in enabling the audit log:

This turns on auditing for every operation. The modification operation is passed to the Tivoli Directory Server server using ldapmodify:

Audit log with calculated response times

When the Tivoli Directory Server audit log is enabled, the server will write information to a file (/var/ldap/audit.log file) every time it sends the response to a request. Here is a snippet of an audit log for a bind followed by a search.

The first line, starting with “AuditV2”, is the same for every operation. After “AuditV2—“ is the timestamp of the reply, followed by "--", then the operation, then "--bindDN:" and the DN that the client was bound as, followed by "--" again. The client IP address and port is delimited by the "client:" strings and "--", the connection ID is delimited by "connectionID:" and "--", then after "received: " is the timestamp that the request was received, then "--" and finally the status.

The built-in capability of the Universal Agent to read formatted log files can be used to parse the audit log. The attribute group to parse the audit log can be defined as follows:

After giving the filename, the tail keyword tells the Universal Agent to watch for new records being appended to the end of the log file. Each record consists of up to 10 lines starting with "AuditV2", which is conveyed to the Universal Agent using the "//RECORDSET" keyword. The DLMSTRBGN keyword gives the string immediately preceding the desired data and the DLMSTREND keyword gives the string that follows. The lines of interest are the ones that begin with "AuditV2--" followed by a timestamp (which begins with "200") so the "+FILTER={MATCH(0,200)}" is used to say that the timestamp must begin with "200. If not, the line is ignored.

All of the fields are strings so the type of "D" is used.

The objective of parsing the audit log is to know how long the server is taking to respond to client requests. This can be calculated by taking the difference between the two timestamps in each audit record. This is a little more complicated arithmetic than the Universal Agent is capable of, so a custom script needs to be written to do the calculations.

Let's say that ldapaudit.awk is the awk script to do this task and the result of running the ldapaudit.awk script against the bind and search examples are:

The data needs to be passed to the socket data provider, which is listening on port 7500. There are several different ways of associating the passed data with the correct attribute definition, but the most flexible way is to pass the name of the metafile when the connection is first made, then include the name of the application and attribute group on each line of the passed data, in the following form:

The following perl script (feedsock.pl) will take care of this for us, when invoked as "perl feedsock.pl -m tds.mdl -n TDS -g Audit_Times"

As evident from the code snippet, the Universal Agent will accept the above data in the attribute group Audit_Times, as defined below:

The above attribute group signifies that the socket data provider expects data in the form of values separated by spaces. The "ms" label after the response time is ignored. The "Z" type for the "Parameters" attribute says to assign the remainder of the line following the status value to it. In other words, the search parameters and the name of the entry being modified or deleted are collected in the attribute named Parameters.

Back to top

Setting up situations

A “situation” is a mechanism whereby the tool can flag conditions that need to be brought to the attention of administrators, for example, if response times are starting to get too high. When a situation fires, a warning icon reflecting the severity of the situation will be displayed next to the attribute group's name in the navigator. Situations are rolled up in the hierarchy of the navigator. Higher levels in the navigator will display the icon of the most severe situation, so one can immediately drill down to the worst problems first.

The IBM Tivoli Monitoring 6.1 User's Guide's information on “situations for event-based monitoring” describes how to set up situations. This article talks about setting up a situation for the Tivoli Directory Server response times.

You can create a similar situation for the cache hit ratios, as well as process memory utilization. If you see that your hit ratio is low while the cache is full, then it would potentially be useful to increase the size of the cache. If, however, the process size is getting close to any limit you've set, then you might not want to increase the cache size, and might even want to consider reducing the cache size to reduce the process size.

More tuning information is in the IBM Tivoli Directory Server Performance Tuning Guide, in the Resources section.

For information on more active notification of situations, such as sending a text message, see the IBM Tivoli Monitoring 6.1 User's Guide (search for "reflex automation").

Back to top

Prototypes

A directory server monitoring solution has been developed which will provide better insights on the above mentioned scripts and metafile. A white paper has been written that describes the uploaded solution. The complete package containing the latest version of the white paper along with the available scripts and metafiles is available on the Open Process Automation Library (OPAL). Search for "directory server using universal agent".

Back to top

Conclusion

This document provides the steps to set up a solution to monitor a Tivoli Directory Server environment, based upon best practices observed from successful customer deployments. Additionally, the instructions provide enough background to develop new views or to customize the offering to suit individual needs.

Resources

• Basic understanding of the IBM Tivoli Directory Server
  
  
• IBM Tivoli Directory Server v 6.0 Administration Guide
  
  
• IBM Tivoli Directory Server v 5.2 Administration Guide
  
  
• IBM Tivoli Directory Server v 6.0 Install and Configuration Guide
  
  
• IBM Tivoli Directory Server v 5.2 Install and Configuration Guide
  
  
• IBM Tivoli Directory Server v 6.0 Tuning Guide
  
  
• IBM Tivoli Directory Server v 5.2 Tuning Guide
  
  
• IBM Tivoli Universal Agent User's Guide
  
  
• IBM Tivoli Universal Agent Install and Configuration Guide
  
  
• IBM Tivoli Monitoring 6.1 User's Guide
  
  
• IBM Tivoli Monitoring 6.1 Install and Configuration Guide

About the authors

		Dave Bachmann is the performance lead for Tivoli Security Products. Based in Austin, Texas in the United States, he has been working on the performance of various distributed systems since coming to IBM in 1992. He has helped with the performance of IBM's Distributed Computing Environment (DCE), IBM Tivoli Directory Server (LDAP), IBM Tivoli Access Manager, IBM Public Key Infrastructure, IBM Risk Manager, IBM Tivoli Identity Manager, IBM Tivoli Federated Identity Manager, IBM Tivoli Directory Integrator and IBM Tivoli Security Operations Manager products. He received a B.S. at Iowa State, an M.S from The University of Michigan, and a Ph.D. from The University of Michigan, where he worked on performance modeling of distributed systems.

		Ramakrishna J Gorthi is a developer for the IBM Tivoli Directory Server, Pune center in India. He has six years of experience in the IT industry, all in IBM, with one year of experience in Level 2 Customer Support for the various versions of the IBM Tivoli Directory Server and the rest of the experience in IBM Tivoli Directory Server development and testing. He has authored the TDS IBM Redbook® titled Understanding LDAP. He has written developerWorks articles pertaining migration and distributed directory scenarios. He holds a degree in Computer Engineering from Pune Institute of Computer Technology, Pune (India). His areas of expertise include IBM Tivoli Directory Server from the Tivoli Security Products and DB2®.

		Amit Bhate is the development lead for IBM Tivoli Directory Server, Pune center in India. He has a Computer Engineering degree from University of Pune (India). He has over seven years experience in IBM and worked on various IBM products. He spent three years in the Level 3 Support team for the IBM Distributed File System product. He was involved in the development activities of IBM DFS 3.0 Client on Windows, Lotus Notes 7.0 and IBM Tivoli Directory Server 6.1.

Rate this page

Please take a moment to complete this form to help us better serve you. Did the information help you to achieve your goal? Yes No Don't know   Please provide us with comments to help improve this page:   How useful is the information? 1 2 3 4 5 Not useful Extremely useful

Share this.... Digg this story del.icio.us Slashdot it!

Back to top