General LDAP server performance considerations

Threads

The commThreads configuration option specifies the number of communication threads that handle requests from clients to the LDAP server. However, the primary role of each of these threads is to serve as a worker thread for processing client requests to the directory.

Each communication thread is shared among client connections and is used to process requests as they occur. Therefore, this option does not need to be set nearly as large as the expected number of concurrently connected clients.

Each communication thread requires some resources of its own, including low storage, a connection to Db2® (when TDBM or Db2-based GDBM is configured), and other system resources associated with threads. Therefore, you might want to avoid making this option larger than is needed.

Set the commThreads to approximately two times the number of processors that are running in your LPAR. However, this is a general rule depending upon the activity that your LDAP server experiences.

If most requests are search requests retrieved from storage in TDBM caches or Db2 buffer pools, then additional commThreads might not provide much benefit. However, if most requests to the directory require I/O wait time, then additional commThreads might allow more client requests to run concurrently.

Debug settings

Activating the LDAP server debug trace facility affects performance. If optimal performance is what you want, active debug only when necessary to capture diagnostic information.

Storage in the LDAP address space

The LDAP server generally requires a minimum of 32 MB to run in 31-bit mode, and 96 MB in 64-bit mode even with a minimal directory. This storage is required for maintaining server-wide information and for processing client requests.
Note: These are estimates only, and the need for storage can increase depending on the size of any LDBM directories configured, and the size of the caches.

LDAP server cache tuning

The LDAP server implements many caches to help reduce processing time and to avoid access to the database. These caches are beneficial when most accesses to the directory are read operations. Tuning these caches involves monitoring their effectiveness and adjusting their size to increase the percent hit rate.
Note: Increasing cache sizes might increase the amount of storage that is required by the server.

Some caches are invalidated by update activities. If this is a frequent occurrence, increasing the cache size might be of little or no benefit. If the cache hit rate is never any higher than zero for a particular cache, the cache can be disabled by setting its size to 0. However, even caches with seemingly low cache hit rates might provide some benefit, therefore, you should generally avoid disabling them unless close monitoring is done to ensure that they are not beneficial.

Most caches in the LDAP server are enabled by default, and the default sizes generally provide some benefit to most installations. However, many installations might benefit from additional tuning. The following approach can be used to evaluate the cache sizes:
  • Monitor the cache performance during typical workloads: You can use either the cn=monitor search or the operator console MODIFY command to retrieve current cache statistics. These are described later in this topic.
    Note: The monitor search must be used with a scope of subtree or one-level to retrieve the cache statistics because the caches are backend specific.
  • Examine the cache hit rate, the current number of entries, and the maximum allowed entries (configured size). Also, note the number of cache refreshes and the average size of the cache at refresh.
  • If the cache hit rate is well below 100% and the cache is frequently fully populated, consider increasing the cache size. Because this is a configuration option, you must change the server configuration file and restart the server to affect the change.

The following caches are implemented in the LDAP server:

ACL source cache
This cache holds information regarding ACL definitions within the database. Retrieval of information from this cache avoids database read operations when resolving access permissions. This cache is implemented in the TDBM and Db2 -based GDBM backends.
DN cache
This cache holds information related to the mapping of distinguished names between their raw form and their canonical form. Retrieval of information from this cache reduces processing required to locate entries in the database. This is a server-wide cache, and is implemented in the internal schema backend. To alter its setting from the default, adjust the dnCacheSize configuration option in the global section of the LDAP server configuration file.
DN to eid cache
This cache holds information related to the mapping of distinguished names in their canonical form and their entry identifier within the database. Retrieval of information from this cache avoids database read operations when locating entries within the database. This cache is implemented in the TDBM and Db2-based GDBM backends.
entry cache
This cache holds information contained within individual entries in the database. Retrieval of information from this cache avoids database read operations when processing entries within the database. This cache is implemented in the TDBM and Db2-based GDBM backends.
entry owner cache
This cache holds information regarding ACL definitions within the database. Retrieval of information from this cache avoids database read operations when resolving access permissions. This cache is implemented in the TDBM and Db2-based GDBM backends.
filter cache
This cache holds information related to the mapping of search request inputs and the result set. This cache is implemented in the TDBM, LDBM, CDBM, and GDBM backends. For TDBM and Db2-based GDBM, retrieval of information from this cache avoids database read operations when processing search requests. For LDBM, CDBM, and file-based GDBM, this cache helps reduce processing time for searches with complex filtering. Note that the GDBM filter cache is disabled, by default.

Operations monitor

If the operations monitor is enabled, the LDAP server monitors search statistics for the types of search patterns that are configured and stores search statistics for each search pattern. The operations monitor supports two types of search patterns, searchStats and searchIPStats. A searchStats pattern consists of the search parameters (search base, scope, filter, and attributes to be returned) and status (success or failure). The searchStats pattern is useful for evaluating the performance of search patterns. A searchIPStats pattern consists of the same elements as searchStats pattern does, but also includes the client IP address. The searchIPStats pattern is useful in determining if there are any specific clients spamming the LDAP server. The operationsMonitor configuration option determines which types of search patterns are monitored. See Monitoring performance with cn=monitor for more information about the operations monitor.

A new search pattern is added to the operations monitor whenever the search pattern of an incoming search does not match one of the existing operations monitor search patterns. When the number of search patterns exceeds the value of the operationsMonitorSize configuration option (the cachesize attribute in the cn=operations,cn=monitor entry), the least recently used search patterns are trimmed. The total number of trimmed search patterns is stored in the numtrimmed attribute of the cn=operations,cn=monitor entry. Typically, trimmed search patterns are not a cause for concern because they are infrequently executed search patterns. If there is a high volume of trimmed data, you should consider increasing the value of the operationsMonitorSize configuration option or possibly monitoring only searchStats patterns. Note that searchIPStats search patterns produce more search patterns than searchStats patterns because searchIPStats creates a new search pattern for each unique client IP address even if the rest of the search pattern is the same. See Table 4 for more information about the cn=operations,cn=monitor entry and its attributes.

Workload manager (WLM)

The z/OS® LDAP server supports Workload Manager (WLM) to allow an installation to set performance goals for work within the LDAP server when compared against other work that is running on the system. The LDAP subsystem type is reserved in WLM to allow the system administrator to set performance goals for z/OS LDAP server operations. The performance goals can be set based on the IP address of the client, distinguished name (DN) of the bound user, both the IP address and DN associated with the request, or a request matching a search pattern in the operations monitor.

The z/OS LDAP WLM support is always active and cannot be deactivated. A default service class must be configured in the WLM ISPF panels for the LDAP subsystem type before running the z/OS LDAP server. If a default service class is not configured in WLM, all LDAP server operations run under the discretionary or SYSOTHER profile and receive a low priority, which impacts LDAP server performance and response times. See z/OS MVS Planning: Workload Management for more information about WLM. See Verifying the service class for the LDAP server for information about determining the service class for the LDAP servers that are running on your system.

The LDAP server uses the WLM health service to indicate a health value to WLM. The WLM health value is used by the TCP/IP sysplex distributor to help route incoming client requests to servers within the sysplex. After the LDAP server initialization, the WLM health value is set to 100%. The WLM health value is calculated by the number of failures during the past 5000 operations if one minute has passed since the value was last calculated. If the percentage of failures changes by 25% or more, the z/OS LDAP server increases or decreases the WLM health value. An LDAP server operation is considered a failure when it has one of the following return codes:
  • LDAP_OPERATIONS_ERROR (1)
  • LDAP_TIMELIMIT_EXCEEDED (3)
  • LDAP_ADMIN_LIMIT_EXCEEDED (11)
  • LDAP_BUSY (51)
  • LDAP_UNAVAILABLE (52)
  • LDAP_UNWILLING_TO_PERFORM (53)
  • LDAP_OTHER (80)

The wlmExcept configuration option can be used to specify the IP address of the client, the distinguished name (DN) of the bound user, or both to route those requests to any configured WLM transaction name (TN). The wlmExcept configuration option can be specified multiple times within the LDAP server configuration file to allow multiple client IP addresses or bound users' DNs to be associated with the same or different WLM transaction name. The order that the wlmExcept configuration options are specified in the LDAP server configuration file determines the order the LDAP server uses to match incoming client requests and route them to the WLM transaction name. See wlmExcept name [IP_address] [dn] for more information about the wlmExcept configuration option.

The WLMEXCEPT operator modify command can be used to change the routing of incoming client requests to new or different WLM transaction names while the server is running. If the operations monitor is configured, the cn=operations,cn=monitor entry has searchStats and searchIPStats attribute values with an ID parameter that indicates the operations monitor ID (OPID). See Monitoring performance with cn=monitor for details about the searchStats and searchIPStats attribute value format. The WLMEXCEPT operator modify command uses the OPID to associate a search pattern to a WLM transaction name. If the transaction name specified on the WLMEXCEPT operator modify command does not exist in WLM, a new WLM enclave is created; otherwise an existing enclave is used. Each time the WLMEXCEPT operator modify command is issued, the new mappings are added before any of the configured wlmExcept configuration options or previously issued WLMEXCEPT operator modify commands. See LDAP server operator commands for more information about the WLMEXCEPT operator modify command.

The WLMEXCEPT operator modify commands last for the life of the LDAP server, however, the RESET WLMEXCEPT can be issued to remove all previously issued WLMEXCEPT operator modify commands and default to using the initial LDAP server configuration. If the operations monitor ID (OPID) is specified on the RESET WLMEXCEPT operator modify command, then just that specific WLM routing for that search is removed. See LDAP server operator commands for more information about the RESET WLMEXCEPT operator modify command.

If the operations monitor is enabled, the searchStats and searchIPStats attributes in the cn=operations,cn=monitor entry can be used to identify spamming client applications or certain search requests that should have a higher priority within the LDAP server. This type of information is very valuable when configuring LDAP to use WLM transaction names and assigning service or report classes to those transaction names. For a spamming client application, a WLM transaction name with a low priority service or report class ought to be used. For important search requests, a WLM transaction name with a high priority service or report class ought to be used.

Configuring LDAP with WLM examples

Assume that WLM has been configured to contain the following transaction names and service classes for LDAP.

Figure 1. WLM ISPF rules classification panel 
Subsystem-Type  Xref  Notes  Options  Help
--------------------------------------------------------------------------   
                  Modify Rules for the Subsystem Type       Row 1 to 3 of 3   
 Command ===> ___________________________________________ Scroll ===> PAGE    
                                                                              
 Subsystem Type . : LDAP        Fold qualifier names?   Y  (Y or N)           
 Description  . . . Use Modify to enter YOUR rules                            
                                                                              
 Action codes:   A=After     C=Copy        M=Move     I=Insert rule           
                 B=Before    D=Delete row  R=Repeat   IS=Insert Sub-rule      
                                                              More ===>       
           --------Qualifier--------               -------Class--------       
 Action    Type       Name     Start                Service     Report        
                                          DEFAULTS: HIGHREQ     ________      
  ____  1  TN         GENERAL  ___                  NORMREQ     ________         
  ____  1  TN         EXCEPT1  ___                  SPAMREQ     ________         
  ____  1  TN         EXCEPT2  ___                  CRITREQ     ________         
****************************** BOTTOM OF DATA ******************************  
                                                                              
                                                                              
  F1=Help    F2=Split   F3=Exit    F4=Return  F7=Up      F8=Down    F9=Swap   
 F10=Left   F11=Right  F12=Cancel

Example 1:

After analyzing the searchStats and searchIPStats attributes returned on a cn=operations,cn=monitor search, it has been determined there is a spamming LDAP client application on IP address 1.2.3.4 that has been affecting performance of the LDAP server. Also, requests from bound user cn=importantguy,o=ibm should have a higher priority within the LDAP server.

An LDAP administrator can add the following wlmExcept configuration options to route these requests to the appropriate WLM transaction name in Figure 1:
wlmExcept EXCEPT1 1.2.3.4
wlmExcept EXCEPT2 cn=importantguy,o=ibm

After the LDAP server is restarted, LDAP client requests originating from IP address 1.2.3.4 are routed to WLM transaction name EXCEPT1 with a service class of SPAMREQ. The server routes requests from bound user cn=importantguy,o=ibm to WLM transaction name EXCEPT2 with a service class of CRITREQ. If there are requests from any other client IP addresses or bound users, the server routes them to the GENERAL WLM transaction name that has a service class of NORMREQ. See z/OS MVS Planning: Workload Management for more information about configuring WLM.

If the transaction name specified on the wlmExcept configuration option or on the WLMEXCEPT operator modify command does not exist in WLM (such as EXCEPT3 does not exist in Figure 1), any client requests associated with that transaction name would use the default service class HIGHREQ.

Example 2:

After analyzing the searchStats and searchIPStats attribute values returned on a cn=operations,cn=monitor search, it has been determined that the search identified by a specific searchIPStats value should be mapped to WLM transaction name EXCEPT2 because it is an important LDAP search.
cn: cn=monitor,cn=operations
searchIPStats: ldap://fe00::f4f7:0:0:7442:750f/OU=_v,O=_v??sub?(|(&(sn=_v)(cn=_v*))
 (description=*_v*))?success,numOps=42,avg=246,rate=5,maxRate=37,maxRat 
 eTimeStamp=20130524132626.545031Z,createTimeStamp=20130524132615.953823Z,
 ID=2741
The following WLMEXCEPT operator modify command is used to route the search pattern in the searchIPStats attribute to the EXCEPT2 WLM transaction name:
F LDAPSRV,WLMEXCEPT EXCEPT2,2741
Note: The operations monitor ID (OPID) value is 2741 for the search on the searchIPStats attribute value.
If it is determined later that the client search requests identified by OPID 2741 no longer need to be routed to WLM transaction name EXCEPT2, the following RESET WLMEXCEPT operator modify command is used to remove just the specific WLM routing for that search:
F LDAPSRV,RESET WLMEXCEPT,2741

See LDAP server operator commands for more information about the RESET WLMEXCEPT operator modify command.

Verifying the service class for the LDAP server

If you are having performance problems with the LDAP server, verify that the LDAP server is running with the correct service class in WLM. If a default service class is not configured in WLM, all LDAP server operations that are run under the discretionary or SYSOTHER profile and receive a low priority, which impacts LDAP server performance and response times.

Use the ISPF SDSF menu and specify ENC (enclaves) to verify the LDAP server service class. Figure 2 depicts the LDAP server is running under the SYSOTHER service class, which results in LDAP operations receiving a low priority.
Figure 2. SDSF enclave display example
SDSF ENCLAVE DISPLAY  DCEIMGVD ALL                     LINE 1-6 (6)            
 COMMAND INPUT ===>                                            SCROLL ===> PAGE 
 NP   NAME             SSType Status   SrvClass Per PGN RptClass ResGroup   CPU-
      3400000008       LDAP   ACTIVE   SYSOTHER   1                             
      2400000002       STC    INACTIVE SYSSTC     1                             
      2800000003       STC    INACTIVE SYSSTC     1                             
      3000000005       STC    INACTIVE SYSSTC     1                             
      2000000001       STC    INACTIVE SYSTEM     1                             
      2C00000004       TCP    INACTIVE SYSOTHER   1