IBM Tivoli Directory Server - SMS to DMS migration

Performance issues with DB2 online backup

This article contains the results of performance tests on IBM® Tivoli® Directory Server (ITDS) running on SMS and DMS during DB2® online backup, recommendations based on the results as well as the steps required for migrating ITDS from SMS to DMS. This content is part of the IBM WebSphere Developer Technical Journal.

Share:

Julius E. Enarusai, Senior Managing Consultant, IBM

Julius EnarusaiJulius Enarusai is a Senior Managing Consultant for the Tivoli Services division of IBM. He has been involved in IBM Tivoli Directory Server (ITDS) product design, implementation and support. Julius is also involved in IBM Tivoli Directory Integrator (ITDI) engagements.



John D. McGarvey, Tivoli Architect, IBM Software Directory, IBM

John McGarveyJohn McGarvey is a Senior Technical Staff Member for Tivoli Security Products, and is the product architect for Tivoli Directory Server.



Eric R. Storch, IT Specialist, IBM

Eric StorchEric Storch works in the IBM Software Services for Tivoli and is a consultant and subject matter expert in ITDS and TCIM.



Yiming Y. Chen, Information Management - UDB Consultant, IBM

Yiming ChenYiming Chen has over 13 years experience on distributed database systems and over 8 years as a database consultant in North America. Currently specialized in DB2 UDB, HADR and DB2 High Availability solutions on single partition, DPF and BCU systems.



Chehayeb R. Chehayeb, Senior IT Specialist, IBM

Chehayeb ChehayebRamy Chehayeb is a Senior IT Specialist in Montreal, Canada. He has been involved with several Tivoli product implementations ranging from Systems management to Security solutions.



19 January 2009

Special acknowledgements

  • Lorraine Krukowski (Project Manager, IBM Global Technology Services) - The team would like to give special thanks to Lorraine for her efforts in putting the team together and working with the customer to provide the environment that was used in performing the tests.

Introduction

For most mission critical IBM Tivoli Directory Server (ITDS) deployments, customers rely on DB2 online backup utilities to take frequent backups while the system is up and running. DB2 online backup utilities provide the added advantage of not requiring stopping ITDS during the backup process. The online backup process works very well for systems that do not contain large binary objects (LOB). However, when an ITDS directory uses DB2 Systems Managed Storage (SMS) and includes LOB data, the online backup process may block access to the database for a period ranging from 72 seconds to 15 minutes, depending on the size of the database. All LDAP transactions may be blocked during this period of time until the DB2 online backup process completes. This lockout is definitely unacceptable as it may violate the customer's service level agreements. By default, when an ITDS DB2 instance is created, the resulting table spaces use SMS as opposed to database managed storage (DMS) table space containers. The lock management scheme used by DB2 for SMS table spaces in both versions 8 and 9 for LOB data have been known to cause lockouts. This problem does not occur with database DMS. In addition, DMS has several advantages over SMS, including:

  • In general, a well-tuned set of DMS table spaces will outperform SMS table spaces. The difference is small on most platforms, but large on others, especially if LOB data is present.
  • The size of a table space can be increased by adding or extending containers, using the ALTER TABLESPACE statement and existing data can be automatically rebalanced across the new set of containers to retain optimal I/O efficiency.
  • A table can be split across multiple table spaces, based on the type of data being stored:
    • Long field and LOB data
    • Indexes
    • Regular table data
  • The location of the data on the disk can be controlled, if this is allowed by the operating system
  • The backup procedure itself is often much faster when using DMS

Determine if LOB data is present

LOB data can be present in Tivoli Directory Server when binary objects like JPEG photographs or PKI certificates or XML documents are stored in the directory, or when some entries in the directory are large. Before using online backup with directory instances and SMS, it is best to check whether LOB data is present. (Offline backup can always be used, but may not suit the requirements of the deployment.) To check where LOB data is present, take a snapshot of the database as follows:

Listing 1. Determining if LOB data exists
		      $db2 connect to db2inst1
		      $db2 update monitor switches using table on
		      $db2 get snapshot for all on db2inst1 > snap.out
		      $db2 connect reset

In the snapshot, there will be several lines for LOB Object Pages. If these lines are absent, or if only one page is present for a given table, there is no LOB data in the directory. Otherwise, online backup should not be used if the directory uses SMS tablespaces.


Other kinds of migration

When an instance of Tivoli Directory Server is migrated to a new machine or a new set of disks, it is usually sufficient to back up the database and then restore it on the new machine or disks. However, there are cases where this approach will not work. If one of the machines is Windows and the other is Linux or Unix, the backup/restore procedure can't be used. If one of the machines is Linux on Intel, and the other is Linux on power, backup/restore won't work. Finally, if the source machine is using SMS, and the target machine is running DMS -- or conversely -- backup/restore won't work. The migration procedure described in this document can be used for each of these cases, and not only for moving from SMS to DMS.


SMS versus DMS - Test Results

The following sections contain details of the test setup and summary results of the actual test runs.

Test Setup

The test environment consisted of three ITDS servers running on IBM pSeries® 510 hardware with 8 Gigabytes of memory. These machines all contained an extra set of local disk drives that were initially intended to be used as mirrors. Since these spare drives were available, the team decided to use them as storage for the DMS database without disrupting the existing data. New file systems were created on the spare drives. Using Perl performance tools created for driving massive amounts of transactions, approximately 150 thousand test entries were created and added to the directories. These scripts simulated a very high volume customer environment with thousands of transactions with multiple threads. The number of threads is configurable and the number was adjusted up to 1000 threads in some of the tests. Peer to peer replication was also enabled, similar to most high volume customer deployments. The following is a summary of the three servers:

  • M1 - This server was not actually used for the test. However, since replication was enabled, all modifications made to M2 and M3 were replicated to this server.
  • M2 - This was the server where most of the tests took place. It was initially used as the benchmark server. Several test runs were performed on it to collect baseline data while running on SMS. The SMS to DMS migration was later performed and another set of tests were run under DMS. The test results contained in the next section are primarily based on the tests run on this server.
  • M3 - This was the server where all the initial SMS to DMS conversion scripts were developed and tested. The performance test results from this server did not compare very well to the results received from M2. Unfortunately, no benchmark tests were run on this server before it was converted to DMS. As a result, we have no way of knowing why the performance results were not as good as those from M2.

Test data

The following tables contain a summary of the test results collected from M2 during the comparisons between SMS and DMS in the development environment.

Benchmark System (M2 running on SMS) - Baseline without online backup running
Test RunNumber of Registrations (per min)Minimum Registration Time (in seconds)Maximum Registration Time (in Seconds)Average Registration Time (in seconds)Remarks
Test 1400558275Note: One registration transaction equals 1 LDAP add, 9 LDAP modifies, 25 LDAP binds and 36 LDAP search operations
Test 2401417961
Test 3414385844.2
Test 4388396247.9
Average400.7543.2570.2557.025
Benchmark System (M2 running on SMS) - Baseline with online backup running
Test RunNumber of Registrations (per min)Minimum Registration Time (in seconds)Maximum Registration Time (in Seconds)Average Registration Time (in seconds)Remarks
60.75328395374Note: With SMS, the systems freezes consistently during the backup, starting between 58% and 69% and finally frees at about 84% to 90%
M2 converted to DMS) - without online backup running
Test RunNumber of Registrations (per min)Minimum Registration Time (in seconds)Maximum Registration Time (in Seconds)Average Registration Time (in seconds)Remarks
Test 1286688476Note: One registration transaction equals 1 LDAP add, 9 LDAP modifies, 25 LDAP binds and 36 LDAP search operations
Test 2616383938
Test 3616383938
Test 4626384842
Average53845.552.548.5
M2 converted to DMS) - With online backup running
Test RunNumber of Registrations (per min)Minimum Registration Time (in seconds)Maximum Registration Time (in Seconds)Average Registration Time (in seconds)Remarks
240100125115Note: With DMS, no lockouts were detected and the backup completes much faster at a rate of almost 10:1

Test summary

The results of the tests indicate the following:

  • Using the DB2 db2pd command, the team clearly saw multiple lock-outs starting at about 58% of an online backup under SMS.
  • No lock-outs were detected when running the same online backup under DMS.
  • The backup process was much faster, at almost 10:1 ratio.
  • Overall performance was much faster. The test system appeared to be performing even better than customer's production system, with half the CPU power, half the memory and no SAN storage.
  • Start-up time was also much faster. Under SMS, the ibmslapd process was taking about 5 minutes to start. When the same system was converted to DMS, it only took a few seconds.

Recommendation

Based on the results summarized in the preceding section, it is recommended that customers experiencing similar lockouts during online backups of ITDS (when LOB data is present) should consider migration of their DB2 table spaces from SMS to DMS. The remainder of this document contains a high level summary of the steps necessary for converting ITDS DB2 table spaces from SMS to DMS.


Migration Procedure

The process for migrating a SMS database to DMS involves multiple steps, including exporting the existing data, dropping the table spaces, recreating table spaces and finally importing the data.

Migrating first ITDS master

Before starting the process, please note that customer specific DB2 maintenance operational procedures (if any) must be following when performing the tasks described in this section. Once the steps described in the operational procedures document have been completed, the following tasks need to be performed in the order specified.

Exporting the data and dropping the table spaces

The following steps need to be performed to convert the first ITDS master server:

  1. Stop the ITDS server, using the following command:
	# ibmslapd -I db2inst1 -k
  1. Take an offline backup using the following commands:
	# su - db2inst1
	$ db2 backup database db2inst1
	$ exit
  1. Create an export directory using the following commands while logged in as root:
	# cd /
	# mkdir export
	# chmod 777 export
  1. Export the database to the export directory and drop the table spaces using the following commands:
	# su - db2inst1
	$ cd /export_fs/export
	$ db2move db2inst1 export -tc * -tn * -sn * -ts *
	$ db2 connect to db2inst1
	$ db2 drop tablespace userspace1
	$ db2 drop tablespace ldapspace

Note

All tables residing in these two table spaces are implicitly dropped with the drop tablespace command.

  1. After dropping the table spaces, you can use the following command to examine any tables remaining in the database:
	$ db2 list tables for all
	$ db2 connect reset
	$ exit

Creating the DMS table spaces

The following steps must be executed sequentially to create the DMS table spaces:

  1. Create dms01 and dms02 subdirectories under /db2data. These are the directories where the new DMS table spaces will be created. These directories must be owned by user db2inst1 and group db2iadm1:
	# mkdir /db2data/dms01
	# mkdir /db2data/dms02
	# chown db2inst1:db2iadm1 /db2data/dms01
	# chown db2inst1:db2iadm1 /db2data/dms02
  1. Recreate the table spaces using the create_dms_tablespace SQL script (described in the resources section) while still logged in as the DB2 instance owner. To run the SQL script, type the following command:
	$ db2 -tvf create_dms_tablespace.sql
  1. The table spaces created using above script have the following characteristic:
    • Use file containers, and have two containers per table space
    • No file system caching
    • Table space page sizes are the same as they are in SMS table spaces
    • Table space disk overhead and transfer rate are set as below (numbers are based on I/O throughput sampling data from the customer's production environment)
      • OVERHEAD=4.18
      • TRANSFERRATE=0.10 for the 4 KB pagesize tablespace, USERSPACE1
      • TRANSFERRATE=0.69 for 32 KB pagesize tablespace, LDAPSPACE
    • Automatic table space resize is turned on based on following space growth estimate:
      • USERSPACE1: 4.7 KB/entry * 10000 new entries/day = 46 MB
      • LDAPSPACE: 35.1 KB/entry * 10000 new entires/day = 343 MB
    • Initial table space size is based on 2,886,985 user profiles in customer test system
      • USERSPACE1: 13,174 MB
      • LDAPSPACE: 98,976 MB
    • The increase size can be changed by using the alter tablespace command. The following is a command to change the increase size to 100 MB:
	$ db2 alter tablespace LDAPSPACE INCREASESIZE 100 M

A word of caution

An additional step may be necessary if the directory is very large. DMS regular tablespaces are limited in size to 16M pages. USERSPACE1 uses 4K pages, so that no more than 64M of data can be stored in USERSPACE1 using a DMS regular tablespace. LDAPSPACE uses 32K pages, so that no more than 496M of data could be stored in it using a DMS regular tablespace. There is a solution. Beginning in DB2 9.1, DMS capabilities have been extended so that one can create a DMS large tablespace, which can be used to store up to 2 terabytes of data. It is possible to convert DMS regular tablespaces to DMS large tablespaces as the directory grows.

Note also that there are actually two different kinds of DMS storage, which are called "cooked" and "raw". DMS cooked storage uses the file system and this is what was used for the testing in this paper. DMS raw storage uses no file system, and reads and writes go directly to the disk. When DMS raw storage is specified, one must indicate the name of the virtual disk to be used, and the cylinders to be used, on the CREATE TABLESPACE command. No file system should be present. DMS raw tablespaces can be faster, especially on Solaris, but because the file system is not present, they do require more skill to manage. See the DB2 product documentation on CREATE TABLESPACE for information about DMS tablespaces.

Importing the data

Import the tables and data, using the following commands:

	$ cd /db2data/export
	$ db2move db2inst1 import
	$ exit
	# ibmslapd -I db2inst1 -n

Migrating second or third ITDS instances

Once one of the ITDS servers has been converted to DMS, the process for converting the second and any additional servers is a lot simpler. The most time consuming operation in the conversion process, which is the import of the data, is no longer required. Instead, a backup of the first DMS database can be taken and restored to the second server after dropping the table spaces and recreating the table spaces using DMS. The following are the required steps for converting the second and third servers to DMS:

  1. Stop the ITDS server, using the following command
	# ibmslapd -I db2inst1 -k
  1. Take an offline backup using the following commands:
	# su - db2inst1
	$ db2 backup database db2inst1
	$ exit
  1. Drop the table spaces using the following commands:
	# su - db2inst1
	$ db2 connect to db2inst1
	$ db2 drop tablespace userspace1
	$ db2 drop tablespace ldapspace
  1. After dropping the table spaces, you can use the following command to examine any tables remaining in the database:
	$ db2 list tables for all
	$ db2 connect reset
	$ exit
  1. Create dms01 and dms02 subdirectories under /db2data. These are the directories where the new DMS table spaces will be created. These directories must be owned by user db2inst1 and group db2iadm1, using the following commands:
	# mkdir /db2data/dms01
	# mkdir /db2data/dms02
	# chown db2inst1:db2iadm1 /db2data/dms01
	# chown db2inst1:db2iadm1 /db2data/dms02
  1. Recreate the table spaces using the following SQL script and command:
	$ db2 -tvf create_dms_tablespace.sql
  1. Stop M3 and take an offline backup using the db2bkupOffline.sh script. Please see Resources section for detailed instructions on the use of the script
  2. Run the db2restOffline.sh script to restore the database from M1 to M2. Please note that customer specific maintenance procedures (if any) for taking a master server offline must be followed when performing the conversion.
  3. 10. Repeat the above steps to convert M3 to DMS. It is highly recommended that the conversion for M1 should be performed during a scheduled maintenance window.

DMS table space monitoring and management

The procedures described in the preceding sections are setup for DB2 to automatically resize both USERSPACE1 and LDAPSPACE table spaces when they run out of space. Table-space resizing is fast and efficient. However, database response time can still be impacted during the resize operation. In order to minimize the chance of any unscheduled table space resizing, particularly during peak business hours, the following table space managing strategy should be considered:

  • Monitor the tablespace utilization routinely. The "db2pd" command can be used to check the space usage of a DMS tablespace. The following are sample commands for checking and printing the USERSPACE1 and LDAPSPACE space utilization:
	$ tbsusage=`db2pd -db db2inst1 -table spaces|grep USERSPACE1 |
		awk '{used=$16; totals=$15; printf "%.2f%\n", used/totals*100}'`
	$ echo "Tablespace USERSPACE1 usage: $tbsusage"

	Tablespace USERSPACE1 usage: 94.77%

	$ tbsusage=`db2pd -db db2inst1 -table spaces|grep LDAPSPACE |
	awk '{used=$16; totals=$15; printf "%.2f%\n", used/totals*100}'`
	$ echo "Tablespace LDAPSPACE1 usage: $tbsusage"

	Tablespace LDAPSPACE1 usage: 94.39%
  • If the utilization has reached the threshold (e.g. 60%), resize the table space manually in a system maintenance window. The following are sample commands that resize all containers of USERSPACE1 to 300,000 pages.
	$ db2 connect to db2inst1
	$ db2 "alter tablespace userspace1 resize (ALL 300000)"
	$ db2 connect reset

	The following are sample commands that resize all
	containers of LDAPSPACE to 50,000 pages.
	$ db2 connect to db2inst1
	$ db2 "alter tablespace ldapspace resize (ALL 50000)"
	$ db2 connect reset

Conclusion

Based on the results summarized in this document, it is clear that an ITDS system using DMS table space containers can be backed up using DB2 online backup without any lockouts. The results also show that DMS has better performance over SMS. The steps outlined in this document can be used by those customers who currently deploy ITDS with SMS table space containers with LOB data to migrate the DB2 table space containers to DMS. The procedures can also be used in other scenarios where normal backup/restore procedures cannot be used (e.g. when restoring to a different operating system).


Download

DescriptionNameSize
Sample DB2 scripts for this articlesample-db2-scripts.tar10KB

Resources

Learn

Get products and technologies

  • Download IBM product evaluation versions and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Tivoli (service management) on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Tivoli
ArticleID=360307
ArticleTitle=IBM Tivoli Directory Server - SMS to DMS migration
publish-date=01192009