Level: Intermediate Nirmala Venkatraman, Performance Architect, IBM
Serge Limoges, Senior Technical Staff, IBM
25 Jul 2006 Find performance data about the IBM Lotus Domino with DB2 feature to help you plan your Lotus Domino messaging environment. This article shows results of performance tests of this Lotus Domino 7 feature using the R6Mail workload.
[Editor’s note: The results presented in this article are from benchmarks executed in a controlled environment. While some effort was made during the creation of the benchmark to include typical user operations, real-world users may make different use of Lotus Domino than the narrow range of functions tested by the benchmark. Therefore, these numbers should be used primarily to understand the relative performance of the Lotus Domino releases. They do not represent recommendations for real-world deployment. For assistance with capacity planning, consult your hardware vendor.]
This is the first IBM Lotus performance article on IBM Lotus Domino 7 with IBM DB2 storage (versus traditional NSF storage) for user mail files. This article describes IBM DB2 Universal Database (UDB) for Linux, UNIX, and Microsoft Windows as measured with Lotus Domino 7.0.1 and IBM DB2 V8.2 Fix Pack 9 (FP9). Lotus Domino with DB2 storage was first available as a limited release for customers in Lotus Domino 7.0. This article focuses on performance benchmark results that simulate Notes mail users by using the R6Mail workload (see the side file, "R6Mail workload description" for more information about this workload).
This article shows how Lotus Domino with DB2 storage running locally on the same hardware configuration scales up to 15,000 Notes users (simulated) on configurations in which there are adequate resources available. It shows the benchmark results obtained on a four-way IBM pSeries server, using a single Domino partition and a 64-bit DB2 instance, running on AIX 5.3.
The primary goal of these tests is to demonstrate the scalability of the Domino mail server with DB2 storage for user mail files that are large enough to represent a typical enterprise mail deployment. Because DB2 server can scale up to several terabytes of data, a starting mail file size of 240 MB was used for each user, compared with the 20 MB starting mail file size in our previous Lotus Domino 7 performance articles. All NotesBench audited reports use a starting mail file size of approximately 20 MB to ease storage space requirements during benchmark runs.
NOTE: The intent of this article is to focus on the performance of Lotus Domino with IBM DB2 storage. This data should not be compared with Lotus Domino and traditional NSF storage.
All the results represent sub-second response times from Lotus Domino. For benchmarking purposes, only the router task is run (except where noted) to avoid spikes in the data from other activity. We hope that you find the information useful and that you gain an understanding of the improvements that have gone into Lotus Domino 7 with DB2 storage.
Server configuration
With 9 TB of storage configured, DB2 can support 15,000 user mail files. Because CPU is not a constraint for this test, we disabled four CPUs on the logical partition (LPAR), using the bindprocessor command at the IBM AIX operating system level.
The AIX server is also configured with a 32 GB swap or paging space on one of the internal disk volumes. Table 1 shows the configuration details of the system tested.
Table 1. Details of the AIX system tested
| Model | P560Q | | Processor type/speed | System has eight physical Power5 CPUs with a clock speed of 1.5 GHz and is set up for a single LPAR. The LPAR used for these tests is configured with four CPUs enabled. | | Installed memory | 32 GB of RAM | | Active physical drives |
- IBM FastT DS4800 with nine drawers filled with 72G 15rpm drives (each drawer is also a logical volume)
- Five internal disks
| | Active logical volumes |
- Nine RAID 0 volumes on the FastT for DB2 data, which contains the DOMINO database, storing all user mail files and the DB2 database transaction log. The first logical volume also has a Domino data directory containing mail.box files and names.nsf (JFS2).
- One internal logical volume for Domino transaction logging (when used)
- One internal logical volume for Domino executables
- One internal logical volume for DB2 executables
- One internal logical volume for holding all test data and results
- One logical volume for the operating system
| | Operating system | IBM AIX 5.3 |
We created DB2 UDB V8.2 FP9 with a 64-bit instance that handles the Domino mail storage for 15,000 users. The entire Domino mail data resides in a single database called DOMINO in the DB2 directory. We tuned several database manager and DOMINO database configuration parameters for optimal performance as listed in table 2.
Table 1. Details of the AIX system tested
| Database manager configuration | DIAGLEVEL = 1
SHEAPTHRES = 20000
MAXAGENTS = 400
All default database monitor switches turned OFF. | | DOMINO database configuration | DBHEAP = 102400
CATALOGCACHE_SZ = 25000
LOGBUFSZ = 256
UTIL_HEAP_SZ = 5000
LOCKLIST = 4096
SHEAPTHRES_SHR = 8192
SORTHEAP = 1024
STMTHEAP = 8192
APPLHEAPSZ = 1024
PCKCACHESZ = 100000
MAXLOCKS = 75
CHNGPGS_THRESH = 50
NUM_IOCLEANERS = 16
NUM_IOSERVERS = 3
MAXAPPLS = 300
MAXFILOP = 1900
LOGFILSIZ = 16384
LOGPRIMARY = 40
LOGSECOND = 5
SOFTMAX = 240 | | Db2set variables | DB2_ASYNC_IO_MAXFILOP=10000
DB2_HISTORY_FILTER=T
DB2_USE_ALTERNATE_PAGE_CLEANING=YES | | BufferPool Settings | 1.6GB Bufferpool for DOMINO database |
The DOMINO database in DB2 is set up with the default 10:1 grouping, which means that 10 user mail databases reside in one tablespace in the DB2 database. With each user mail file starting at an initial 240 MB size, each tablespace in DB2 is approximately 2.4 GB in size, and there are 1,500 tablespaces in the nine logical RAID 0 volumes (configured as JFS2 filesystems).
Lotus Domino 7.0.1 is installed on the LPAR under its own UNIX account (which is part of the DB2 admin group) and is set up to work with DB2 as described in the Lotus Domino Administrator help. We set up an additional 32-bit DB2 instance under this account for the Domino server’s CLI access, but this 32-bit instance is not started during the test. We also tuned many of the server's Notes.ini settings to optimize Domino server performance.
Notes.ini settings
The INI settings with a DB2 prefix are primarily for Lotus Domino with DB2 storage. The other INI settings apply to NSF storage. The reason that NSF_BUFFER_POOL_SIZE_MB is set low is that with a DB2 backend, DB2 is buffering all the mail database data, so Lotus Domino doesn’t require a big bufferpool that caches the mail database data in memory.
NSF_Buffer_Pool_Size_MB=210
Server_Pool_Tasks=100
Server_Max_Concurrent_Trans=100
NSF_DbCache_MaxEntries=15000
ServerTasks=Router,LDAP
DB2DMS_CREATE_SIZE=400
DB2DMS_EXTEND_SIZE=256
DEBUG_DB2SEARCH_CTL=00030005
DB2NIF_POOLSIZE_MB=320
DB2POOLRESIZEMAX=250 |
Performance data for Lotus Domino with DB2 storage
When running the R6Mail benchmark using NotesBench, we found that Lotus Domino 7 with DB2 storage can support 15,000 users on a four-way AIX server. Figure 1 shows the performance characteristics of the server as a function of load (simulated users).
Figure 1. Server performance as a function of load
As figure 1 shows, Lotus Domino with DB2 storage scales linearly with the number of simulated R6Mail users up to a maximum of 15,000 simulated users tested in this configuration. There were no hardware resource constraints on this server, except the disk-space limitation for storing the larger user mail files.
Because we used a 64-bit DB2 instance in this configuration, we tuned several memory-pool and heap sizes to take advantage of the 32 GB of physical RAM; specifically, the bufferpool for the DOMINO database in DB2 was increased to 1.6 GB to cache more data. At 15,000 R6Mail users, we see an 80 percent bufferpool hit rate on the DB2 server. Also, because Lotus Domino generates many different unique SQL statements for accessing data in DB2, we increased the package cache specified by database configuration parameter PCKCACHESZ to 400 MB. At 15,000 R6Mail users, the DB2 server is executing an average of 160,000 SQL statements per minute, including the SELECT, INSERT, DELETE, and UPDATE statements.
Lotus Domino uses connection pooling for all SQL requests against DB2. To limit the maximum concurrent applications that can be run on the DB2 server, we set MAXAPPLS to 300. This controls the resources, such as the total application heap, consumed by the concurrent applications on the DB2 server. It is still enough to support the high concurrency of DB2 requests at 15,000 R6Mail users. At 15,000 R6Mail users, we see approximately 240 concurrent DB2 agent processes running on the server. This parameter must be tuned in production environments based on concurrent DB2 requests. We found that 150-200 is a good starting value for database configuration parameter MAXAPPLS for Lotus Domino with DB2 storage.
The Server_Pool_Tasks and Server_Max_Concurrent_Trans values in the Notes.ini are set to support the high end-user numbers. Before changing these settings from their default values, however, it is recommended that you analyze them to optimize the values used.
Summary
Currently, Lotus Domino 7 with traditional NSF storage supports 15,000 concurrent Notes mail users using the R6Mail workloads on almost all server platforms.
For Lotus Domino 7, we worked extensively to provide a scalable Domino server with DB2 storage for storing Domino data – both mail files and application data. We demonstrated that Lotus Domino 7.0.1 with DB2 storage for mail environments can easily support 15,000 concurrent Notes mail users using the R6Mail workload, provided the server has enough CPU and memory resources.
We also wanted to show that DB2 can host several gigabytes or terabytes of data with larger and more realistic mail file sizes. The results described here demonstrate the scalability of the Domino mail server with DB2 storage for user mail files that are large enough to represent typical enterprise mail deployment.
We have performed extensive performance testing and analysis to optimize both Lotus Domino 7 (and later) code and DB2 8.2 server to achieve these scalability goals. We will continue to improve the resource consumption and optimization of the Lotus Domino with DB2 storage configuration in upcoming Lotus Domino releases to provide a good alternative to traditional NSF storage.
Resources Learn
Get products and technologies
Discuss
About the authors  | | | Nirmala Venkatraman is a Performance Architect on the Domino Server Performance Team. |
| | | Serge Limoges is a Senior Technical Staff member on the IBM DB2 UDB Performance team responsible for application performance using DB2 UDB. |
Rate this page
| 
