Editor's note: Know a lot about this topic? Want to share your expertise? Participate in the IBM Lotus software wiki program today.
| IBM Lotus Notes and Domino wiki |
|---|
This article is part 3 of a three-part series on performance topics for Lotus Domino 8.5. Be sure to read these developerWorks® articles:
- Part 1, "IBM Lotus Domino 8.5 performance for IBM Lotus Notes users"
- Part 2, "IBM Lotus Domino 8.5 performance for iNotes users"
One of the important performance goals of Lotus Domino 8.5 server was to reduce customers’ disk storage costs without additional processor usage. The data presented in this article shows that these goals were achieved. They were accomplished by improving document and attachment compression, database cache, and router throughput and by adding a new feature, Lotus Domino attachment and object services (DAOS). In addition to the excellent storage cost reduction that has been achieved, processor utilization on the Lotus Domino mail servers was also reduced.
In this article, we show how customers can incrementally upgrade their Lotus Domino 8 mail server cluster infrastructure to Lotus Domino 8.5 and reap processor and I/O improvements after the upgrade process.
Benchmark workload description and configuration setup
We used the Notesbench N8Entmail and N85Entmail, also known as the enterprise mail workloads, to make the comparisons. The new workloads are improvements over the older Entmail workload because they are much heavier and realistic. The new workloads attempt to model real enterprise environments more closely. The workloads, N8Entmail and N85Entmail, model Lotus Notes® client NRPC (Notes remote procedure call) transactions initiated by the Lotus Notes 8 and Lotus Notes 8.5 clients, respectively, for the same tasks or mail user actions. Although the two workloads do the same tasks, they have slightly different NRPC transactions because the newer client and server have fewer NRPC transactions for accomplishing the same tasks while still supporting new features.
The new enterprise mail workload is run using two Lotus Domino mail servers that are clustered, with replication between the servers. Twenty percent of the users have their databases as local applications, not on the servers except where noted. We used a total of 6000 concurrent users with 3000 active users on each server for the comparison between Lotus Domino 8.5 and 8.0. It should be noted that we did not run the nightly tasks while running the workload. Also, we ran the tests with the Microsoft® Windows® 64-bit operating system and the 32-bit version of Lotus Domino.
All the tests were set up with 6000 users defined in each of the Lotus Domino directories on the servers. At the beginning of the test, each user had a mail file of roughly 256 MB of uncompressed documents, with 3000 messages in the inbox and approximately 380 messages in the trash. The messages in the trash were set to expire and were deleted at the rate of two messages every 15 minutes throughout the test. These tests had Lotus Domino transaction logging enabled with the favor runtime setting, and mail journaling was set to journal all messages locally. Lotus Domino domain monitoring (DDM) probes were enabled for messaging and operating systems, and all users had mail rules that blocked mail from 10 users external to the test.
With Lotus Domino 8.5, we enabled document compression on the mail databases, which reduced their size from about 250 MB to about 170 MB. In addition, we enabled the DAOS database property on some of the tests after the mail databases were created, and we also enabled DAOS on the mail boxes and mail journal database. DAOS is a method of saving disk space when multiple mail users are sent the same attachment. For additional information on DAOS, refer to the developerWorks Lotus article, "Achieving ultimate storage and server cost savings with DAOS in IBM Lotus Notes and Lotus Domino 8.5."
A more complete description of the enterprise mail workload is shown in table 1.
Table 1. Enterprise mail workload user actions: six-hour period, 24 iterations
| Activity | Description |
|---|---|
| Refresh inbox | 24 actions |
| Read message | 120 actions (reads five per iteration) |
| Reply to all | 12 actions |
| Send message to one recipient | 24 actions |
| Send message to three recipients | 12 actions |
| Create appointment | 1 action |
| Send invitation | 1 action |
| Send RSVP | 1 action |
| Move to folder NotesBench | 24 actions |
| New mail poll | 24 actions |
| Delete two documents | 24 actions |
| Replicate local mail databases with server | 20 percent of the users work with a local replica. The local copy is replicated with the server every 15 minutes. |
| Cluster replication of all mail databases | Streaming cluster replication feature was enabled on both the servers in the two-node cluster. |
| Replicate all databases hourly | All databases are replicated (pushed) hourly from both servers. The replication schedule is set up in the Lotus Domino directory. |
| Servers run with transaction logging | Running on both |
| Mail journaling | All documents |
| DDM performance probes | All operating system staistics and mail statistics |
| Virus checker | Running on both servers |
For the most part, the tests used the default settings of the Lotus Domino server and mail template. A few settings, though, were changed to enhance performance and enable document compression and DAOS on Lotus Domino 8.5. The changes used on both servers for the Lotus Domino 8.5 and Lotus Domino 8.0 comparisons are detailed in table 2.
Table 2. Lotus Domino server settings
| Configuration | Lotus Domino 8.0 | Lotus Domino 8.5 |
|---|---|---|
| Transaction log | Enabled / favor runtime | Enabled / favor runtime |
| Additional, non-default, mail database options | Don’t overwrite free space | Don’t overwrite free space Disable automatic update of views Compress document data Use DAOS (except where noted) |
| Server tasks | Replica,Router,Update,AMgr,Adminp,Sched,CalConn,RnRMgr,LDAP | Replica,Router,Update,AMgr,Adminp,Sched,CalConn,RnRMgr,LDAP |
| Notes.ini additions | NLCACHE_SIZE=159383552 server_pool_tasks=80 server_max_concurrent_trans=100 server_show_performance=1 RouterDbCacheSize=6100 SCHEDULE_NO_VALIDATE=1 NSF_DBcache_maxentries=6000 CREATE_R8_DATABASES=1 NSF_Buffer_Pool_Size_MB=250 | NLCACHE_SIZE= 159383552 server_pool_tasks=80 server_max_concurrent_trans=100 server_show_performance=1 RouterDbCacheSize=6100 SCHEDULE_NO_VALIDATE=1 NSF_DBcache_maxentries=6000 CREATE_R85_DATABASES=1 DEBUG_NSF_COMPRESS_ALL_NOTES=1 CREATE_R85_LOG=1 ENABLE_LZ1_ENCRYPTED_NOTES=1 NSF_Buffer_Pool_Size_MB=250 |
Table 3 lists the details of the message distribution in different workloads.
Table 3. Message distribution in workloads
| Message size distribution (using the profile option) in bytes | Percentage of messages | Attachment size (if any) in bytes |
|---|---|---|
| 500 | 10% | - |
| 10000 | 30% | - |
| 50000 | 40% | - |
| 50000 | 10.5% | 525000 |
| 150000 | 9.5% | - |
The Lotus Domino servers were two IBM 3850s with two 3.6 GHz Xeon processors, 8 GB of physical memory, each with a DS4300 with 42 fibre disks; the operating system was Microsoft Windows 2003 Server Enterprise 64-bit Edition.
The Notesbench load driver system was a Linux server with local storage capable of handling up to 6000 Lotus Notes N85Entmail simulated users. See figure 1.
Figure 1. Test configuration
NOTE: The results in this article came from benchmarks executed in a controlled environment. While some effort was made during the creation of the benchmark to include typical user operations, it is likely that real users will make different use of Lotus Domino than the narrow range of function that is tested by the benchmark. These numbers should therefore be used primarily to understand the relative performance of the Lotus Domino releases; they do not represent recommendations for real-world deployment. For assistance with capacity planning, consult your hardware vendor and IBM TechLine.
Server upgrade comparison: Lotus Domino 8.0 to Lotus Domino 8.5
The first step in the upgrade process is to upgrade the Lotus Domino server installation from Lotus Domino 8.0 to Lotus Domino 8.5 without upgrading the design and on disk structure (ODS) version of the user mail databases.
Table 4 shows the full comparison on all key resource metrics for Lotus Domino 8.0 and Lotus Domino 8.5 using the Mail8.ntf and ODS48-based mail databases.
Table 4. Comparison of Lotus Domino 8.0 and Lotus Domino 8.5 using Mail8.ntf and ODS48
| Resource | Lotus Domino 8.0 | Lotus Domino 8.5 | % change |
|---|---|---|---|
| Mail template | Mail8.ntf | Mail8.ntf | N/A |
| Database ODS version | 48 | 48 | N/A |
| Percent local replicas | 20 | 20 | N/A |
| Workload | N8Entmail | N85Entmail | N/A |
| Simulated users | 6000 | 6000 | N/A |
| Response time in ms | 33 | 28 | -15% |
| Transactions/minute | 15342 | 12824 | -16% |
| Total % processor | 48.1 | 43.4 | -10% |
| Disk I/Os | 807 | 598 | -26% |
| Disk reads/second | 298 | 145 | -51% |
| Disk writes/second | 509 | 453 | -11% |
| Disk MB/second | 9.19 | 5.76 | -37% |
| Disk MB read/second | 2.35 | 0.81 | -66% |
| Disk MB write/second | 6.84 | 4.95 | -28% |
| Network kbps | 22400 | 22069 | -1% |
| nserver virtual memory in MB | 1580 | 1577 | 0% |
Table 4 shows a comparison of Lotus Domino 8.0 and Lotus Domino 8.5 using the Lotus Domino 8.0 mail template and Lotus Domino 8.0 database ODS version 48. There was a significant reduction of resource usage when we moved from Lotus Domino 8.0 to Lotus Domino 8.5 with the same Mail8.ntf and ODS48 mail databases. We saw a processor reduction of approximately 10 percent by simply upgrading the server from Lotus Domino 8.0 to Lotus Domino 8.5. Other reductions were achieved:
- Reduction of disk I/Os per second (commonly referred as IOPS) of about 26 percent
- Reduction of disk MB per second (also referred as disk bandwidth) of about 37 percent by upgrading the server from Lotus Domino 8.0 to Lotus Domino 8.5
The next step in the Lotus Domino 8.5 upgrade process was to upgrade the design of user mail databases to be based on the new Mail85 template (Mail85.ntf) and to upgrade the database ODS version from 48 to 51. These upgrades can be accomplished by setting the CREATE_R85_DATABASES=1 in the server’s notes.ini and running compact on all the user mail databases.
Table 5 shows the full comparison on all the key resource metrics between Lotus Domino 8.0 and Lotus Domino 8.5 using the Mail85 template and ODS51 databases.
Table 5. Comparison of Lotus Domino 8.0 and Lotus Domino 8.5 using the Mail85.ntf template and ODS51
| Resource | Lotus Domino 8.0 | Lotus Domino 8.5 | % change |
|---|---|---|---|
| Mail template | Mail8.ntf | Mail85.ntf | N/A |
| Database ODS version | 48 | 51 | N/A |
| Percent local replicas | 20 | 20 | N/A |
| Workload | N8Entmail | N85Entmail | N/A |
| Simulated users | 6000 | 6000 | N/A |
| Response time in ms | 33 | 30 | -9% |
| Transactions/minute | 15342 | 12855 | -16% |
| Total % processor | 48.1 | 39.5 | -18% |
| Disk I/Os | 807 | 643 | -20% |
| Disk reads/second | 298 | 170 | -43% |
| Disk writes/second | 509 | 473 | -7% |
| Disk MB/second | 9.19 | 6.13 | -33% |
| Disk MB read/second | 2.35 | 0.96 | -59% |
| Disk MB write/second | 6.84 | 5.17 | -24% |
| Network kbps | 22400 | 22091 | -1% |
| nserver virtual memory in MB | 1580 | 1595 | 0% |
Table 5 shows that there was a significant reduction of resource usage by upgrading from Lotus Domino 8.0 to Lotus Domino 8.5 with the Mail85 template and ODS51. These reductions were seen:
- Processor reduction of about 18 percent
- Reduction of disk I/Os per second of 20 percent
- Reduction of disk MB per second of 33 percent
Now let's look at the effect of some of the new Lotus Domino 8.5 features, such as DAOS, on Lotus Domino 8.5 server performance. Table 6 shows a comparison of Lotus Domino 8.5 and Lotus Domino 8.5 using DAOS. Both tests used mail databases built with the Mail85 template and ODS51. DAOS was enabled in the server document of both cluster mates and enabled on router mailbox databases, mail journal databases, and mail databases on both the cluster mates.
Table 6 shows the full comparison of all key resource metrics on a Lotus Domino 8.5 server with and without DAOS enabled.
Table 6. Comparison of Lotus Domino 8.5 with and without DAOS
| Resource | Lotus Domino 8.5 | Lotus Domino 8.5 | % change |
|---|---|---|---|
| Mail template | Mail85.ntf | Mail85.ntf | N/A |
| Database ODS version | 51 | 51 | N/A |
| DAOS | Disabled | Enabled | N/A |
| Percent local replicas | 20 | 20 | N/A |
| Workload | N85Entmail | N85Entmail | N/A |
| Simulated users | 6000 | 6000 | N/A |
| Response time in ms | 30 | 31 | 3% |
| Transactions/minute | 12855 | 12819 | 0% |
| Total % processor | 39.5 | 39.5 | 0% |
| Disk I/Os | 643 | 636 | -1% |
| Disk reads/second | 170 | 166 | -2% |
| Disk writes/second | 473 | 470 | -1% |
| Disk MB/second | 6.13 | 5.52 | -10% |
| Disk MB read/second | 0.96 | 0.91 | -5% |
| Disk MB write/second | 5.17 | 4.61 | -11% |
| Network kbps | 22091 | 21856 | -1% |
| nserver virtual memory in MB | 1595 | 1591 | 0% |
From table 6, we can see that by enabling the DAOS feature on the Lotus Domino 8.5 server, the total disk MB/second was reduced by 10 percent . Lotus Domino 8.5 customer deployment data shows that the DAOS feature in Lotus Domino 8.5 reduced the storage volume required on the server by 20 percent or more. This reduction in the size of the mail databases offers benefits that help to reduce the cost of database maintenance.
We performed another interesting test on the Lotus Domino 8.5 server using all the new storage space features. We studied the effect of having local replica users as opposed to server-based mail file users. For this test, we had 100 percent of the users in the Lotus Domino directory configured for server-based mail files; in other words, none of the test mail users had local mail replicas.
Table 7 compares the results of the previous Lotus Domino 8.5 measurement with 20 percent of the users enabled for local replicas (the remaining 80 percent of the users had server-based mail files) with the Lotus Domino 8.5 test with no local replica users (100 percent of users had server-based mail files).
Table 7. Comparison of Lotus Domino 8.5 with 20 percent of users enabled for local replicas and 100 percent of users with server-based mail files
| Resource | Lotus Domino 8.5 | Lotus Domino 8.5 | % change |
|---|---|---|---|
| Mail template | Mail85.ntf | Mail85.ntf | N/A |
| Database ODS version | 51 | 51 | N/A |
| DAOS | Enabled | Enabled | N/A |
| Percent local replicas | 20 | 0 | N/A |
| Workload | N85Entmail | N85Entmail | N/A |
| Simulated users | 6000 | 6000 | N/A |
| Response time in ms | 31 | 37 | 19% |
| Transactions/minute | 12819 | 14089 | 10% |
| Total % processor | 39.5 | 53.4 | 35% |
| Disk I/Os | 636 | 728 | 15% |
| Disk reads/second | 166 | 184 | 11% |
| Disk writes/second | 470 | 544 | 16% |
| Disk MB/second | 5.52 | 6.31 | 14% |
| Disk MB read/second | 0.91 | 1.04 | 14% |
| Disk MB write/second | 4.61 | 5.28 | 15% |
| nserver virtual memory in MB | 1591 | 1713 | 8% |
With 100 percent users having server-based mail files, there was an increase in the server utilization of resources:
- The processor utilization increased by 35 percent.
- I/O increased by about 15 percent with 100 percent of users having server-based mail files when compared to 80 percent of users having server-based mail files and 20 percent using local replicas.
- The local replica user network traffic is approximately 50 percent that of a server-based mail user. The local replica users put less load on the server.
Performance analysis: Upgrading Lotus Domino 8.0 to Lotus Domino 8.5
Based on the data reported in this article, we can see that the greatest improvements in resource usage in upgrading from Lotus Domino 8.0 to Lotus Domino 8.5 are the disk MB/second and disk I/O/second results. For 6000 simulated users spread evenly between two cluster nodes, disk MB/second reduced by as much as to 37 percent and disk I/O/second reduced by as much as 26 percent with Lotus Domino 8.5. Lotus Domino 8.5 also offered a considerable processor reduction ranging from 10 to 18 percent as compared to Lotus Domino 8.0.
From the testing, we believe that upgrading to Lotus Domino 8.5 from Lotus Domino 8.0 can reduce total cost of ownership by reducing storage expenses, saving processor resources, and using available system resources more efficiently.
- Participate in the discussion forum.
-
Read part 1 of this series, "IBM Lotus Domino 8.5 performance for IBM Lotus Notes users" on developerWorks Lotus.
-
Read part 2 of this series, "IBM Lotus Domino 8.5 performance for iNotes users" on developerWorks Lotus.
-
Refer to the developerWorks Lotus article, "Achieving ultimate storage and server cost savings with DAOS in IBM Lotus Notes and Lotus Domino 8.5."
-
Contribute to the IBM Lotus Domino wiki.
-
Refer to the developerWorks IBM Lotus Notes and Domino product page.
Harry Murray has been a member of the Lotus Domino performance team for 11 years and worked as a performance engineer for Digital Equipment Corporation prior to that.
Comments (Undergoing maintenance)





