IBM Lotus Domino 8.5 server performance, part 3: Enterprise cluster mail performance

IBM® Lotus® Domino® 8.5 server offers a lot of features aimed at reducing the overall total cost of ownership (TCO) of the Lotus Domino mail server cluster infrastructure in a large enterprise. In this article, we show you how you can leverage these features incrementally to realize TCO benefits while upgrading to Lotus Domino 8.5.

Harry Murray, Lotus Domino Performance Engineer, IBM  

Harry Murray has been a member of the Lotus Domino performance team for 11 years and worked as a performance engineer for Digital Equipment Corporation prior to that.



Nirmala Venkatraman, Performance Architect, IBM

Nirmala Venkatraman is a Performance Architect on the Lotus Domino server performance team. You can reach her at nvenkatr@us.ibm.com.



Razeyah Stephen, Manager, Lotus Domino Performance Team, IBM

Razeyah Stephen is a manager of the IBM Lotus Domino and IBM Lotus Sametime performance teams. She joined IBM in October 1998.



28 July 2009

Also available in Chinese Russian Japanese

Editor's note: Know a lot about this topic? Want to share your expertise? Participate in the IBM Lotus software wiki program today.

Introduction

This article is part 3 of a three-part series on performance topics for Lotus Domino 8.5. Be sure to read these developerWorks® articles:

One of the important performance goals of Lotus Domino 8.5 server was to reduce customers’ disk storage costs without additional processor usage. The data presented in this article shows that these goals were achieved. They were accomplished by improving document and attachment compression, database cache, and router throughput and by adding a new feature, Lotus Domino attachment and object services (DAOS). In addition to the excellent storage cost reduction that has been achieved, processor utilization on the Lotus Domino mail servers was also reduced.

In this article, we show how customers can incrementally upgrade their Lotus Domino 8 mail server cluster infrastructure to Lotus Domino 8.5 and reap processor and I/O improvements after the upgrade process.


Benchmark workload description and configuration setup

We used the Notesbench N8Entmail and N85Entmail, also known as the enterprise mail workloads, to make the comparisons. The new workloads are improvements over the older Entmail workload because they are much heavier and realistic. The new workloads attempt to model real enterprise environments more closely. The workloads, N8Entmail and N85Entmail, model Lotus Notes® client NRPC (Notes remote procedure call) transactions initiated by the Lotus Notes 8 and Lotus Notes 8.5 clients, respectively, for the same tasks or mail user actions. Although the two workloads do the same tasks, they have slightly different NRPC transactions because the newer client and server have fewer NRPC transactions for accomplishing the same tasks while still supporting new features.

The new enterprise mail workload is run using two Lotus Domino mail servers that are clustered, with replication between the servers. Twenty percent of the users have their databases as local applications, not on the servers except where noted. We used a total of 6000 concurrent users with 3000 active users on each server for the comparison between Lotus Domino 8.5 and 8.0. It should be noted that we did not run the nightly tasks while running the workload. Also, we ran the tests with the Microsoft® Windows® 64-bit operating system and the 32-bit version of Lotus Domino.

All the tests were set up with 6000 users defined in each of the Lotus Domino directories on the servers. At the beginning of the test, each user had a mail file of roughly 256 MB of uncompressed documents, with 3000 messages in the inbox and approximately 380 messages in the trash. The messages in the trash were set to expire and were deleted at the rate of two messages every 15 minutes throughout the test. These tests had Lotus Domino transaction logging enabled with the favor runtime setting, and mail journaling was set to journal all messages locally. Lotus Domino domain monitoring (DDM) probes were enabled for messaging and operating systems, and all users had mail rules that blocked mail from 10 users external to the test.

With Lotus Domino 8.5, we enabled document compression on the mail databases, which reduced their size from about 250 MB to about 170 MB. In addition, we enabled the DAOS database property on some of the tests after the mail databases were created, and we also enabled DAOS on the mail boxes and mail journal database. DAOS is a method of saving disk space when multiple mail users are sent the same attachment. For additional information on DAOS, refer to the developerWorks Lotus article, "Achieving ultimate storage and server cost savings with DAOS in IBM Lotus Notes and Lotus Domino 8.5."

A more complete description of the enterprise mail workload is shown in table 1.

Table 1. Enterprise mail workload user actions: six-hour period, 24 iterations
Activity Description
Refresh inbox24 actions
Read message 120 actions (reads five per iteration)
Reply to all12 actions
Send message to one recipient24 actions
Send message to three recipients12 actions
Create appointment1 action
Send invitation1 action
Send RSVP1 action
Move to folder NotesBench24 actions
New mail poll24 actions
Delete two documents24 actions
Replicate local mail databases with server20 percent of the users work with a local replica. The local copy is replicated with the server every 15 minutes.
Cluster replication of all mail databasesStreaming cluster replication feature was enabled on both the servers in the two-node cluster.
Replicate all databases hourlyAll databases are replicated (pushed) hourly from both servers. The replication schedule is set up in the Lotus Domino directory.
Servers run with transaction loggingRunning on both
Mail journalingAll documents
DDM performance probes All operating system staistics and mail statistics
Virus checkerRunning on both servers

For the most part, the tests used the default settings of the Lotus Domino server and mail template. A few settings, though, were changed to enhance performance and enable document compression and DAOS on Lotus Domino 8.5. The changes used on both servers for the Lotus Domino 8.5 and Lotus Domino 8.0 comparisons are detailed in table 2.

Table 2. Lotus Domino server settings
ConfigurationLotus Domino 8.0 Lotus Domino 8.5
Transaction logEnabled / favor runtime Enabled / favor runtime
Additional, non-default, mail database optionsDon’t overwrite free spaceDon’t overwrite free space
Disable automatic update of views
Compress document data
Use DAOS (except where noted)
Server tasksReplica,Router,Update,AMgr,Adminp,Sched,CalConn,RnRMgr,LDAPReplica,Router,Update,AMgr,Adminp,Sched,CalConn,RnRMgr,LDAP
Notes.ini additionsNLCACHE_SIZE=159383552
server_pool_tasks=80
server_max_concurrent_trans=100
server_show_performance=1
RouterDbCacheSize=6100
SCHEDULE_NO_VALIDATE=1
NSF_DBcache_maxentries=6000
CREATE_R8_DATABASES=1
NSF_Buffer_Pool_Size_MB=250
NLCACHE_SIZE= 159383552
server_pool_tasks=80
server_max_concurrent_trans=100
server_show_performance=1
RouterDbCacheSize=6100
SCHEDULE_NO_VALIDATE=1
NSF_DBcache_maxentries=6000
CREATE_R85_DATABASES=1
DEBUG_NSF_COMPRESS_ALL_NOTES=1
CREATE_R85_LOG=1
ENABLE_LZ1_ENCRYPTED_NOTES=1
NSF_Buffer_Pool_Size_MB=250

Table 3 lists the details of the message distribution in different workloads.

Table 3. Message distribution in workloads
Message size distribution (using the profile option) in bytesPercentage of messages Attachment size (if any) in bytes
50010%-
1000030%-
5000040%-
5000010.5%525000
1500009.5%-

Test configuration

The Lotus Domino servers were two IBM 3850s with two 3.6 GHz Xeon processors, 8 GB of physical memory, each with a DS4300 with 42 fibre disks; the operating system was Microsoft Windows 2003 Server Enterprise 64-bit Edition.

The Notesbench load driver system was a Linux server with local storage capable of handling up to 6000 Lotus Notes N85Entmail simulated users. See figure 1.

Figure 1. Test configuration
Test configuration

NOTE: The results in this article came from benchmarks executed in a controlled environment. While some effort was made during the creation of the benchmark to include typical user operations, it is likely that real users will make different use of Lotus Domino than the narrow range of function that is tested by the benchmark. These numbers should therefore be used primarily to understand the relative performance of the Lotus Domino releases; they do not represent recommendations for real-world deployment. For assistance with capacity planning, consult your hardware vendor and IBM TechLine.


Server upgrade comparison: Lotus Domino 8.0 to Lotus Domino 8.5

The first step in the upgrade process is to upgrade the Lotus Domino server installation from Lotus Domino 8.0 to Lotus Domino 8.5 without upgrading the design and on disk structure (ODS) version of the user mail databases.

Table 4 shows the full comparison on all key resource metrics for Lotus Domino 8.0 and Lotus Domino 8.5 using the Mail8.ntf and ODS48-based mail databases.

Table 4. Comparison of Lotus Domino 8.0 and Lotus Domino 8.5 using Mail8.ntf and ODS48
ResourceLotus Domino 8.0Lotus Domino 8.5% change
Mail templateMail8.ntfMail8.ntfN/A
Database ODS version 4848N/A
Percent local replicas2020N/A
WorkloadN8EntmailN85EntmailN/A
Simulated users60006000N/A
Response time in ms3328-15%
Transactions/minute1534212824-16%
Total % processor48.143.4-10%
Disk I/Os807598-26%
Disk reads/second298145-51%
Disk writes/second509453-11%
Disk MB/second9.195.76-37%
Disk MB read/second2.350.81-66%
Disk MB write/second6.844.95-28%
Network kbps2240022069-1%
nserver virtual memory in MB158015770%

Table 4 shows a comparison of Lotus Domino 8.0 and Lotus Domino 8.5 using the Lotus Domino 8.0 mail template and Lotus Domino 8.0 database ODS version 48. There was a significant reduction of resource usage when we moved from Lotus Domino 8.0 to Lotus Domino 8.5 with the same Mail8.ntf and ODS48 mail databases. We saw a processor reduction of approximately 10 percent by simply upgrading the server from Lotus Domino 8.0 to Lotus Domino 8.5. Other reductions were achieved:

  • Reduction of disk I/Os per second (commonly referred as IOPS) of about 26 percent
  • Reduction of disk MB per second (also referred as disk bandwidth) of about 37 percent by upgrading the server from Lotus Domino 8.0 to Lotus Domino 8.5

The next step in the Lotus Domino 8.5 upgrade process was to upgrade the design of user mail databases to be based on the new Mail85 template (Mail85.ntf) and to upgrade the database ODS version from 48 to 51. These upgrades can be accomplished by setting the CREATE_R85_DATABASES=1 in the server’s notes.ini and running compact on all the user mail databases.

Table 5 shows the full comparison on all the key resource metrics between Lotus Domino 8.0 and Lotus Domino 8.5 using the Mail85 template and ODS51 databases.

Table 5. Comparison of Lotus Domino 8.0 and Lotus Domino 8.5 using the Mail85.ntf template and ODS51
ResourceLotus Domino 8.0Lotus Domino 8.5% change
Mail templateMail8.ntfMail85.ntfN/A
Database ODS version 4851N/A
Percent local replicas2020N/A
WorkloadN8EntmailN85EntmailN/A
Simulated users60006000N/A
Response time in ms3330-9%
Transactions/minute1534212855-16%
Total % processor48.139.5-18%
Disk I/Os807643-20%
Disk reads/second298170-43%
Disk writes/second509473-7%
Disk MB/second9.196.13-33%
Disk MB read/second2.350.96-59%
Disk MB write/second6.845.17-24%
Network kbps2240022091-1%
nserver virtual memory in MB158015950%

Table 5 shows that there was a significant reduction of resource usage by upgrading from Lotus Domino 8.0 to Lotus Domino 8.5 with the Mail85 template and ODS51. These reductions were seen:

  • Processor reduction of about 18 percent
  • Reduction of disk I/Os per second of 20 percent
  • Reduction of disk MB per second of 33 percent

Now let's look at the effect of some of the new Lotus Domino 8.5 features, such as DAOS, on Lotus Domino 8.5 server performance. Table 6 shows a comparison of Lotus Domino 8.5 and Lotus Domino 8.5 using DAOS. Both tests used mail databases built with the Mail85 template and ODS51. DAOS was enabled in the server document of both cluster mates and enabled on router mailbox databases, mail journal databases, and mail databases on both the cluster mates.

Table 6 shows the full comparison of all key resource metrics on a Lotus Domino 8.5 server with and without DAOS enabled.

Table 6. Comparison of Lotus Domino 8.5 with and without DAOS
ResourceLotus Domino 8.5Lotus Domino 8.5% change
Mail templateMail85.ntfMail85.ntfN/A
Database ODS version 5151N/A
DAOSDisabledEnabledN/A
Percent local replicas2020N/A
WorkloadN85EntmailN85EntmailN/A
Simulated users60006000N/A
Response time in ms30313%
Transactions/minute12855128190%
Total % processor39.539.50%
Disk I/Os643636-1%
Disk reads/second170166-2%
Disk writes/second473470-1%
Disk MB/second6.135.52-10%
Disk MB read/second0.960.91-5%
Disk MB write/second5.174.61-11%
Network kbps2209121856-1%
nserver virtual memory in MB159515910%

From table 6, we can see that by enabling the DAOS feature on the Lotus Domino 8.5 server, the total disk MB/second was reduced by 10 percent . Lotus Domino 8.5 customer deployment data shows that the DAOS feature in Lotus Domino 8.5 reduced the storage volume required on the server by 20 percent or more. This reduction in the size of the mail databases offers benefits that help to reduce the cost of database maintenance.

We performed another interesting test on the Lotus Domino 8.5 server using all the new storage space features. We studied the effect of having local replica users as opposed to server-based mail file users. For this test, we had 100 percent of the users in the Lotus Domino directory configured for server-based mail files; in other words, none of the test mail users had local mail replicas.

Table 7 compares the results of the previous Lotus Domino 8.5 measurement with 20 percent of the users enabled for local replicas (the remaining 80 percent of the users had server-based mail files) with the Lotus Domino 8.5 test with no local replica users (100 percent of users had server-based mail files).

Table 7. Comparison of Lotus Domino 8.5 with 20 percent of users enabled for local replicas and 100 percent of users with server-based mail files
ResourceLotus Domino 8.5Lotus Domino 8.5% change
Mail templateMail85.ntfMail85.ntfN/A
Database ODS version 5151N/A
DAOSEnabledEnabledN/A
Percent local replicas200N/A
WorkloadN85EntmailN85EntmailN/A
Simulated users60006000N/A
Response time in ms313719%
Transactions/minute128191408910%
Total % processor39.553.435%
Disk I/Os63672815%
Disk reads/second16618411%
Disk writes/second47054416%
Disk MB/second5.526.3114%
Disk MB read/second0.911.0414%
Disk MB write/second4.615.2815%
nserver virtual memory in MB159117138%

With 100 percent users having server-based mail files, there was an increase in the server utilization of resources:

  • The processor utilization increased by 35 percent.
  • I/O increased by about 15 percent with 100 percent of users having server-based mail files when compared to 80 percent of users having server-based mail files and 20 percent using local replicas.
  • The local replica user network traffic is approximately 50 percent that of a server-based mail user. The local replica users put less load on the server.

Performance analysis: Upgrading Lotus Domino 8.0 to Lotus Domino 8.5

Based on the data reported in this article, we can see that the greatest improvements in resource usage in upgrading from Lotus Domino 8.0 to Lotus Domino 8.5 are the disk MB/second and disk I/O/second results. For 6000 simulated users spread evenly between two cluster nodes, disk MB/second reduced by as much as to 37 percent and disk I/O/second reduced by as much as 26 percent with Lotus Domino 8.5. Lotus Domino 8.5 also offered a considerable processor reduction ranging from 10 to 18 percent as compared to Lotus Domino 8.0.


Conclusion

From the testing, we believe that upgrading to Lotus Domino 8.5 from Lotus Domino 8.0 can reduce total cost of ownership by reducing storage expenses, saving processor resources, and using available system resources more efficiently.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Select information in your profile (name, country/region, and company) is displayed to the public and will accompany any content you post. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into IBM collaboration and social software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Lotus
ArticleID=416652
ArticleTitle=IBM Lotus Domino 8.5 server performance, part 3: Enterprise cluster mail performance
publish-date=07282009