This article is the third in a series dedicated to performance improvements we have measured with Domino 7. The first article, “Lotus Domino 7 server performance, Part 1: Lotus Notes client workloads," talked about standard Domino mail performance. The second article, “Lotus Domino server performance, Part 2: Domino 7 performance for Domino Web Access users,” discussed Domino HTTP performance results we observed with simulated Domino Web Access users. This article is focused on a new benchmark called Enterprise Mail. This benchmark simulates Notes users in an enterprise environment. (A future article will examine several Notes.ini variables that are important for setting optimal performance in Domino 7.)

The first part of this article will explain the reason we developed the Enterprise Mail workload, and describe what the workload does. The rest of the article will show the benchmark results we obtained, on a variety of platforms. We hope you will find the information useful, and gain an appreciation for the improvements that have gone into Domino 7.

Note: The test results displayed in this article were from benchmarks executed in a controlled environment. While great effort was made during the creation of the benchmarks to include typical user operations, it is likely that real users will make use of Domino in ways different from the narrow range of function tested by the benchmark. These numbers should therefore be used primarily to understand the relative performance of the Domino releases, and do not represent recommendations for real-world deployment. For assistance with capacity planning, we recommend you consult your hardware vendor.

Also, while we show results on a variety of hardware platforms, these configurations are not of uniform capacity. It is our intent to focus on the performance of Domino itself, and this data should not be used to compare platforms against each other.

The Enterprise Mail workload benchmark

A benchmark is a test of the performance of a computer system. Benchmarks can test many things. In this case, we compared different releases of software (Domino 7 and Domino 6.5) on the same machine. Unlike our other benchmarks, where we attempted to show the maximum number of users that the test system could support, this benchmark focused on the hardware resources needed to support a fixed number of users.

The Enterprise Mail (EntMail) workload is a set of commands that are included in the NotesBench tool and in the Server.Load tool. We started with the R6mail workload as a base:

We then added several new activities:

When executed, the Enterprise Mail workload attempts to simulate a variety of user activities on the target servers. These activities include messaging, calendaring, database searches, database replication between servers, local database replication on the client, and cluster replication. The Enterprise Mail workload was developed early in the Domino 7 development cycle, to exercise a wider range of Domino functionality, so we could optimize the performance of the various components of Domino. The workload is much closer to a true customer's environment that includes replication, clustering, local mail database replication, full text indexing, transaction logging, and so on.

This Enterprise Mail test measures the following metrics on the server:

• Throughput of completed Notes operations.
• Average response time at maximum capacity.

The resulting resource utilization at a given user population is then compared between Domino releases.

Neither real-time virus scans nor other third-party vendor software were running on the servers. Although we tried to make the setup as real as possible, we needed to eliminate work that was both variable in time, and also variable between different customer sites. For more on how to run an Enterprise Mail test using Server.Load, see the sidebar.

Back to top

General results

The percent CPU improvement for Domino 7 on all platforms varied by the number of active users. Generally, the more users, the better the improvement for Domino 7 compared with Domino 6.5.

Resource usage for each platform was measured at steady state at the highest number of total users reported. The average was not recorded until two hours after steady state, and the results were averaged over approximately six hours. The number of users reported was the total number of users running against both servers. So, half of the users were running against one of the servers, and half were run against its cluster mate. The number of total users was sized so that Domino 6.5 was not overwhelmed by the load. This allowed Domino 6.5 to keep up with the load, and reach steady state.

The number of cluster replicators was set so that the cluster replication would not get any further behind than 15 minutes (900 seconds).

Notes.ini settings

We used several performance-related Notes.ini variables to help optimize server performance in our tests. For example, the settings FTUpdate_Idle_Time and Update_Idle_Time were introduced in Domino 6.5.x. FTUpdate_Idle_Time specifies when (in seconds) full text updating occurs, and Update_Idle_Time sets the frequency of view updating. For both variables, the default is 5 seconds. Domino 7 has more efficient cluster replication, so we used these two settings to slow down the updating in Domino 7, to ensure it did approximately the same amount of work as in Domino 6.5. We were then able to make a valid comparison of CPU usage.

The setting cluster_replicator sets the number of cluster replicators. In Domino 6.5, this setting needed to be higher than in Domino 7, to keep the cluster replication from getting further behind than 15 minutes.

We set ServerTasks to update,replica,router,amgr,adminp. And we set EnableForcedFillOnFileExtend to 1, to enable Domino to explicitly use the operating system/file system to extend the database file (NSF).

The variable NSF_DBCache_Max_Clean_Hold_Time determines the length of time, in minutes, that the database is held in DbCache in the clean state before it is removed from cache. Setting this to a higher value allows the databases to remain in cache longer, resulting in less work for the server.

In addition, we recommend setting the variable Server_Transinfo_range on all Domino production clustered machines. This will determine the sensitivity to failing over client requests to the cluster member when the server gets busy. Determining the value at which to set this is an iterative process, based on monitoring the Server Expansion Factor and the Server Availability Index. For a complete understanding of these values and settings, refer to the Domino 7 documentation on configuring the Server Availability Index. (We did not set up our servers to use failover, so this setting had no impact on our tests.)

You can find full information on these and all other Notes.ini variables in the Domino 7 documentation. Also, an upcoming developerWorks Lotus article will discuss Domino 7 performance-related Notes.ini variables.

In the next sections of this article, we discuss specific test results we obtained on the AIX, iSeries, Solaris 9, Windows 2003, zSeries Linux, and z/OS platforms.

AIX

The tests on AIX were run on a pSeries Model p570 Power 5 machine, running the latest version of the AIX 5.3 operating system. We used a dedicated Logical Partition (LPAR) with two physical 1.65 GHz CPUs defined to it. AIX 5.3 and the Power 5 hardware supports Symmetrical MultiThreading (SMT), and provides us with (in this case) two additional virtual CPUs. We use a Network Appliance Filer configured as a Network Attached Storage unit. The LPAR is configured with two Domino partitions that also run as a Domino server cluster. All tests are run with transaction logging enabled, and run using circular mode. The tests were run with 6000 total concurrent simulated users, with 3000 users defined to have their home mail server on each Domino server. We had 20 percent of the users defined to have local mail file replicas. Pull replication occurred between the Domino servers on an hourly schedule. Update and full-text indexing were also enabled.

The following table shows our hardware/software setup for AIX testing:

To help optimize performance, we entered the following settings into the test servers' Notes.ini files:

The Server_Pool_Tasks and Server_Max_Concurrent_Trans values were set to support the high-end user numbers achieved on each Domino release. Before you change from the default values of these settings, we recommend that you perform analysis to optimize the values used. We also needed to adjust the size of the cluster replication pool for the Domino 7 test runs, via the setting RTR_MAX_POOLSIZE_MB. (This setting was only expected to be needed in a test environment, but if the pool size limit maximum is reached, an error message appears.) The client response time was less than 1 second during the tests. Figure 1 shows our test results.

As figure 1 shows, the same number of simulated users, running the same workload for the same amount of time, results in a 27 percent reduction in CPU usage between Domino 7 and Domino 6.5. And when Domino 7 ran mail7.ntf and Domino 6.5 ran mail6.ntf, the reduction in CPU was 22 percent.

The following tables contain resource utilization numbers for tests we performed with both templates. The first table shows 6000 simulated users running the mail6.ntf mail template:

In the second table, our simulated Domino 7 users ran the mail7.ntf mail template, and Domino 6.5 users ran the mail6.ntf mail template:

The disk and network numbers shown in these tables were measured at the Network Appliance Filer, using the sysstat command.

iSeries

Two Domino partitions were configured for the iSeries environment, with data points up to and including 4000 users. Five clients emulated 800 users each, for a total of 4000 mail users. By default, on iSeries, data is spread evenly across disk drives. Network access was through a single one-GB Ethernet adapter. The following table lists the configuration for our iSeries tests:

We entered the following settings into the test servers' Notes.ini files:

As was true on other platforms, Domino 7 running on iSeries offers improvement in CPU utilization and scalability. Domino 7 produced reduction in CPU utilization vs. Domino 6.5, at multiple data points, up to and including 4000 mail users (see figure 2).

The client response time was less than 1 second during the tests.

The following tables show how resource utilizations compare for each template. The first table shows 4000 simulated users running the mail6.ntf mail template:

In the second table, our simulated Domino 7 users ran the mail7.ntf mail template, and Domino 6.5 users ran the mail6.ntf mail template:

Domino 6.5 at 4000 users utilized 54.8 percent CPU, with average response time of 0.123 seconds. Domino 7 with the mail6.ntf mail template at 4000 users consumed 46.8 percent CPU, with average response time of 0.111 seconds; and Domino 7 with the mail7.ntf mail template at 4000 users used 49.2 percent CPU, with average response time of 0.100 seconds. At 4000 users, these numbers produced a 15 percent reduction in CPU for Domino 7 with the mail6.ntf mail template, and a 10 percent CPU savings for Domino 7 with the mail7.ntf mail template.

Solaris 9

The Domino cluster used for our Solaris tests contained two Sun servers, each an eight-CPU 6800, and a significantly faster 12-way, dual-core 2900. We’ll show details from the slower 6800, because it is the limiting factor in this test. The Sun 6800 consisted of an 8-CPU domain from a 12-CPU system. We have seven T3 arrays, with nine drives each used in this test. We installed the Domino executables on the first array, and spread the user databases evenly across the first six arrays. The seventh array was used for the Domino transaction log.

There were a total of 14,000 users defined in the Domino directory, all of which were active during the test. These simulated users were spread across 14 client driver workstations, each simulating 1000 users. Half of these users had the 6800 as their home server. This table shows an overview of the 6800 hardware:

Configuration settings for the Domino 6.5 and Domino 7 tests were similar, with the main difference being that with Domino 7, we could reduce the number of cluster replicator tasks from eight to six, and still meet the benchmark goal of keeping the average cluster replication queue time under 15 minutes. The Domino 7 server was also configured to take advantage of the Solaris large page support. The following table shows the changes we made to each server's Notes.ini file:

The next table compares the system resources used by Domino 6.5 and Domino 7, while running 14,000 users active across the cluster. For the Domino 6.5 tests, the mail files were based on the mail6.ntf template, while the Domino 7 results used the mail7.ntf template.

The client response time was less than 1 second during the tests.

Unlike our other benchmarks, where we attempted to show the maximum number of users that the test system will support, this benchmark focused on the hardware resources needed to support a fixed number of users. Here, we can see that the all-important CPU utilization has dropped 28 percent with Domino 7, while network and memory usage remain about the same.

Windows 2003

For Windows testing, Domino 7 was set up as a single-partition clustered server on an eServer xSeries 360, and a Compaq Proliant DL580 server running Windows 2003 Enterprise Server with two real processors, no hyperthreading, and with approximately 3.5 GB of available memory. The Domino executable files were installed on a single internal drive along with the operating system. The mail databases were spread across two IBM EXP arrays (direct connect SCSI), configured as RAID 0 on each server. Network access was through a single one-GB Ethernet adapter, running in full duplex mode. The transaction log was on a separate disk, favored runtime, and was set to a maximum size of 4 GB.

The following tables shows the hardware/software configuration for our Windows tests. This first table contains the xSeries configuration data:

And this table displays the hardware/software configuration for our Compaq Proliant DL 580 tests:

We entered the following settings into the test servers' Notes.ini files:

We ran 4000 defined users for both the Domino 7 and Domino 6.5 tests.

For this testing, the two servers were not balanced (in other words, they were not equal in power). The Compaq server was a few years older, and had lower clock speed CPUs. The number of users on the cluster was sized so that Domino 6.5 on the less powerful server was not overwhelmed. We will point out a few of the differences on how Domino 6.5 ran on xSeries versus the Compaq server.

Very little difference was seen in the CPU usage for Domino 7 (using databases created with the Domino 6.5 template), and the Domino 7 template on Windows, at 2000 users per server. All Domino 7 testing for Windows was done with databases using the mail7.ntf template, and Domino 6.5 testing was done using the mail6.ntf template. The client response time was less than 1 second during the tests.

The following table shows the percent of CPU improvement for Domino 7 compared to Domino 6.5:

This table is a comparison of resource usage for the xSeries server.

As you can observe from our data, Domino 7 running on Windows 2003 had a significant CPU usage performance advantage over Domino 6.5. The memory, disk, and network resource usage was about the same between the two releases. This, and previous tests, confirm that Domino 7 improves on the scalability, performance, and TCO, when compared to Domino 6.5.

zSeries Linux

All zSeries Linux performance test results discussed in this section come from one logical partition (LPAR) on a series z990 model 2084-C24. The z990 has 24 CPUs available, 3 of which were used in the performance test. The remaining 18 CPUs, as well as some other machine resources, were shared among 13 other LPARs used for Domino development and test activities. For these tests, we only used three of the six CPUs to drive the load with higher CPU utilization. The performance test LPAR was configured with 12 GB of memory. On SLES 8, only two GB were used for central memory because of the 31-bit operating system, and two GB expanded memory for swap space. On SLES 9, we used 12 GB total. Two Domino partitions were run on this LPAR. We used two one-GB Ethernet Open Systems Architecture (OSA) cards. Our LAN was isolated. All disks were allocated from an Enterprise Storage Server (2105 Model 800) array, with each disk configured as a 3390 model 3.

There were separate file systems allocated on single volumes (disks) for the Domino execution, data (excepting client mail databases), and the Domino Directory (Names.nsf), and two volumes in a logical volume manager (LVM) file system for transaction logging. Client mail databases were distributed evenly over a 52 LVM file system, each allocated across 5 volumes in a single LVM, providing 11.5 GB of useable space per file system. The EXT 3 file system was used on Linux for zSeries. The operating systems installed were SLES 8 with SP3, or SLES 9 with SP1. We ran transaction logging enabled, with hardware compression data instead of LZ1 software. This feature is only available in Domino 7 on zSeries.

The following table shows the hardware/software configuration for our zServer Linux tests:

The next table shows the changes we made to each server's Notes.ini file:

Figure 3 shows the CPU improvement from Domino 6.5 vs. Domino 7 running an Enterprise Mail workload, implementing either the mail6.ntf template from Domino 6.5, or the mail7.ntf template:

The client response time was less than 1 second during the tests.

The next three tables display resource utilization numbers when testing 6000 users with both the mail6.ntf and mail7.ntf templates. The first table shows results obtained with the mail6.ntf template on SLES 8:

The second table shows our numbers with Domino 7 users running the mail7.ntf template on SLES 8:

The third table shows our numbers with Domino 7 users running the mail6.ntf template on SLES 9:

The tests showed CPU improvement of 37 percent running Domino 7 with the mail6.ntf template, 31 percent running Domino 7 with the mail7.ntf template, and 30 percent running Domino 7 on SLES 9 with the mail6.ntf mail template. Clearly, Domino 7 improved CPU with both mail6.ntf and mail7.ntf compared to Domino 6.5. As a result, CPU reduction on Domino 7 translates into improved stability at high workload levels, allowing more clients to be supported by a single Domino 7 on Linux on zSeries server. More importantly, the lighter CPU requirements of Domino 7 can produce substantially lower total cost of ownership compared to Domino 6.5 on Linux on zSeries.

z/OS

All z/OS performance tests described in this section were run on the identical hardware system described for the zSeries Linux test (see the preceding section). There was a separate z/FS file system allocated on a single volume (disk) for the Domino execution, data (excepting client mail databases), and the Domino Directory (Names.nsf). A file system that spanned two volumes was allocated for transaction log data. Client mail databases were distributed evenly over 52 z/FSs, each allocated across 5 span volumes providing 11.5 GB of useable space per file system. The operating system installed was z/OS version 1 release 5. We ran transaction logging enabled, with hardware compression data instead of LZ1 software. This feature is only available in Domino 7 on zSeries.

This table shows our hardware/software configuration for our z/OS tests:

The next table shows the changes we made to each server's Notes.ini file:

Figure 4 shows the CPU improvement from Domino 6.5 vs. Domino 7 running our Enterprise Mail workload, implementing either the mail6.ntf template from Domino 6.5, or the mail7.ntf template:

The client response time was less than 1 second during the tests.

The next two tables display resource utilization numbers when testing 6000 users with both templates. The first table shows results obtained with mail6.ntf:

The second table shows our numbers with Domino 7 users running the mail7.ntf template:

The tests showed CPU improvement of 31 percent running Domino 7 with the mail6.ntf template, and 26 percent running Domino 7 with mail6.ntf. Domino 7 improved CPU with both mail6.ntf and mail7.ntf compared to Domino 6.5.

Back to top

Conclusion

Lab measurements showed Domino 7 does more work with a given amount of CPU effort than previous releases. In addition, Domino 7 uses significantly less system resources for a given amount of work. This savings was measured using a benchmark that more closely represents real customer environments than any benchmark has done before. The net result is a significant reduction in total cost of ownership using Domino 7.

Resources

• The first part of this article series, “Lotus Domino 7 server performance, Part 1: Lotus Notes client workloads," discusses Domino 7 performance results we obtained by simulating Notes client users.
  
  
• The second article in this series, “Lotus Domino 7 server performance, Part 2: Domino 7 performance for Domino Web Access users," discusses Domino 7 performance results we obtained by simulating Domino Web Access users.
  
  
• For additional details on the R6Mail workload used for all of the tests in this article, see the developerWorks: Lotus article, "The New Domino 6 NotesBench workloads: Heavier by request!"
  
  
• You can also read more about Server.Load and the R6Mail workload in the Domino Administrator Help.
  
  
• For more information about all the features in Lotus Notes 7, see the developerWorks: Lotus article, "New features in Lotus Notes and Domino Designer 7.0."
  
  
• And for more information about new features in Lotus Domino 7, see the article, "New features in Lotus Domino 7.0."
  
  
• See the Lotus Notes/Domino product page for more information about the Lotus Notes and Domino products.
  
  
• Get involved in the developerWorks community by participating in developerWorks blogs.
  
  

About the authors

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• The Enterprise Mail workload benchmark
• General results
• Conclusion
• Resources
• About the authors
• Comments

Next steps from IBM

• Try: Lotus Notes and Domino Administrator
• Demo: Lotus Notes 8 composite applications
• Webcast: Modernization approaches to reduce the total cost for multi-platform enterprise applications
• Briefing: Smarter collaboration with IBM Lotus solutions
• Buy: Lotus Notes and Domino Administrator

My developerWorks community

Dig deeper into Lotus on developerWorks

Calling all developers

Download no charge Lotus Domino Designer

Special offers