In a previous LDD Today Performance Perspectives column, "Assessing the impacts of new Transaction Logging features," we previewed Domino 6 transaction logging and how it outperforms R5 transaction logging. Transaction logging, first added in Domino 5, provides added reliability and availability to server data. It does this by writing all database changes in a special format to a transaction log file in addition to storing those changes in memory. These changes are later made to the appropriate databases. This gives you:
- Faster database recovery and better server availability in the event of a crash-without data loss
- Backup solutions that allow you to restore a database to the last database update
By exploiting Domino 6 transaction logging (and its new view logging feature), you can improve your system's responsiveness, while providing more thorough disaster recovery. Enhanced logging, along with other Domino 6 performance improvements, helps increase the number of users supported by each of your servers-which in turn is an excellent way to decrease your site's total administrative and equipment costs.
In this month's column, we dive deeper into Domino 6 transaction logging, presenting and explaining results we obtained testing performance on different protocols and platforms. We also show how you can improve transaction logging performance in your own environment. Our goal is to help you better understand transaction logging from a "performance perspective" and to aid you in planning your server configurations.
This column assumes that you're an experienced Domino administrator.
A quick review of Domino 6 transaction logging
Before we jump into all the graphs and numbers, let's briefly review the new and improved features in Domino 6 transaction logging and how they provide better performance. These features include:
- View logging
View logging is a new feature in Domino 6. It allows transaction logging of view elements. This gives you faster access to a database with complex views after a system error by preserving the view information in the transaction log and eliminating the need for a view rebuild. View logging does have a small performance cost, so you should enable it only on complex views. This feature is enabled per database view through Domino Designer. By default in Domino 6, view logging is enabled on the $User view of the Domino Directory.
- Lock manager
The lock manager has been enhanced in Domino 6. Lock manager now locks only the database elements required to perform a function. (In R5, the whole database was locked.) This allows multiple simultaneous operations on a database (for example, read access while doing writes).
- Flush algorithm
This is another feature enhanced in Domino 6. The flush algorithm now flushes the buffers (in other words, writes data to the databases) more frequently. This produces shorter write locking interruptions to the databases.
Now let's look at our performance test setup, described in the next section.
Server test configurations
We conducted our study of Domino 6 transaction logging performance on three platforms: Windows 2000, Solaris 8, and AIX 5.1. The systems we used for testing were not state-of-the-art. Instead, we tried to represent typical computing power available to most Domino customers.
We performed our Windows 2000 tests on a server running Domino with one partition. Client PCs, running NotesBench workloads, simulated users running against the Domino server. The server ran with default settings, except where noted.
All Windows 2000 tests used Netfinity 8500Rs with eight 550 Mhz P III Xeon processors and 4 GB of memory. Windows 2000 and Domino executable files shared a 9 GB drive that was part of a Raid array in a Raid 1 configuration. User mail databases were distributed evenly across four Raid 0 configured arrays that were divided among three SCSI adapters. The data disk configuration was large enough not to be a bottleneck in any test. Network access was through a single 100 MB ethernet adapter running in full duplex mode. The operating system was Windows 2000 Advanced Server with SP2 applied. The pagefile resided on a Raid 0 array. Transaction log files were on a dedicated drive set up in a Raid 0 configuration.
The limiting resource for the iNotes test was CPU, and for NRPC tests, it was memory. (Note that the maximum amount of memory available to Domino running under Windows 2000 is 2 GB.) To help alleviate the load on these resources, we did the following:
- For our iNotes Web Access tests, we reduced the value of the Number of active threads field in the Internet Protocols - HTTP tab of the Server document from 200 to 64. We also set the Maximum Cached Users field in the Internet Protocols - Domino Web Engine tab to 5050, and the Cached User Expiration Interval field in this tab to a number longer than the test duration.
- For our NRPC tests, we set the NSF Buffer Pool to 400 MB by adding the variable NSF_Buffer_Pool_Size_MB=400 to Notes.ini. Also, for the NRPC tests, we added the Notes.ini variables Server_Pool_Tasks=100 and Server_max_concurrent_trans=100 to ease a task processing bottleneck at the high end of the simulated users. (We could set these parameters higher without causing problems because we had plenty of excess CPU bandwidth.)
All Solaris tests ran on a Sun Enterprise 4500 server, equipped with 12 CPUs (400 Mhz Ultra Sparc II) and 12 GB memory. This system was attached to the network with a single 100 MB ethernet adapter running in full duplex mode. The Solaris 8 operating system and Domino executable files were stored on three single 9 GB internal SCSI disks. User databases were evenly spread across four A1000 arrays, each on a separate SCSI adapter. The A1000s each contained 12 drives (9 GB each) configured as Raid 0. A single instance of Domino (no partitions) was used in all cases.
During each test, the server was loaded by a group of driver workstations, all running the NotesBench test tool to simulate typical user actions.
For our Domino on AIX tests, we used an S80 computer with 12 (400 Mhz) CPUs and 16 GB of RAM. The operating system was AIX version 5.1 with patches at the Domino 6 published levels. We configured hard drives as follows: a single 9 GB internal drive for the operating system and 32 drives (9 GB each) set up as two Notes Data directories. These drives were part of a Raid 0 array with two SSA Controllers using 16 MB Fast_Write cache and connected to the LAN via a 100 MB ethernet full-duplex connection.
Performance test results
In this section, we review the results of our Domino 6 with transaction logging testing. In these tests, we used the NotesBench R5Mail, R6Mail, and R6iNotes workloads.
As mentioned previously, our test servers weren't necessarily state-of-the-art. But we did do some optimizing for each test, removing CPU and disk bottlenecks where possible. We also limited the Domino tasks to only those necessary for the test being run. This allowed us to focus solely on Domino and/or platform restrictions, thus providing a good basis for comparison. But as with any test results like this, our numbers don't necessarily reflect those you may experience in a real-world environment. Furthermore, by no means do we intend to compare the relative merits of the platforms we tested or to recommend one platform over another. The configurations used for each of the platforms were sufficiently different to render such efforts an "apples-and-oranges" comparison.
Nevertheless, we feel these results will give you a good understanding of the cost of running transaction logging in your environment. As previously mentioned, there is a cost to use this option-it consumes small amounts of CPU, disk, and memory resources. But we feel this cost is more than worth it because you'll now benefit from better crash and data recovery.
For comparison, we included some R5 data in these results.
New lock manager and flushing algorithm performance
We ran the NotesBench R5Mail workload test, simulating NRPC (Notes Remote Procedure Call) Notes mail users on a Windows 2000 R5 server with transaction logging enabled. This let us emulate what many customers with busy systems experience in R5. The following illustration displays our results:
Figure 1. Domino 5 running R5Mail workload with transaction logging on
The response time line starts off well, but spikes during a transaction log flush period. A transaction log flush is the process of writing data to the databases. In R5, log flushing locks the databases from all other access until the write is finished. As more users are added producing more transactions, these flush periods become more frequent and cause slower response time (because the databases are locked more often). As we approached 6,000 test users, we reached a state of almost constant flushing which resulted in degraded response.
We then ran Domino 6 with log flushing on the same server running the same R5Mail workload:
Figure 2. Domino 6 running R5Mail workload with transaction logging on
As you can see, response time in the Domino 6 test was significantly better, thanks to the new flushing algorithm. The databases can still be accessed during a transaction log flush, so there are no long response spikes. This results in much faster and more consistent user response time, all the way up to our maximum load of 10,000 simulated users. In fact, this test indicates that for our simulated NRPC (Notes Remote Procedure Call) Notes mail users on Windows 2000, Domino 6 consumes 39 percent less CPU than R5 and can support 3,000 more users.
Interestingly enough, where to put the transaction log files is extremely important and can have a major impact on server performance. (Note that the transaction log files do not need to be relegated only to an internal drive.) You should use whatever disk configuration on your system gives you the highest throughput. Another thing to keep in mind is the data integrity of the transaction log files. If these files are destroyed, some data recovery may be impossible, resulting in loss of data (not a good thing). So you may decide to take a couple of drives in a Raid array and configure them in a 0+1 configuration. You can then access the drives through a high-speed channel that has no (or minimal) additional disk traffic.
The following two graphs show results we obtained performing two tests in which we ran R6Mail NotesBench workloads on Solaris. Both tests ran on the same server with transaction logging configured in circular mode with the Favor Runtime option selected. The first chart shows a run where the transaction log files were installed on an internal SCSI drive:
Figure 3. R6Mail workload on Domino 6 running on Solaris - internal SCSI drive
This configuration supported 3,000 users with a sub-second response time:
|Number of users||1,000||2,000||3,000||4,000|
|Response time (seconds)||0.145||0.218||0.322||2.191|
The second chart shows the results of a test run where the transaction log files were moved to a disk on a Raid array. This disk had a much higher throughput than the SCSI configuration:
Fiure 4. R6Mail workload on Domino 6 running on Solaris - Raid array drive
This configuration supported 8,500 users with sub-second response:
|Number of users||1,000||2,000||3,000||4,000||5,000||6,000||7,000||8,000||8,500|
|Response time (seconds)||0.044||0.048||0.056||0.064||0.072||0.085||0.108||0.273||0.844|
These two tests demonstrate the importance of selecting the correct drive to store transaction log files. With a slow drive, the number of simulated mail users that the server supported was reduced by more than half. These tests also show that using drives on a Raid device (if configured properly) have no detrimental effect. Bear in mind, however, you should never put transaction log files on a drive with other files. This will only slow down the transaction logs throughput.
This section discusses tests we ran using R6Mail and R6iNotes workloads on Domino servers running on Windows 2000.
R6Mail on Windows 2000
The following chart shows the results of running the R6Mail NotesBench workload test on the following server configurations:
- R5.0.10 with transaction logging enabled
- Domino 6 with transaction logging enabled
- Domino 6 with transaction logging disabled
We found that Domino 6 transaction logging consumes more resources (about 21 percent more CPU than running Domino 6 without transaction logging), but still allowed us to support 10,000 simulated users. R5.0.10 with transaction logging enabled only reached 7,000 users. With both running 7,000 simulated users, Domino 6 consumed about 39 percent less CPU than R5.0.10:
Figure 5. R6Mail workload on Windows 2000 - total CPU utilization
R6iNotes on Windows 2000
Next, we ran the R6iNotes workload, simulating iNotes Web Access mail users. In this test, we again compared Domino 6 with and without transaction logging enabled. We also ran the R6iNotes workload on R5.0.10, but unlike our test described in the previous section, we disabled transaction logging. We did this because there has been substantial performance improvement in Domino 6 iNotes Web Access:
Figure 6. R6iNotes workload on Windows 2000 - total CPU utilization
Domino 6 with transaction logging supported 3,265 simulated users with sub-second response time, while Domino 6 without transaction logging attained 3,520 users. The issue here is CPU resources. Transaction logging uses more CPU resources, which can be expected because it's doing more work. Comparing Domino 6 with transaction logging enabled to R5 without transaction logging at the 2,255 simulated user mark, we see Domino 6 utilizes approximately 27 percent less CPU. Also Domino 6 with transaction logging supports about 1,010 (45 percent) more simulated users than R5.0.10 with transaction logging disabled.
Next, we ran the tests using R6Mail and R6iNotes workloads on Domino servers running on Solaris 8.
R6Mail on Solaris
On the Solaris platform running the R6Mail workload, we again compared Domino 6 with and without transaction logging enabled and R5.0.10 with transaction logging enabled:
Figure 7. R6Mail workload on Solaris - total CPU utilization
Transaction logging consumed more resources in Domino 6. At 8,000 simulated users, Domino 6 with transaction logging used about 29 percent more CPU than without transaction logging. However, we noticed that in this test there were signs of a disk bottleneck on the mail database drives. Therefore, we concluded that if we added another array for the mail databases, performance would have been better. Even with that in mind, comparing Domino 6 and R5.0.10 both with transaction logging enabled at the 8,000 simulated user mark shows that Domino 6 utilizes around 33 percent less CPU.
R6iNotes on Solaris
We then compared running the R6iNotes workload on Solaris using Domino 6 with and without transaction logging and R5.0.10 without transaction logging:
Figure 8. R6iNotes workload on Solaris - total CPU utilization
Again, there is an increase in CPU utilization with Domino 6 with transaction logging enabled versus Domino 6 without transaction logging (about 12 percent at 3,500 simulated users). But if we compare Domino 6 with transaction logging enabled to R5 without transaction logging with both running 2,500 simulated users, we see that Domino 6 utilizes approximately 47 percent less CPU (almost half). Also Domino 6 with transaction logging supported about 1,000 more users than R5.0.10 with transaction logging disabled. This leads us to believe you can expect better CPU utilization and support more users running Domino 6 with transaction logging enabled than you currently see with R5 without transaction logging.
For our final test, we ran R6Mail and R6iNotes workloads on Domino servers running on AIX 5.1.
R6Mail on AIX
As with Windows 2000 and Solaris, we ran the R6Mail workload on AIX to compare Domino 6 with and without transaction logging enabled and R5.0.10 with transaction logging enabled:
Figure 9. R6Mail workload on AIX - total CPU utilization
In Domino 6, transaction logging uses a little more resources. At 9,000 simulated users, Domino 6 with transaction logging utilized about 20 percent more CPU than without transaction logging. But if we compare Domino 6 and R5.0.10 both with transaction logging enabled at the 8,000 simulated user mark, Domino 6 utilizes around 35 percent less CPU.
R6iNotes on AIX
In this set of tests, we compared running the R6iNotes workload on AIX using Domino 6 with and without transaction logging and R5.0.10 without transaction logging:
Figure 10. R6iNotes workload on AIX - total CPU utilization
The increase in CPU utilization is small with Domino 6 with transaction logging enabled versus Domino 6 without transaction logging (approximately 13 percent at 6,000 simulated users). But if we compare Domino 6 with transaction logging enabled to R5 without transaction logging both at 3,000 users, we see that Domino 6 utilizes about 54 percent less CPU. Also, Domino 6 with transaction logging supports around 3,000 more simulated users than R5.0.10 with transaction logging disabled-approximately double!
When reviewing our test results, it's clear Domino 6 transaction logging has been improved over R5. We believe you'll experience better CPU performance and can support more users on your current servers on all platforms. This follows the common theme with Domino 6-enabling you to do more with less. As we stated in the beginning of this article, transaction logging has a small overall CPU cost, as does nearly all important new features. If you're familiar with transaction logging from R5, you're already aware of this. But you also know the benefits of higher reliability and availability transaction logging gives you. What we have done in Domino 6 is to minimize the performance impact as much as possible, and we feel that the results are evident: Transaction logging on Domino 6 is much more efficient than on R5!