IBM Lotus Domino with BlackBerry Enterprise Server: Performance best practices

Aaron Payne (aaron@uk.ibm.com), Senior IT Specialist, IBM

Aaron Payne is a Senior IT Specialist at IBM UK Ltd and is working within Lotus software. Aaron is a Lotus collaboration infrastructure consultant working as part of Premium Services with some of IBM’s biggest customer deployments of Lotus collaboration products. Aaron is also co-lead for the IBM Software Services for Lotus Worldwide Domino Community within IBM. You can reach Aaron at aaron@uk.ibm.com.

Summary: This article is designed to act as a guide and reference paper for configuring BlackBerry Enterprise Server with IBM® Lotus® Domino®. It focuses on the load implications of running BlackBerry Enterprise Server within a Lotus Domino mail environment. It is designed to highlight areas where this load can be controlled and minimized.

Date: 07 Oct 2008
Level: Intermediate
Activity: 3342 views
Comments:

The use of BlackBerry Enterprise Server (BESW) in mail and messaging environments is increasing. More companies are exploring the benefits of mobile working and investing in technologies that mobilize their workforces. This technology is no longer restricted to mail, calendar, and contact synchronization. The use of mobile instant messaging is becoming a core part of the mobile toolkit. BlackBerry Enterprise Server allows companies to mobilize the core technology functions within their business. Customers running IBM Lotus Domino and IBM Lotus Sametime® environments can quickly and easily extend the functionality provided by these products to a mobile device.

The introduction of BlackBerry Enterprise Server inevitably consumes resources within your mail environment, particularly as the environment scales up to hundreds or thousands of users. With careful planning and the use of the latest versions of the software products, you can reduce and control the load burden on the mail.

This article is aimed at Lotus Domino administrators who either are planning on implementing BlackBerry Enterprise Server or need to better understand the interaction between IBM Lotus Domino and BlackBerry Enterprise Server.

Included in this article is a testing section that shows how product improvements in BlackBerry Enterprise Server and IBM Lotus Domino reduce the load burden on a mail environment. These product improvements are set to continue as both software platforms evolve and IBM continues to collaborate and work with Research in Motion (RIM), the maker of BlackBerry Enterprise Server.

Performance improvements in BlackBerry Enterprise Server 4.1.2 and later

BlackBerry Enterprise Server version 4.1.2 was the first version that contained a number of performance-related enhancements jointly produced by RIM and IBM.

The performance enhancements in BlackBerry Enterprise Server 4.1.2 and later work with a new set of API calls that were introduced by IBM in Lotus Domino 7.0.2. Installing BlackBerry Enterprise Server 4.1.2 using a Lotus Domino server earlier than 7.0.2 does not see performance improvements from these new APIs.

These new API calls relate to the way BlackBerry Enterprise Server interacts with the Lotus Domino server to determine when a database change has occurred. Mail databases are polled by BlackBerry Enterprise Server every 20 seconds by default. The new API calls provided with BlackBerry Enterprise Server 4.1.2 and Lotus Domino 7.0.2 improve the efficiency of this polling cycle and reduce the number of open database requests to the Lotus Domino server.

Figures 1 and 2 show the improvements to the polling interval from version 4.1 and earlier version compared with version 4.1.2.

Figure 1. BlackBerry Enterprise Server polling in version 4.1 and earlier

BlackBerry Enterprise Server 4.1 and earlier uses an API call NSFDbModifiedTime. This API is called for each database every 20 seconds (default). BlackBerry Enterprise Server takes a DBHANDLE that requires that the database has been opened first. A database open operation is required every time BlackBerry Enterprise Server polls a mail database, even if no change has occurred. This requirement can result in large numbers of DBOpens, which can affect the DBCache and overall mail server performance. This method also required multiple API calls to complete the transaction.

Figure 2. BlackBerry Enterprise Server polling in 4.1.2 and later with Lotus Domino 7.0.2

In BlackBerry Enterprise Server 4.1.2 and later in conjunction with Lotus Domino 7.0.2, BlackBerry Enterprise Server uses a new API call, NSFDbModifiedTimeByName, which is called for each database every 20 seconds (default). This API uses the name of a database rather than a DBHANDLE, therefore removing the need to open every database even when a change did not occur. With the DbName call, DBDirman can return the appropriate results without requiring BlackBerry Enterprise Server to open the database.

BlackBerry Enterprise Server polling test scenarios

To further reinforce the benefit of upgrading BlackBerry Enterprise Server to version 4.1.2 and later, IBM carried out a series of tests to prove and quantify the overall saving that BlackBerry Enterprise Server 4.1.4 would provide based on a standard production configuration.

RIM has published performance benchmarks. This test cycle was designed to show the significant drop in the number of DBOpen operations on the mail server from BlackBerry Enterprise Server 4.0.x to BlackBerry Enterprise Server 4.1.2 and later.

The test environment was configured using BlackBerry Enterprise Server 4.0.6 with both mail databases and personal address books hosted on the mail server. This test did not use over the air (OTA) synchronization of contact data. This feature was introduced in later BlackBerry Enterprise Server versions, and this test was designed to show how a legacy configuration would differ if it were upgraded in place. Further performance gains could be realized by moving to OTA contact synchronization, which is discussed in more detail later in this article.

The aim of the test was to simulate and quantify the DBOpen load to mail and personal address book databases. It was not possible to simulate full mail flow to the handheld device as no physical BlackBerry devices were connected to the test environment. BlackBerry Enterprise Server was configured with 300 accounts; each account was associated with an individual mail database and personal address book (PNAB).

Test environment

The test environment consisted of a Lotus Domino 7.0.2 FP1 server with 300 BlackBerry Enterprise Server-enabled mail accounts and associated personal address books.

Mail-01/ACME: IBM System p® Server – 300 users, 300 PNABs.
BES-01/ACME: Dedicated Microsoft® Windows® BlackBerry Enterprise Server server.

The following conditions were observed for each test:

Activity collection was configured on the Lotus Domino mail servers.
At the start of each test, the log.nsf and any previous activity.nsf databases were removed.
BlackBerry Enterprise Server logging was set to Level 6 under version 4.0.6 and Level 7 under version 4.1.4.
Performance Explorer (PEX) trace data was captured at an operating system level for the mail servers.
Each test lasted for a one-hour period.

Tests were run using the environment at BlackBerry Enterprise Server 4.0.6 running against a Lotus Domino 7.0.2 mail server. After the test cycle, all performance data was gathered and verified as valid. Then the BlackBerry Enterprise Server was upgraded to version 4.1.4, and the test was repeated using the same conditions and parameters.

Test results

The test results data shown in figure 3 was obtained from two primary sources. The activity logging database on the mail server is used to measure and record the activity on the mail server from the BlackBerry Enterprise Server server. The PEX data was run on the pSeries® mail server to record the transaction processor. The term transaction refers to the total number of transactions driven from BES-01/ACME.

This test results, detailed in figure 3, showed the following:

BES-01/ACME at BlackBerry Enterprise Server 4.1.4 drove 219,000 transactions using Mail-01/Acme.
BES-01/ACME at BlackBerry Enterprise Server 4.0.6 drove 499,000transactions using Mail-01/Acme.
A reduction of 57 percent occurred in the total number of transactions driven on the mail server.
The total number of database open operations (DBOpen operations) dropped from 104,000 to 37,000, for a drop of 64 percent.

Figure 3. BlackBerry Enterprise Server 4.1.4 and 4.0.6 polling comparison

The comparison in figure 3 shows a drop of 50 percent in the number of DBOpen operations with BlackBerry Enterprise Server 4.1.4 compared with BlackBerry Enterprise Server 4.0.6. During the test period with BlackBerry Enterprise Server 4.0.6, DBOpen operations peaked at just under 12,000, compared with 6,500 in BlackBerry Enterprise Server 4.1.4.

Figure 4 shows the processor gains for the nBES task on the BES-01/ACME server.

Figure 4. Processor utilization of nBES task on BES-01/ACME
CPU Utilization of nBES task on BES-01/ACME

Figure 4 clearly shows the processor gains on BlackBerry Enterprise Server following an upgrade to version 4.1.4.

Test summary

The test results have shown that clear performance benefits can be gained by upgrading to BlackBerry Enterprise Server 4.1.4 or later. IBM recorded a drop in transaction count on the mail server and a processor reduction on BlackBerry Enterprise Server. Testing has shown that an upgrade to BlackBerry Enterprise Server 4.1.4 benefits both the Lotus Domino mail environment and the BlackBerry environment.

Thread memory allocation

Additional work has been done in BlackBerry Enterprise Server 4.1.4 to lower the memory thread allocation. This work lowers the amount of memory each BlackBerry Enterprise Server thread consumes from 1MB to 192KB. A BlackBerry Enterprise Server with 200 configured threads would consume 200MB of memory just to create and maintain the thread pool. In version 4.1.4 or later, this consumption drops to 38MB. This change can be seen in an NSD output taken from two BlackBerry Enterprise Servers , one running version 4.1.3 and one running version 4.1.4. See listings 1 and 2.

Listing 1. NSD from 4.1.3 NBES Task

############################################################
### thread 2/19: [ NBES: 17fc: 0f24] 
### FP=00f4feec, PC=7c82ed54, SP=00f4fe7c
### stkbase=00f50000, total stksize=8192, used stksize=388
############################################################
[ 1] 0x7c82ed54 ntdll+191828 (78,c8,0,0)
[ 2] 0x77e6ba42 KERNEL32+178754 (78,c8,5afdbe,c8)
[ 3] 0x005b9245 nbes+1806917 (0,0,0,0)
Invalid stack frame detected: Invalid frame pointer (BP): 0

89 00e50000 00e50000 1012.0K RESERVE private -C---- thread 0f24 stack (reserved) 

(for those not familiar with NSD output….this is the memory required to create and 
maintain the thread pool?)

90 00f4d000 00e50000 4.0K COMMIT private GCrw-- thread 0f24 stack (guard)
91 00f4e000 00e50000 8.0K COMMIT private -Crw-- thread 0f24 stack

Listing 2. NSD from 4.1.4 NBES Task

############################################################
### thread 2/16: [ NBES: 0870: 1900] 
### FP=00defea4, PC=7c82ed54, SP=00defe34
### stkbase=00df0000, total stksize=16384, used stksize=460
############################################################
[ 1] 0x7c82ed54 ntdll.KiFastSystemCallRet+0 (94,c8,0,7ea360)
[ 2] 0x77e6ba42 KERNEL32.WaitForSingleObject+18 (94,c8,5ce9a2,c8)
[ 3] 0x005d64a5 nBES.Event::Wait+21 (c8,0,7ea360,deffb8) @C:\ABS10\
Components\NotesBES\4.1.4\LocalBuild\wutility\src\event.cpp(125)
[ 4] 0x005ce9a2 nBES.DebugLogger::ThreadProc+498 (3a4bb8,6a94da,7ea360,0) 
@C:\ABS10\Components\NotesBES\4.1.4\LocalBuild\wutility\src\debug.cpp(2706)
[ 5] 0x005d63b5 nBES.Thread::ThreadProcStub+37 (7ea360,0,0,3a4bb8) 
@C:\ABS10\Components\NotesBES\4.1.4\LocalBuild\wutility\src\Thread.cpp(286)
[ 6] 0x006a94da nBES.__beginthreadex+202 (3a4bb8,0,0,3a4bb8)
[ 7] 0x77e6608b KERNEL32.GetModuleFileNameA+235 (6a947b,3a4bb8,0,c8)

89 00dc0000 00dc0000 172.0K RESERVE private -C---- thread 1900 stack (reserved)
90 00deb000 00dc0000 4.0K COMMIT private GCrw-- thread 1900 stack (guard)
91 00dec000 00dc0000 16.0K COMMIT private -Crw-- thread 1900 stack

PIM and journal synchronization

BlackBerry Enterprise Server provides two ways to synchronize your personal address book and journal with your handheld device. The preferred method with older versions of BlackBerry Enterprise Server was to locate a replica copy of your personal address book or journal on the mail server. BlackBerry Enterprise Server would then poll these databases for changes and synchronize with the handheld device. Later versions of BlackBerry Enterprise Server work by storing contacts within the mail database and synchronizing over the air (OTA).

As we have seen through the testing with mail databases, a significant improvement has been made to the way in which BlackBerry Enterprise Server handles DBOpen operations with version 4.1.2. These improvements also extend to both the journal and personal address book databases. The BlackBerry Enterprise Server handles these databases in the same way that it would a mail database. BlackBerry Enterprise Server polls the database for changes and updates and synchronizes them to the handheld device.

As previously noted, upgrading to BlackBerry Enterprise Server version 4.1.2 reduces the number of database open operations required to support both the journal and address book, resulting in a reduced load on the mail server.

Two other solutions to personal address book and journal synchronization also exist that can further reduce the load on the mail server:

In large enterprise environments, create a new, separate infrastructure to store the personal address book and journal databases. This approach removes all DBOpen and synchronization load from the mail server.
Synchronize over the air.

Starting with BlackBerry Enterprise Server 4.1.2 and Lotus Domino 7.0.2, personal address book and journal documents are stored in hidden views within the mail file, thus eliminating the need for server-based replica copies. Synchronizing over the air is the preferred method as all address book and journal entries are stored in the mail database.

Changes to the personal address book entries are synchronized using a new feature on the replicator tab that was introduced in Lotus Notes® version 8. See figure 5.

Figure 5. The Synchronize Contacts feature

Changes to the journal database are synchronized manually using a menu option in the mail database. Choose Actions – Domino Web Acess – Synchronize Notebook as shown in figure 6.

Figure 6. Menu option for manual synchronization

These two features add the documents from the personal address book and the journal into hidden views in the mail database. BlackBerry Enterprise Server then detects changes to these documents and synchronizes them to the handheld device. Using OTA synchronization means that all the work carried out by the BlackBerry Enterprise Server using the mail server is done using a single DBOpen request if a database change has occurred.

In an earlier BlackBerry Enterprise Server environment, where users have mail, personal address book and journal on the mail server, three DBOpen commands would occur for each user per poll (default 20 seconds). OTA synchronization reduces this count to just one DBOpen command. The enhancements in BlackBerry Enterprise Server 4.1.2 with Lotus Domino 7.0.2 reduce this further to a single DBOpen command when a database change has occurred.

When using BlackBerry Enterprise Server 4.1.2 or later and LotusDomino 7.0.2 or later, synchronizing over the air is the preferred method as all address book and journal entries are stored in the mail database.

BlackBerry Enterprise Server threading model

BlackBerry Enterprise Server uses threads to complete certain tasks. Interaction between BlackBerry Enterprise Server and a Lotus Domino mail server is handled through threads and thread pools. The number of threads per BlackBerry Enterprise Server and the ratio of BlackBerry Enterprise Server to mail servers have an implication on the total loading a BlackBerry Enterprise Server environment places on the mail server.

This topic is discussed in length in an article published by Research in Motion, “Threading model for BlackBerry Enterprise Server software version 4.0 and 4.1 for IBM Lotus Domino.”

Do not change from the default BlackBerry Enterprise Server threading model. There might be circumstances in which customers have a technical need to change the threading model, and they should consider the net effect on the mail environment of such a change.

BlackBerry Enterprise Server and Lotus Domino server deployment considerations

This section covers some best-practice guidelines on how to deploy BlackBerry Enterprise Server with Lotus Domino. Careful planning at the start of the deployment can reduce the overhead of ongoing maintenance and streamline future upgrades.

As discussed in previous sections, BlackBerry Enterprise Server connects to Lotus Domino using a threading model. Each BlackBerry Enterprise Server server opens a persistent number of threads to the mail server. Depending on the thread model in use, fixed or dynamic, this model affects the amount of resources BlackBerry Enterprise Server consumes on the Lotus Domino server.

Consider an environment in which BlackBerry Enterprise Server has a fixed thread pool of 50 threads and a total of five BlackBerry Enterprise Server servers are connecting to a mail server. We find that just to maintain the persistent connections from BlackBerry Enterprise Server we consume 250 threads on the mail server.

In the majority of deployments, the threading model remains unchanged from the default, and for nearly all customer environments this approach is recommended. If you have a technical requirement to change the threading model, then consider the net effect on the mail server.

As a guide for performance optimizations, minimize the ratio of BlackBerry Enterprise Server to mail server. This guide holds particularly true in larger deployments where each BlackBerry Enterprise Server could hold more than 500 to 800 users. Also consider that the BlackBerry Enterprise Server load is constant 24x7, but user load occurs in peaks and troughs.

In a smaller deployment, where the total number of BlackBerry users as a percentage of the total user base is low, it is common have a single BlackBerry Enterprise Server serving multiple mail servers.

Careful planning on the deployment and ratio of BlackBerry Enterprise Server to the mail server can also streamline future upgrades, particularly where one version has a dependency on another. For customers planning to move to Lotus Domino 8, this upgrade requires an upgrade to BlackBerry Enterprise Server 4.1.5 or later. As an example, if a single mail server serves only two BlackBerry Enterprise Server servers, then the BlackBerry Enterprise Server servers can be upgraded at the same time as the mail server. If the mail server was accessed by six BlackBerry Enterprise Server servers, this deployment could influence your upgrade sequence and priorities.

In an environment where BlackBerry Enterprise Server connects to multiple mail servers and each mail server supports multiple BlackBerry Enterprise Server installations, server upgrades become more complex and time-consuming. In some instances, the BlackBerry Enterprise Server environment requires an upgrade before the mail server.

Conclusion

The aim of this article was to explore the different configuration and deployment options that are open to system administrators who deploy and support this technology. This article has shown that IBM and RIM have jointly collaborated on the latest versions of their products to streamline the interaction and lower the BlackBerry Enterprise Server impact on the Lotus Domino mail environment.

Testing carried out to create this article shows areas where the overall impact of BlackBerry Enterprise Server on a Lotus Domino email infrastructure can be reduced. The most obvious gain results from an upgrade to Lotus Domino 7.0.2 or later and BlackBerry Enterprise Server 4.1.3 or later (version 4.1.3 includes changes to the thread memory allocation).

Carefully considering the deployment of BlackBerry Enterprise Server with Lotus Domino also helps streamline the interaction and lower the performance cost. Limit the ratio of BlackBerry Enterprise Server to Lotus Domino servers. This recommendation is particularly true for large enterprise deployments where lowering thread allocation and polling overhead can have a significant effect on performance.

Over-the-air synchronization of both personal address book and journal remove the extra DBOpen and synchronization workloads from the mail server. If a change to the databases has occurred, the change is synchronized to the handheld device using a single DBOpen command. New tools in the Lotus Notes 8 client make OTA synchronization more user friendly.

Resources

Learn