Editor's note: Know a lot about this topic? Want to share your expertise? Participate in the IBM Lotus software wiki program today.
|IBM Lotus Notes and Domino wiki|
The Lotus Domino 8.0 and 8.5 server versions are focused on delivering improvements in overall storage. These versions include technology that can significantly reduce the total cost of ownership (TCO) for your Lotus Domino environment. With these versions, Lotus developers carefully picked what are often the highest value targets for reducing TCO: disk space, storage costs, I/O operations, and the cost of operational maintenance.
These new enhancements in Lotus Domino 8.0 and 8.5 can improve the overall cost of ownership in several ways:
- Fewer bytes written to disk. Users can achieve huge reductions in the disk space used to provide the same level of service, making it possible to do the same job but with fewer disks on your servers.
- Fewer disk I/O operations. In addition to writing fewer things as shown in the preceding item, some disk operations can be eliminated altogether for certain use cases.
- Attachments separated in a separate storage space. Attachments are typically excellent candidates to use lower-cost disk storage.
- Operational maintenance improvements. Although Lotus Domino environments typically have a low cost of operation, improvements in backups, storage, and administration make it possible to reduce maintenance even more by doing the following:
- Reducing the overall amount of data stored, which reduces backup costs
- Reducing backup times because the separate storage space used for attachments can be incrementally backed up
- Reducing view rebuild times
- Reducing the time needed to defragment (compact) Lotus Notes databases
Before we begin our discussion of how you can leverage these improvements in your own Lotus Domino environment, let's make sure that you have a good understanding of the components that make up a Lotus Notes database. A Lotus Notes database is composed of several different types of elements. These elements include the following:
- Design elements. The elements that display in Lotus Domino Designer: forms, views, agents, and so on. These elements typically have a fixed size, but that size can be significant (for example, the Lotus Notes 8.5 Mail template is almost 30 MB).
- Views. Lotus Domino saves the data in a view, for example, the inbox view, as a separate element in the Lotus Notes database. If a database has many views, especially views with sortable column headings, this structure can use a substantial amount of disk space.
- Documents. This element includes the body of a mail message, for example.
The new enhancements in Lotus Domino 8.0 and 8.5 act directly on these elements. In the next few sections, we help you understand each of these enhancements and show you how to leverage them so that you achieve the ultimate storage savings in your Lotus Domino environment.
As outlined earlier, there are four main elements in a Lotus Notes database that can affect overall storage on your Lotus Domino server: design elements, view-related data, document data, and attachments. Lotus Domino 8.0 and 8.5 provide enhancements that are geared directly toward these four elements.
Design elements define the views and forms that are used by a database and the database logic, validation rules, agents, LotusScript®, and so on. Even though Lotus Domino has the ability to separate design elements from individual Lotus Notes databases using the Single Copy Template feature, many users choose to bundle them into each copy of the database (which is also the default). This approach makes the databases self-contained and easier to manage. The amount of space taken up by the design elements of a database varies with the complexity of the application, but the amount of space can be significant.
Compressing the database design reduces the space needed to store these design elements. This reduction is a fixed amount of savings for each database. For example, the Lotus Notes 8.5 mail template as provided uses almost 30 MB of disk space per mail file. Using the design compression feature, this metric can be reduced to 11 MB of disk space, a savings of about 19 MB per mail file. This feature requires ODS 48 and is enabled through a database property. This feature is available for both clients and servers; it can also help provide quota relief.
Figure 1. Advanced properties – Enabling Design Compression
Every view in a Lotus Notes database stores a summary of the underlying document data that allows users to navigate quickly and easily through the documents. For example, the Inbox view in a mail file allows you to quickly see emails that you have received. In addition, the designer of a Lotus Notes view can improve the usefulness of the view by allowing users to sort different columns. Again, using the Inbox view example, this view allows you to sort by sender, subject, date, size, and more. For each of these sortable columns, Lotus Notes builds a language-specific sort index. These indexes take up disk space and processing time on the server when the views are rebuilt.
In Lotus Domino 8.0, you can specify for each sortable column in a view that the sort index is not rebuilt until it is needed or used and that it should be discarded after some period of time if it is not used. This feature requires ODS 48 and is enabled by the view designer using a column property on the database. This feature is available for both clients and servers and can also help provide quota relief. Figure 2 shows this feature.
Figure 2. View Column Properties – Defer Index Creation until first use
Data documents are the actual contents of most applications, for example, a mail message. Data document compression reduces the space needed to store your application's data. This feature was introduced in Lotus Notes and Domino 8.0.1 and also requires ODS 48. It is enabled by a database property, is also available for both clients and servers, and provides quota relief. See figure 3.
Figure 3. Advanced Properties – Enabling Data Document
The last new feature we discuss here, which was introduced in Lotus Domino 8.5 is Lotus Domino attachment and object services (DAOS). The Lotus Domino server employs DAOS to save significant space at the file level by sharing data identified as identical between databases (applications) on the same server. Document attachments are the first components to use the DAOS feature in Lotus Domino.
In databases that use DAOS, Lotus Domino no longer stores an attachment within the document itself. Instead, the DAOS-enabled server saves a single copy of the attachment in the DAOS repository and replaces the attachment with a reference to that DAOS object. If an attachment exists in multiple databases, the attachment is stored only once per server partition, and the reference counts are increased. Therefore, when an attached file is sent to multiple users on the same server, disk space usage is substantially reduced. As impressive as the reduction in disk space can be, there is an even greater potential cost savings associated with DAOS. DAOS can store attachments on a different device than the Lotus Notes databases themselves. In our internal email performance benchmark, DAOS accounts for less than 2 percent of the overall I/O (see the real-world examples that follow). Because of this separation, DAOS can use lower-cost disks without affecting overall response times.
It is also important to note that attachment consolidation is not limited to mail databases; it works on any Lotus Notes database on a DAOS-enabled server. This feature requires ODS 51 (Lotus Domino 8.5) and is enabled through a database property. This feature, shown in figure 4, is available for servers only.
Figure 4. Advanced Properties – Enabling DAOS
Prior to the gold release of Lotus Domino 8.5, the storage-savings features outlined previously were being implemented in two domains in IBM. This implementation was made so that the developers could gain a better understanding of how these features work and so that we could collect some performance metrics for some real-world Lotus Domino deployments. The following sections outline some implementation examples and document the positive results that were achieved.
IBM Global Services (IGS) runs the main production Lotus Domino domains in IBM. This implementation consists of 18 Lotus domains and more than 1000 servers worldwide with approximately 500,000 users. To test the storage-savings impact in this domain, three tests were performed. These tests and the results for these tests are outlined below.Test 1: Dedicated IBM Research mail server
The first server tested was a Lotus Domino mail server dedicated to the IBM Research division. This server houses 112 active user mail files and is running IBM AIX 184.108.40.206 in 64-bit mode Prior to upgrade or enablement of any of the preceding features, this server had 65.9 GB of NSF data in the Lotus Domino data directory.
After upgrading this server to Lotus Domino 8.5 and enabling DAOS, the results shown in table 1 were realized.
Table 1. Results for a dedicated IBM Research mail server
|NSF data in data directory||23.9 GB|
|NLO data in DAOS||25.3 GB|
|Data directory footprint||63 percent reduction in storage|
|Overall footprint||25 percent reduction in storage usage|
|I/O||8 percent reduction in average I/O volume|
1 percent increase in average I/O per second
|Relative I/O rates||DAOS disk has 1.3 percent of the I/O|
As mentioned previously, DAOS benefits are not limited to mail files. Therefore, the second server tested was a Lotus Domino application server also used by the IBM Research division. This system is running Microsoft® Windows 2003 in 32-bit mode, with 17.29 GB of application data in the Lotus Domino data directory prior to the implementation of DAOS.
After upgrading this server to Lotus Domino 8.5 and enabling DAOS, the results detailed in table 2 were seen.
Table 2. Results for a Lotus Domino application server
|NSF data in data directory||9.85 GB|
|NLO data in DAOS||5.28 GB|
|Data directory footprint||43 percent reduction in storage|
|Overall footprint||12 percent reduction in storage usage|
|Relative I/O rates||Not measured|
The third test was also run on a mail server operated by IBM Global Services, but one with more active user mail files that was not dedicated to IBM Research. As such, this server provides a much more indicative view of the average mail server. This system ran AIX 220.127.116.11 in 64-bit mode with 70.9 GB of data in the Lotus Domino data directory.
After upgrading this server to Lotus Domino 8.5 and enabling DAOS, the results detailed in table 3 were seen.
Table 3. Results for a Lotus Domino mail server
|NSF data in data directory||40.3 GB|
|NLO data in DAOS||15.4 GB|
|Data directory footprint||43 percent reduction in storage|
|Overall footprint||21.5 percent reduction in storage usage|
|I/O||12.99 percent reduction in average I/O volume|
16 percent increase in average I/O per second
|Relative I/O rates||Not measured|
The last deployment covered in the initial testing numbers was done outside of IBM at a Business Partner. The target server for this test housed mail archive files for the entire company. Active mail files were stored on a primary server, and older documents were migrated to mail archives on a dedicated archive server. Although direct user activity to this archive server is low, the volume of data involved is fairly large.
In this test, a sample of 89 mail archive files was DAOS-enabled. This system ran IBM OS 400®-V5R4M0 with 87.3 GB of data in the Lotus Domino data directory.
After upgrading this server to Lotus Domino 8.5 and enabling DAOS, the results listed in table 4 were seen.
Table 4. Results for an archive mail server
|NSF data in data directory||16.9 GB|
|NLO data in DAOS||334.6 GB|
|Data directory footprint||80 percent reduction in storage|
|Overall footprint||41 percent reduction in storage usage|
|Relative I/O rates||Not measured|
The numbers shown in table 4 represent only a selection of the IBM Business Partner's live data. The size of the current archive files exceeds 25 TB. If we extrapolate the 41 percent reduction in overall storage to their full archives, DAOS alone could save them more than 10 TB of disk space. Additionally, the 80 percent reduction in the data directory footprint can significantly reduce their backup volume, requiring less time for backups to complete.
As you can see from these examples, DAOS can significantly reduce the size of the data directory and, in many cases, reduce the overall size of disk storage. In many scenarios, DAOS can utilize lower-cost storage, which would significantly reduce storage costs. It is also important to note the savings in disk I/O rates in Test 1 and Test 3 after DAOS was enabled. Because disk I/O has been a limiting factor to Lotus Domino server performance for many customers, these reductions in I/O rates are extremely beneficial.
To help show how DAOS can benefit you before you enable it, a DAOS Estimator tool can outline overall savings across all servers in the domain. When this tool was run on the entire IBM domain Lotus Domino servers, we saw the results shown in table 5.
Table 5. Results of running the DAOS estimator tool
|NSF data in data directory prior to compression|
NSF data in data directory after compression
Total savings from compression alone
43.7 GB or 33.2 percent
|NLO data in data directory prior to DAOS|
NLO data in DAOS
Total savings from DAOS
19.3 GB or 14.7 percent
|Overall data directory footprint||70.8 percent reduction|
|Overall footprint||47.8 percent reduction|
As you can see from the tool results, IBM Global Services can expect amazing savings in overall storage for the IBM domain. Enabling compression and DAOS across the entire domain can significantly reduce storage for Tier 1 storage and reduce backup volumes and other server operations.
Because the storage savings features in Lotus Domino 8.5 provide such significant savings, you might wonder if they come at a price on your Lotus Domino servers. For example, how do they affect your backup and recovery features? How is disk or processor I/O affected? Do you have to change your administration functions or methods? These are all great questions, and you'll be happy to learn that while these features significantly reduce your storage, they do not come at a high price. To discuss this further, let's focus on the two main areas that most of our customers have asked us about: backups and operational changes.
When you implement DAOS in your Lotus Domino environment, you can potentially make changes to your backup configurations and the frequency of backups. DAOS decreases the size of your NSF data volume, and savings can also occur in your backup volumes and backup times. For example, in the IBM domain, after the implementation of DAOS, the NSF footprint went from 64 GB to 23 GB, a savings of about 64 percent. As a result, the overall backup volume of that server (both NSF and NLO data) also reduced from 64 GB to 25 GB, a savings of about 61 percent overall. Therefore, IBM was able to modify backup times and backup volume requirements to accommodate the decreased volume.
Of course the amount of reduction in the overall backup data volume in your environment depends on the number of unique attachments in the DAOS repository. This number can vary based on your attachment usage in the environment.
An example of this range is shown in table 6. Table 6 shows how backup durations of a single server in the IBM domain changed throughout the implementation of DAOS in the environment. After DAOS was fully implemented, backup times were dramatically reduced. Only new attachments, approximately 300 MB per day, needed to be backed up. You can see here that IBM went from backups that took 154 minutes a week for the NSF volume to backups that took 36 minutes a week for the NSF volume plus 0.75 minutes a day for the NLO data. That's a significant savings in the amount of time required to obtain a backup of the system.
Table 6. DAOS reduction of backup data volume and duration
The implementation of the storage savings features outlined in this article can also make a difference to your maintenance schedules and server operations in general. As you know, the size of the NSF files affects server operations such as compact and fixup. Smaller Lotus Notes databases decrease the time required to perform these tasks and can therefore decrease maintenance times and potentially even how often you need to perform maintenance on your servers.
For example, compact operations act like a disk defragmentation tool for your NSF files. In Lotus Domino, because of the algorithm used when the compact operation is run, the server continuously moves large components (such as attachments) to the end of the NSF while filling the resulting space with smaller components. This process is repeated for every large and small component in the NSF. After all the small holes are filled, compacting then puts the larger components back at the end. This process can cause large components to be moved multiple times, increasing the time required to run the compact operation on a large NSF file. When you move those large components from the NSF into DAOS, the compact operation no longer needs to act on those components, and the compact operation takes less time to complete.
As you can see from the real-world examples we have shown here and the various features you can enable on the Lotus Domino 8.5 server, you can realize significant cost savings in your Lotus Domino environment. Implementing the three main features we outlined, design compression, document compression, and DAOS, can significantly affect your disk storage requirements and your overall operational costs.
In addition, one advantage we have not given much discussion here is the option you now have to move portions of your disk storage to lower-cost options. For example, in the past, Lotus Domino database storage recommendations specified that you should house Lotus Domino data in high-end data storage with as many arms as possible due to the frequency of data access and I/O requirements that accompany large mail files. With DAOS, however, you can move your attachment store to a lower-cost disk structure, saving on infrastructure costs because attachments are typically accessed infrequently and don't require heavy I/O operations to access.
Another benefit you can see is the I/O rate of your DAOS disks as it pertains to the overall I/O requirements for your Lotus Domino server. Many customers have high I/O rates and are "I/O bound" because of mail file sizes. When you move attachments out of the NSF stores, you see a reduction in your I/O rates, and you can notice that the relative I/O requirement for DAOS is significantly less, leaving more available I/O for the Lotus Domino databases.
The savings you can see in your Lotus Domino environment are real. Within IBM, we have been able to save considerable storage space and operation times. Run the DAOS Estimator today and realize your own savings.
Refer to the IBM Lotus Notes and Domino wiki.
Refer to the wiki articles on IBM Lotus Domino attachment and object services (DAOS).
Refer to the wiki articles on IBM Lotus Notes and Domino compression.
Read what Ed Brill had to say about customer results when they run the DAOS Estimator tool.
Read what the Domino Blog had to say about DAOS.
Get products and technologies
Download the DAOS Estimator tool.
Andrea is a Collaboration Evangelist and Subject Matter Expert at IBM in the WPLC product development area. She has been working with Lotus Notes and Lotus Domino technologies since 2006 in various roles. She has spent the last two years focusing on helping customers understand how to overcome business challenges with disparate workers and improve productivity through the use of Lotus Notes and Lotus Domino and social networking tools. She can be reached at email@example.com.
Gary Rheaume is the lead architect of IBM Lotus Domino attachment and object services (DAOS). He has a broad background in programming, storage technologies, and database applications. When he is not tracking down and eliminating unnecessary I/O, Gary enjoys spending time with his family and playing guitar and accordion. Gary can be reached at firstname.lastname@example.org.
Patrick graduated from Penn State University in 1984 with a degree in Computer Science and started to work at IBM. He left a few years later to work for Lotus Software Development in Cambridge, MA on several 1-2-3 projects as well as NotesPump/LEI, DECS, Domino/DB2, and Domino internals. Recently, he has been working as an architect and developer of the DAOS feature in IBM Lotus Domino. He can be reached at email@example.com.