Lessons in secure messaging using Domino 6

This article explains how the major pieces of Notes/Domino 6 messaging security fit together, using a clear, easy to understand example starring a legendary figure in the oil business.


Timothy Speed, Consulting IT Architect, IBM, Software Group

Timothy Speed is an infrastructure and security architect for IBM Software Services for Lotus (ISSL). Tim has been involved in Internet and messaging security since 1992. He also participated with the Domino infrastructure at the Nagano Olympics and assisted with the Lotus Notes systems for the Sydney Olympics. His certifications include MCSE©, VCA (VeriSign Certified Administrator), Lotus Domino CLP Principal Administrator, and Lotus Domino CLP Principal Developer. Tim has also co-authored four books: The Internet Security Guidebook,ISBN: 0122374711, February, 2001; The Personal Internet Security Guidebook,ISBN: 0126565619, October, 2001; Enterprise Directory and Security Implementation Guide: Designing and Implementing Directories in Your Organization,ISBN:0121604527; and Internet Security: A Jumpstart for Systems Administrators and IT Managers, ISBN 1555582982.

Raj Balasubramanian, Consulting IT Architect, IBM, Software Group

Raj Balasubramanian is a Consulting IT Architect for IBM Software Group. He works on customer engagements delivering application and infrastructure related projects. His interests range from anything technical to history and physics. During his copious spare time, he enjoys dating his wife and talking about robots with his sons. You can read about his technical and personal escapades on his personal blog Gurukulam (http://balasubramanians.com/blog)

06 July 2004

Also available in Japanese

Let me tell you a story about a man named Jed, who has been in the oil business (or as we say in Texas "the bidness") for about 30 years. As most of us know, the oil business requires drilling lots of holes into the ground. So Jed often rents from oil equipment vendors, including drilling equipment, tractors, and oil pumps. And he sometimes purchases pipes and other hardware.

One day Jed's well-meaning nephew Jethro showed up at the door looking for a job. So Jed made Jethro "VP of Vendor Management for all services in Hondo County." (Jethro liked the long title.) A few months later Jethro called to tell Jed about a large amount of equipment that had shown up unexpectedly at his front door. Initially, Jed suspected that the easily-distracted Jethro had actually ordered the equipment, but had forgotten about it. Jethro insisted that he had not ordered any of this equipment and that it should be returned. Unfortunately, the equipment rental company now wanted delivery charges and a full day's rental--a considerable sum for this kind of hardware. So Jed contacted the company and asked them to verify that the order had in fact been generated by Jethro.

After a full investigation, it turned out that the vendor had hired Jethro's cousin Ellie May. Ellie's former boyfriend Bob Hacker was angry with her after their very vocal break-up. So in spite, Bob sent Ellie a fake email in Jethro's name, ordering 30 tanker trucks and 14 drilling rigs to be delivered to Jethro's home address.

As soon as the facts were all straightened out, Jed knew what to do to prevent anything like this from every happening again. No, he didn't move to Beverly Hills and buy an overpriced mansion with a bowling alley. Instead, he called IBM to work with both Jethro and the vendors to set up secure Domino messaging. (At this point you may be thinking, "Just set up a Web page at each vendor, use SSL (Secure Sockets Layer), and off we go!" Yes, this is one option, but it would ruin our story--and your education--about secure messaging. And hard though it may be to believe, not everyone has a Web site, and many companies use email as the backbone of their business.)

This article explains how our mythical hero implemented secure messaging for his vendors and himself. We assume that you're familiar with Notes and Domino. (An acquaintanceship with the old TV show "The Beverly Hillbillies" is strictly optional.)


We all know that Lotus Notes/Domino is a robust, secure, and scalable messaging system. Unfortunately, not everyone has Notes/Domino. So let's spend some time talking about a cross-vendor messaging solution: secure SMTP messaging.

SMTP (Simple Mail Transfer Protocol) is the de facto standard for email that traverses the Internet. In fact, the initial specification is a standard (RFC821) first published in 1982 before the Web as we know it even existed. (For perspective, the standard defining the URL was published in 1994.) If you send a message on the Internet, you use SMTP. Notes/Domino fully supports SMTP messaging, providing many powerful mechanisms to keep the message secure.

At about the same time SMTP was defined, PKI was developed. PKI (Public Key Infrastructure) is a secure system for a trusted third party (known as a Certificate Authority or CA) to provide digital identities to users and servers and to publish those identities using X.509 V3 digital certificates. The identity consists of an asymmetrical key pair: a Public Key that anyone can use (thus the name) and a unique Private Key known only to the user. This digital identity can then be used to exchange text securely by encryption using the key pair.

There are two basic ways the key pair is used in messaging:

  • Message integrity
    The sender creates a message digest (also called hash value, fingerprint/thumbprint, or watermark) based on the contents of the message. The sender encrypts the message digest under her private key. The digest is attached to the message being sent. This digest is not very large, typically 128-bits. Because of the properties of asymmetric key encryption technology, the recipient can then verify that the message was really from the sender by using the sender's public key (because it is publicly available) to verify the message digest (for example, MD5) and can also verify that the message has not been modified in transit.
  • Message encryption
    The sender uses the recipient's public key (available via published digital certificates) to encrypt the message. The recipient then uses his private key to decrypt the message.

These techniques are known as S/MIME (secure MIME) in the Internet world.


How does all this help Jed? Before we cover that, let's talk about nonrepudiation. This is a process for binding an end user to an action or transaction. Nonrepudiation provides evidence that a particular action occurred and can be traced to a specific entity (user or business). In our example with Jethro and Jed, Ellie could have verified the integrity of the message before executing the equipment order. The verification of the message can be accomplished using a digital signature. This is referred to as message integrity.

S/MIME is one of the solutions for this problem. Jed and Jethro need to send important messages via the Internet. Because the Internet is a public space, a malicious person like Bob Hacker can intercept messages with little effort. In our example, Bob "spoofed" Jethro's address from another address in an SMTP email. Here is what it looked like:

Hidden Header Info (start)
Return-Path: <JethroB@jedco.xyz>
Received: from relay.hondo.texas.vendor.biz ([])
	by amailserver.somecompanysomwhere.biz
	(InterMail vM. 201-2115-109-103-1231231231) with ESMTP
	for <Ellie-May@thevendorcompany.xyz>; Tue, 23 Mar 2004 04:05:11 -0500
Date: Tue, 23 Mar 2004 04:04:13 -0500 (EST)
Message-Id: <11234234-23864-64000886-1234234>
From: Jethro VP at Jeds Company <Jethro@jedco.xyz>
To: <Ellie-May@thevendorcompany.xyz>
Subject: Please order all of this stuff for me
MIME-Version: 1.0
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Hidden Header Info (end)
Dear Cousin Ellie:
Jed wants me to rent some equipment. We are going to drill for oil
over in front of my home. Send me this stuff -
30 tanker trucks
14 drilling rigs

Ellie received the message, accepted as genuine, and had the equipment sent over to Jethro's home.

Let's look at an open standard method to ensure that a message is actually sent from Jethro. As we have said before, S/MIME provides a method to send and receive secure MIME messages. MIME (Multipurpose Internet Mail Extensions) defines how an electronic message is formatted. MIME specifications describe a way to deliver text via various character sets and multimedia.

Message confidentiality

Secure messages are created via a system of digital certificates. Certificates are part of digital identities that are linked to a specific user or server. In our example, a digital identity (an asymmetric key pair) is placed into each user's email system. The first part of the digital identity (the public key) is placed into the public directory in the form of a digital certificate, and the second part (the private key) is placed into the email client:

Figure 1. S/MIME configuration
S/MIME configuration

In our example, Jethro sends a message to Ellie. Jethro finds Ellie's public key published via a digital certificate in the directory and sends her a message. Jethro’s workstation software encrypts the message which is then sent via the Internet (SMTP) to Ellie’s workstation. When Ellie opens the message, it is automatically decrypted using her corresponding private key. Now Ellie can reverse the process. If she has Jethro's public key (or access to the directory where Jethro’s key is stored), then Ellie can send an encrypted message back to Jethro. This process is accomplished by the S/MIME protocol. Although the actual encryption process is complex, the preceding example illustrates the concept and provides a high-level explanation of the process.

So far we have discussed the basic S/MIME encryption process. This will prevent Bob Hacker from reading messages in transit. But how can Bob be kept from sending messages impersonating Jethro? In the past, there was no way to stop that. Today there are several new options, including reverse DNS checks and so on. In this article, we focus on encryption and digital signatures.

Message integrity

The same digital certificate used for reading encrypted mail can also be used to digitally sign documents. A digital signature serves as the computer equivalent of a handwritten signature or thumb print. A digital signature is a math-based method of attaching a "personal" identity to a message. Digital signatures rely on public key cryptography. Usually the messaging program doesn't actually sign the original message itself with the user's private key. Instead, it produces a condensed version of the message. This condensed version is the message digest we talked about earlier.

The message that you sign is scrambled into a message digest using your private key. (It's the message that actually gets signed.) After the recipient receives it, the message digest is unscrambled using the sender's public key. By convention, anyone can read the signed message digest by using the signer's known public key.

To produce a message digest, you need a scrambling mechanism. There are many different algorithms for creating these message digests. For example, MD2, MD4, and MD5 are hash functions developed by RSA Security. They all produce 128-bit digests. The MD5 algorithm is the de facto hashing standard for digests. Public domain versions are available for most platforms on the Internet, and it is widely used in integrity checking systems. In addition, SHA-1 (Secure Hashing Algorithm) is an NIST-sponsored hashing function and has been adopted by the U.S. government as a standard. It produces a 160-bit digest.

When a document is signed, you can verify who signed (and possibly sent) the document. As we mentioned, with messages sent on the Internet, it is easy to modify the From field and to spoof the identity of the sender. If Jed requires that senders digitally sign all orders, he can prove that the messages really originated from the vendor. (The assumption here is that the order arrived and that Jed's order-processing department did not make an error). This provides virtually unforgeable evidence that a specific action occurred (or in our example, a transaction that was executed originated from a party whom we know). This can be done via a digital certificate published in X.509 V3 format (assuming, of course, the vendor has complete control of the certificate and unauthorized people don't have it). This process can be used by both the sender of the order and the recipient.

(Remember the MD2 and MD5 message digests? If any one character of a message is changed, then up to half of the message digest changes. It is very difficult to modify a message and to keep the message digest the same. For those of us not versed in higher mathematics, it is impossible.)

The process for verifying a digital signature is shown in Figure 2:

Figure 2. Digital signature verification process
Digital signature verification process

In the example shown in Figure 2, Mike digitally signed a message and sent it to Bubba. Bubba's software automatically attempted to verify that the signature was from Mike via the public directory.

Closed vs open certificate issuing systems

There are two basic systems for issuing certificates: a closed (private hierarchy) system and an open (public hierarchy) system.

Closed private certificate issuing systems

The closed system has a certificate authority as part of one entity. In theory, you could have several organizations and organizational units. And you could distribute the responsibility for supporting and maintaining those organization certifiers. With a closed system, your company is in the business of being a CA. There are many reasons to do this. One would be that you have several bank branches and that you want to encrypt data between each branch. In that case, you are the CA and issue the certificates. In practice, you should place the certifier under lock and key and may even put a guard (an actual person!) in charge of handing out the key via a very controlled mechanism. You can use a single corporate-wide CA or multiple CAs.

The advantages of closed systems include:

  • You are the CA and control all certificates.
  • You control the binding of the user to the certificate. (For instance, you could physically look at each person and then issue the certificate, although obviously this does not scale very well!)
  • You manage the certificate structure, naming, validation, and expiration.
  • You control your own destiny. No one else can use certificates or impersonate yours, at least in theory. (We talk more about this later in this article).

The major disadvantage to a closed CA system is the fact that because you are the CA, you need a support structure to handle this responsibility. This includes installing and maintaining the proper software and/or hardware to implement the solution. And you need a support staff to manage the certificates. (Certificates expire, people may need to change their names, and so on.) In addition, you need a policy on how to handle issues involving people leaving your company (more on this later).

If you are your own CA, you control that CA as long as no one else has it. Unfortunately, to create a certificate someone (or something) needs to touch the CA and to act as a certifier. The CA certifier is a physical entity that exists somewhere (in a document, file, hard drive, floppy, and so on). As a result, it's always possible for an unauthorized person to obtain a copy of an unguarded CA. And even if the CA file has a password, this information could leak out as your administrators leave and move on to other companies. And if the CA certificate(s) fall into the wrong hands, outsiders can create certificates that can impersonate a valid member of your organization. If you do become your own CA, work with your software vendor to determine methods to protect the CA certificate.

Open public certificate issuing systems

An individual or company can go to a public system to request certificates. The public CA is typically an independent entity, providing a service for issuing certificates. In some cases, this CA is known as a third-party CA.

Advantages of using open public systems are:

  • Even though you are not the CA, you can set up a service via a public CA to provide certificates on your behalf. You still control the certificates, but you do not control the root certificate.
  • You can control the binding of the user to the certificate. Many vendors have automated systems that can register users and issue certificates.
  • You manage the certificate structure, naming, validation, and expiration, but as we mentioned, you cannot control the root certificate.
  • Overall, the cost of an open public system can be less expensive than a closed system because you don't have to deploy an infrastructure and people to support the system.
  • Most well-known public systems have a proven track record. (We have never heard of a case where a public infrastructure had an outage.)

One disadvantage of using an open public system is that you cannot own the root of the certificate. Therefore, someone at the service could create an unauthorized certificate. If your company does business with a public CA, you need to consider the vendor's level of service, legal liability and responsibility, code of ethics, and Certificate Practice Statement (CPS). Each of these needs to be reviewed and agreed upon. Also, define the consequences of any breach of any of these issues. In the end, you should have a Service Provider agreement. This agreement defines the rights and obligations of the service provider and the company requesting or purchasing the services.

Bottom line: Consult your legal department before using a public CA.

Choosing a certificate issuing system

So which system is better, closed private or open public? Don’t select the technology before understanding the business problem that you are trying to solve. Here are some issues that we recommend you review before making your decision:

  • What are the business requirements? What is the need, and what is being supported?
  • What are the user requirements?
  • What is the cost?
  • What are the support requirements?
  • Which user issues need to be addressed (number of users, locations, user types--office, roaming, remote--bandwidth, language)?
  • What are the training requirements (users, administrators, help desk)?
  • What happens if the root certificate CA is compromised? (This question needs to be asked for both public and private systems.)
  • Can the technology be used on other applications or business services?
  • Which service levels are required?
  • What process will be used to bind the user to the certificate? (This should be detailed in a Certificate Practice Statement.)

The Domino CA

Notes and Domino 6 provides many features that you can use with both inbound and outbound SMTP messaging. Included with Domino 6 is a PKI management process. The process is known as the Certificate Authority (CA) process. The CA process is a very powerful tool that can automatically issue certificates to a Notes 6 client or any Web user. The CA process runs as an automated server task on a Domino 6 server. This task allows you to set up a Domino CA. This authority can create, manage, update, and provide denial service (known as a Certificate Revocation List or CRL). Only one instance of the CA process can run on a server, but you can set up multiple certifiers on each server.

Digital signature verification

Here is how the digital signature process works in Notes/Domino. Returning to our example, Jethro sends Ellie a message. Ellie wants to verify that the message has not been altered and that was really sent by Jethro. The message is as follows:

Hello Ellie: How are you today? I would like to order 100 units of oil equipment. Please send it to the office on the 5th. Thanks, Jethro

The following table shows each letter of this message in ASCII, hex, and binary code:

Figure 3. ASCII, hex, and binary code
ASCII, hex, and binary code

At this point, Jethro signs the message. Using a generic MD5 utility generates the following message digest:


Now let's modify the message, as a hacker might using a "man in the middle" attack:

Hello Ellie: How are you today? I would like to order 1000 units of oil equipment. Please send it to the office on the 5th. Thanks, Jethro

Notice the addition of a single character -- an extra 0 to the number 100, resulting in an order 10 times larger than Jethro really wants. The codes for this single character are as follows:

Letter ASCII Hex Binary 0 48 30 110000

This single addition changes the message digest completely:


When Ellie receives the message from Jethro, her Lotus Notes client checks his signature via his digital certificate containing his public key that was published in a directory. If Jethro's signature is valid, then the message is re-hashed and compared to verify that it's valid.

Setting up the CA

To put this process in place, let's start with Ellie and use the Domino 6 CA process to create an X.509 certificate for her. In this example, we use a single certificate for both signing and for encryption (S/MIME). When Jed upgraded to Notes/Domino 6 several months ago, he set up a stand-alone registration server, a special Domino server that used the CA server task. This task manages and processes certificate requests (both Notes and Internet certificates). Jed set up the CA process to provide his company with a unified mechanism for issuing Notes and Internet certificates. He also set up a registration authority (RA) role, which he uses to delegate the certificate approval/denial process to lower-echelon administrators in the organization (such as Bubba).

Jed liked the idea that he did not need to provide direct access to the certifier ID and ID password to each administrator. After Jed enabled the certifiers for the CA process, he assigned the registration RA role to one of the administrators. The RA can then register users and manage certificate requests without having direct access to the certifier ID and password. Each certifier has an Issued Certificate List (ICL) database that is created when the certifier is migrated to the CA process. The ICL database stores a copy of each unexpired certificate that it has issued.

Figure 4 illustrates the basic architecture that Jed uses for his Notes/Domino 6 architecture:

Figure 4. Example Notes/Domino 6 security architecture
Example Notes/Domino 6 security architecture

This architecture includes a DMZ layer that protects the trusted network. A message sent from the outside world is processed and reviewed by the DMZ, and then delivered to one of the SMTP servers. The SMTP server delivers the message to one of the messaging servers. Outbound messages do much the same, the end user (Jethro) sends a message from his client, then the message is transferred to one of the SMTP servers. From there, it is sent into the DMZ where a smart-host device transfers the message to a targeted recipient.

Jed decided to set up a stand-alone CA registration server. This isolated the CAs from all other servers. Jed then put a password on the server.id file. This provides additional security for the server. (Obviously, the server cannot be automatically restarted. This is one of the reasons that the CA server was isolated so that the impact of the downed CA server would not impact SLA-based production servers.) In addition, access to the server is restricted to CAs and RAs only via access restriction in the server document. Also, no cross certificates are issued for this server. As a result, this server is isolated and restricted to just a few individuals and servers. (Jed also set up the IDPR process on this server.)

Jed next set up a stand-alone X.509 CA server. After the CA had been created, Jed worked with his main administrator to migrate the X.509 CA into the Domino CA process. To do this, he created two CA roles, one for himself and one for cousin Billy Joe Bob. He then set up three RA roles, one each for Billy Joe Bob, Bubba, and himself.

Using certificates

To deploy the certificates, Jed instructed Bubba to give Ellie an X.509 certificate. To do this, Bubba opened the Domino Administrator client and accessed the Person and Groups tab, then selected Ellie's name and selected Action - Add Internet Cert to Selected People:

Figure 5. Adding Internet certificates
Adding Internet certificates

Bubba then selected the correct certifier and clicked OK. After the process ran, the following message appeared indicating the certificate was successfully assigned:

Figure 6. Processing Statistics message
Processing Statistics message

This is basically a one-step operation. The next time that Ellie accesses her server, her new X.509 certificate is automatically pushed into her Notes ID file. If necessary, Ellie can look at the ID file using the File - Security - User Security drop-down menu on the Notes Client:

Figure 7. ID Properties screen
ID Properties screen

Clicking the Advanced Details button shows additional information about the certificate. Because Ellie has only one certificate, the "Use this certificate as your default signing certificate" is grayed out:

Figure 8. Certificate Advanced Details screen
Certificate Advanced Details screen

Jethro also obtained a certificate from his administrator for Microsoft Outlook. Now Ellie and Jethro both have X.509 certificates.

Public keys

By definition, public keys can be shared. With large enterprise companies, this is done via directory assistance and LDAP. This example shows a simple tactical method that uses the local Microsoft Outlook address book and the Lotus Notes personal address book.

The first step is for both Ellie and Jethro to send a signed message to each other. Jethro starts by creating a new message in Outlook, opening the Message Options dialog box, and selecting "Add digital signature to outgoing message:"

Figure 9. Signing a message in Microsoft Outlook
Signing a message in Microsoft Outlook

Next, Ellie on her Notes 6 client sends a signed message to Jethro:

Figure 10. Signing a message in Notes 6
Signing a message in Notes 6

In this template, there are two options to set the Sign flag. This is all you need to do -- the Notes client is very smart and will determine that the recipient is an Internet-based user. So the message will be signed using the X.509 certificate instead of the Notes.id file.

This process obtains access to each user's public key. This is done automatically using Lotus Notes. The Notes client receives the message, obtains cross certificates (after prompting the user to confirm this action), and then adds the sender's public key to the address book. Once the public keys are available, encrypted (S/MIME) messages can be sent between the users.

When Ellie on her Notes 6 client first receives a message from Jethro, she is prompted to create an Internet Cross Certificate in her personal address book:

Figure 11. Issue Cross Certificate prompt
Issue Cross Certificate prompt

When Ellie clicks the Cross certify button, a cross certificate is created in her personal address book on her Notes 6 client. Next, Ellie adds (or updates) Jethro's information to her address book by selecting Actions - Tools - Add Sender to Address Book. This displays the following dialog box:

Figure 12. Add Sender to Address Book dialog box
Add Sender to Address Book dialog box

Ellie clicks OK and is done; she now has Jethro's public key. Now it is time for Jethro to obtain Ellie's public key. The steps for Jethro are just as simple. Remember that Ellie sent Jethro a signed message. All he needs to do is to open the message to see an indication that he received a signed message. In Outlook this appears as follows:

Figure 13. Signed message in Outlook
Signed message in Outlook

Jethro right-clicks Ellie's email address in this message and then selects Add to contacts. This adds Ellie's name to Jethro's address book with Ellie's public key. Now Jethro can send Ellie an encrypted S/MIME message, and Ellie can do the same. The screenshot below shows Jethro sending his order to Ellie:

Figure 14. Sending an encrypted order
Sending an encrypted order


As you can see, Notes/Domino 6 makes sending secure mail a very easy process. Jethro creates the messages and selects Ellie from his address book. Then from the Options menu, he selects Sign and Encrypt and sends the message. Ellie receives the message, which is automatically decrypted. Ellie responds with a confirmation, also signed and encrypted.

Now Jethro can send Ellie signed and encrypted orders for equipment. This way Bob Hacker and his ilk cannot impersonate Jethro. Also, the orders are secret, so no one knows about that big oil find that Jethro discovered outside the back door of his home. This was all accomplished by the use of Notes/Domino 6 and X.509 certificates.



developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into IBM collaboration and social software on developerWorks

Zone=Lotus, Security
ArticleTitle=Lessons in secure messaging using Domino 6