Two popular themes floating around information technology today are pervasive email and self-service applications. Notes applications excel in both respects because they can respond to messages they receive and can initiate a workflow process on behalf of the senders. Fortunately, it doesn't take much programming to add an automated message "process and respond" capability to your applications.

Consider this product support scenario:

1. Sue has a problem with the payroll software she is using.
2. She sends an email describing her problem to the mail-in database address of the help desk application used by the IT department to track user requests for assistance.
3. An agent in the application transforms Sue's memo into a new Problem Report, starting a workflow process on her behalf.
4. At the same time, the agent distills the key terms in Sue's email request and sends back a newsletter with doclinks to related and possibly helpful FAQs. The response also includes a link to the Problem Report that was created on Sue's behalf so that she can check the status of the work being done on her problem.

Programming the help desk application to act upon and to respond to email it receives, without human intervention, offers these advantages:

• Even though Sue doesn't have the right to create Problem Reports in the help desk application, a new Problem Report is created for her using information from her email. This saves Sue a call to the IT department for help, and because her email results in a formal Problem Report, her request is officially logged and tracked. This increases the efficiency of the IT department.
• The newsletter may in fact help Sue solve her own problem without further intervention by the IT department. This reduces the workload of the IT department and helps make Sue more self-sufficient.
• Because Sue was sent a direct doclink to her Problem Report, she can monitor the progress being made on her request. So the IT department is more accountable to its customers.

This article is the third in a series about Notes application strategies. In the previous two articles, we covered developing a rules-based task assignment application to assign work to individuals based on certain rules or conditions being met. In the second article, we explained how to add an interactive search form to an application to improve search capability. (Elements from both previous articles are referenced in this one.) Continuing our discussion of application strategies for Notes/Domino 6 and later, this article outlines automatic email processing as an application strategy that combines design elements and techniques to make up a value-add feature that you can easily add to any service-oriented or knowledge-based application. Similar to the previous articles, this one assumes that you are an experienced Notes application developer with knowledge of LotusScript.

Install the example database

The code shown in this article is from the Mail Processor database (mailprocessor.nsf), which you can download from the Sandbox. To prepare the application for demonstration, your server administrator must perform these steps: Because the database must run on a Domino server to receive mail, copy the database to the data directory on a non-production server.

1. Assign you Designer-level or higher access.
2. Open the server's Domino Directory to the Configuration\Messaging view and create a Mail-In Database document for mailprocessor.nsf. This document maps a unique mail address (Mail-in name field) to the database so that like a Notes user, it can receive messages. Here is an example of a Mail-In Database document:
   
   Figure 1. Mail-In Database document
   
   
   It is assumed that the server is otherwise configured to receive and deliver messages sent from your Notes client (the router is running and has the necessary certification/cross certification, Connection documents, and so on).
3. Create the full-text index for mailprocessor.nsf.
4. Sign the mailprocessor.nsf database elements (specifically agents) with a Notes ID file from your organization. You can do this from Domino Administrator. Select the server, and then click the Files function tab. Right-click the mailprocessor.nsf file and choose Sign.

As you dig into the design of the Mail Processor database, you will recognize the Problem Report form from the article, "Notes application strategies: Rules-based task assignment," and the full-text search routine from the article, "Notes application strategies: Interactive search."

AutoProcess overview

There are actually two aspects of automatic email processing demonstrated in the Mail Processor application: AutoProcess and AutoRespond. In the case of AutoProcess, a message is converted to a Problem Report. This diagram shows the flow of a message through the system as it is automatically processed:

There are two parts of automatically processing mail:

1. Receive the message.
   This is built into Domino. The router uses a Mail-In Database document to locate the database and to deliver the message. The message becomes a document in the Mail Processor database.
2. Process the message.
   Messages are initially stored in the MailBox view in the Mail Processor database. The MailProcessor agent runs hourly (but can also be run manually from an action button in the MailBox view) to pick up messages from this view and to convert them to Problem Report documents.

Changing the form name is easy. The difficulty of converting the message is intelligently setting the various status fields based on the content of the email: From (the name of the requestor in the Problem Report), AppliesTo, Priority, Problem Type, and Severity. To do this, the message Subject and Body are distilled into a string, or digest, of unique words (minus numbers, punctuation, common words, and full-text query search terms). Loop through the rules, in view order, and compare the search terms in each rule to see if the term exists in the digest (using Instr). If found, use the status field values in the rule to set the fields in the Problem Report. If none of the rules apply, the defaults from the ProfileDoc are used instead.

AutoRespond overview

The second aspect of automatic mail processing demonstrated in the Mail Processor application is responding to the user's email. This diagram shows the flow of a message through the system as it is automatically replied to:

The AutoRespond part of the MailProcessor agent is responsible for these actions:

• Generate reply message.
  Create the document that will be mailed back to the original sender.
• Perform search for FAQs and create newsletter.
  Using the same digest distilled from the original message, use the FTSearch method to return a collection of FAQs. With this collection of documents, create a NotesNewsletter object and insert it into the document.
• Create a link.
  This link points to the now-transformed Problem Report and is appended to the document.
• Send the reply to the user.
  Even if you don't elect to send along a newsletter or link to the Problem Report, this reply lets the user know that the email was received and that help is on the way.

Opening the database

When you open the Mail Processor application, you see a page with an embedded outline control in the left frame and a frame with the MailBox view on the right.

This table describes the HomeOutline outline entries:

Back to top

Configuration

Before we talk about how the various aspects of the application are coded, let's take a look at how to configure the application. Click the Application Settings outline entry to open the ProfileDoc form.

AutoProcess tab

The AutoProcess tab enables (or disables) automatic mail processing and sets the default values for the various status fields that will be used when an email is converted to a ProblemReport:

Keep in mind that these defaults apply only if you have not defined any Rule documents or if the memo being processed doesn't match any of the rules. (Rules are covered below.) If you do not automatically process memos, they will appear in the MailBox view in the Assignment Queue category:

They will stay there until you manually open the documents and set the various status fields. Once saved, they move out of the MailBox view into the Requests by\Posted Date view.

AutoRespond tab

The AutoRespond tab enables (or disables) automatic responses to messages and defines what to include in the response. Here is an example of an automatic response:

These fields on the AutoRespond tab control the content of the automatic response:

You can change the Please respond to field to a junk mail database address (change the replication settings on that database to purge documents after just one day) and then change the Remark to say something like, "Please do not reply to this email." This discourages continued interaction via email so that all follow-up action is performed in the structured environment of the Problem Report.

Search tab

The Search tab\Common Words subtab defines the stop word list (common words) that are stripped out of the user's message to create a unique array of important words. This array is then used by:

• AutoProcess to match the words with the word found in each rule to determine how to set the four status fields
• AutoRespond to build a full-text query to create the newsletter that is sent to the user

To give you a head start on a common word list, click the Add Defaults button. You can then add more words to ignore during the search. The Search tab\Search Parameters tab defines how the search for words contained in the user's message is conducted against the FAQs (using the InfoDoc form):

These options are almost the same ones that you can set from the More panel in the Search Bar, giving you the ability to refine the query and the display of the results. The settings on both of these tabs are explained in the article "Notes application strategies: Interactive search."

MailProcessor Agent tab

The MailProcessor Agent tab enables (or disables) the MailProcessor agent.

When you click the button, you are prompted to select the server on which the agent will run. In this demonstration, be sure to select the same server where the database is installed. (In production, there may be multiple replicas of the database, and you only want the agent to run on one of the servers.) By default, the agent runs hourly, but you can easily change the schedule from the MailProcessor Agent properties. Click the Schedule... button to change the schedule and the server on which the agent runs. (Change the comment to match your new schedule so you can tell from the Agent list when the agent is scheduled to run.) Do not, however, change the target, as the agent should only consider new documents for efficient operation.

Note: If you want to run the agent manually, open the MailBox view and click the Process Mail action button.

Rules

Imagine an incoming email containing any possible number of terms to describe the user's problem. How should the application mine the text to determine the problem? By default, all messages are treated the same and assigned status field values using the settings in the AutoProcess tab. If you want all incoming messages to be set the same, you don't need to create any rules. The assumption is that a person will sort through the new Problem Report documents and manually set the various fields.

But you can add a degree of intelligence to the AutoProcessing aspect of the MailProcessor agent by creating one or more Rule documents. A Rule document tells the agent to compare the unique word list distilled from the user's email to the term defined in the rule. You create and manage Rule documents from the MailBox view. The rules are used in the order they appear in the view from top to bottom. To change the sequence of the rules, select a rule and click the Up or Down action button to shift it up or down. In this example rule, LOTUSSCRIPT is the search word. If found in the list of words distilled from the user's email, the Problem Report status fields are set accordingly:

You can add multiple search word values (separated by semi-colons) and even use search phrases (spaces between words), for example: LOTUS NOTES. The memo digest is not sorted, so the words stay in the same order as in the memo. Keep in mind that we use Instr to compare each search word (or entire phrase) in the rule to the memo digest. If you enter multiple search word values, the AutoProcess part of the MailProcessor agent processes each value, ending the inspection with the first match.

Here is a snippet of code from the MailProcessor agent that loops through the rules for each document mailed into the database:

Do Until Ruleve Is Nothing
	Set RuleDoc = Ruleve.Document
	Forall SW In RuleDoc.SearchWords 
	'Msgbox abstractplus 'debug see log.nsf on server for output
		If Instr(abstractplus, SW) Then 
		Call AddFieldValue(RuleDoc, doc, "RequestType", "RequestType")
		Call AddFieldValue(RuleDoc, doc, "AutoAppliesTo", "AppliesTo")
		Call AddFieldValue(RuleDoc, doc, "Priority", "Priority")
		Call AddFieldValue(RuleDoc, doc, "ProblemCategories", "Categories")
		Call AddFieldValue(RuleDoc, doc, "Severity", "Severity")
		Exit Do
	End If
	End Forall 
 	Set Ruleve = Rulevc.GetNextEntry(Ruleve)
Loop

The preceding code snippet constitutes the outer loop through each of the rules. The Forall that begins on line 3 and ends on line 13 constitutes the inner loop through all the possible multiple values in the SearchWords field in the rule. The string comparison occurs in the line: If Instr(abstractplus, SW) Then. If the string is found in the memo digest (at this point in the code stored in the abstractplus string variable), the status fields are set in the next five lines that begin with Call. The line Exit Do exits the outer loop because there was a match. No additional search words or rules are considered.

Of course, if the search words in the rule are in a different order or misspelled either in the rule or in the memo, there will be no match to any rule. Therefore, the default status fields from the ProfileDoc are used.

MailProcessor agent

Let's step back and look at how a document received via email is processed by the MailProcessor agent. Open the agent in Domino Designer to follow along with the annotations. The MailProcessor agent uses the AutoProcess script library, which contains a handful of helper functions and subs that perform specific tasks. We only briefly mention the purpose of select functions/subs as we go along.

The overall structure of the MailProcessor agent is to loop through all the memos in the MailBox view. With each memo, perform these actions:

1. Create a digest of unique words by calling the MemoDigest function from the AutoProcess Script Library. MemoDigest concatenates an abstract of the memo Body field with the Subject field and strips out any numbers and punctuation. The function then strips out the common words defined in the ProfileDoc and removes any full-text search terms. It also eliminates any duplicate entries.
2. Perform AutoProcess actions. Loop through the rules looking for a term that matches any words in the list of unique words distilled from the email. If there is a match, use the rule to set the various status fields. If there is no match, use the default status field values from the ProfileDoc. Call the AgentMorphMemo sub to convert the Memo to a ProfileDoc.
3. Perform AutoRespond actions. Create a new response memo. Add the text from the ProfileDoc document and append a doclink to the Problem Report document. Using the unique list of words, perform a full-text search of the FAQs and generate a newsletter. Send the response email to the user.
4. Delete extraneous fields by calling the CleanFields sub from the script library that are used in the Memo form, but are not needed in the ProblemReport form.
5. Save the document.

Back to top

Test the application

It is very easy to test the agent's AutoProcess and AutoRespond capabilities (using the included FAQs for now).

Note: This procedure assumes that you have installed and configured the application as shown earlier.

Follow these steps to test the Mail Processor application:

1. From Notes, create an email addressed to the Mail Processor application. Be sure to include terms that are found in the existing FAQs, for example, "LotusScript." Send the email.
2. Switch over to the Mail Processor database and open the MailBox view. Refresh the view and you'll see your email in the Response Queue category.
3. Click the Process Mail action button to run the MailProcessor agent immediately.
4. Click the Requests by Date entry in the outline control. Expand today's date to find the new Problem Report. Open the document to see that the status fields are assigned the values set in the rule.
5. Return to your mail database, where you'll find an email that contains the remark you set in the ProfileDoc document, a link to your Problem Report document in the Mail Processor database, and a newsletter containing documents returned from a full-text search of the FAQs.
6. If you are curious about the full-text search query used to build the newsletter, open the Document properties for the email and read the value of the QUERY field. The query was built using the settings on the Search tab in the ProfileDoc document and unique words extracted from your email. If you find that too many common words are creeping into searches, you can add them to the common word list. In this example, we would add the words: help, class, work, thanks, and having to increase the efficacy of the search results used to create future newsletters.

Back to top

Hard-coded values you can change

In most cases, you configure the application from the ProfileDoc document. But there are also places where we hard coded values because it was easier. You can change these values in the code as you see fit:

• MailProcessor agent (Initialize event). On Line 2, change the value of the Debug variable to True. This lets you run the agent from the Agent list with the LotusScript Debugger running.
• MailProcessor agent (Initialize event). On Line 106, we define the line of text to place above the user's original memo. If you don't want to include the user's email text at all, comment out Lines 106 through 108.
• AutoProcess (Script Library): MemoDigest function. These three variables starting on Line 10 affect how the Body and Subject fields of the user's memo are abstracted:
  
  MaxWords = 100
BodySize = 10000
SubjectSize = 500
  
  MaxWords limits the size of the unique word list that is used to create the search query by AutoRespond and used when comparing the terms found in the user's memo to the rules. Setting this value to 100 seems about right, but you can change this to better meet your needs. Keep in mind that the actual items in the list will be fewer as duplicate values are removed later in the code and that the higher the number, the more work is required. One thing we found after running a few tests is that you really do need to expand and refine the common words list if you want to increase the accuracy of the full-text search for the newsletter. But of course, our list will differ from yours, so you will have to edit the list in a way that makes sense for your organization and the content of the documents being searched. BodySize and SubjectSize tell the Abstract function how many characters to take from the user's email before eliminating the common and duplicate words.
• AutoProcess (Script Library): MemoDigest function. Starting on Line 27, we define an array of characters that we want to strip out of the AbstractPlus variable. As opposed to the words defined in the common word list, these are single characters (numbers and punctuation) that we want removed. (We didn't remove the apostrophe, so we could remove contractions by adding them to the common word list). We call the MultReplace function from the library to remove all the instances of each character. You can change the array of characters as you see fit. For example, if part numbers are important, you may want to remove the numbers from the array assignment. Or, if you are using a language other than English, there may be other symbols you want removed as well.

Back to top

Caveats and workarounds

There are several caveats to using the Mail Processor application:

• Because of the free-form nature of an email, it is difficult to determine exactly what users are asking for (short of employing a natural language interpreter). This strategy makes a best-effort attempt at finding important words by first removing common, reserved, and duplicate words before performing a full-text search (for the AutoRespond newsletter) or comparing the words with the terms in the rules (for AutoProcessing).
• Most likely your application uses forms other than InfoDoc (FAQs) or ProblemReport, so you will have to modify the application to use your forms and views.
• By default, the MailProcessor agent is scheduled to run hourly. If the server is not overloaded, you may want to schedule the agent more frequently. To run the agent before or after new mail arrives requires just a few minutes of work on your part.
• There is no mechanism for users to tell you if the FAQs listed in the newsletter were helpful and/or if they solved the problem themselves. (In a full-blown application, the user would be assigned Author access to the Problem Report, allowing self-editing of certain fields and/or deletion of the request.)
• Rules can be extended to include other fields, for example, to automatically assign the request. See the article "Notes application strategies: Rules-based task assignment" for ideas on how to use assignment rules to determine who should be assigned to a task.
• As designed, the newsletter contains doclinks, so is useful only to Notes users. With a small amount of work, the MailProcessor agent (and associated Script Library functions) can discriminate the sender as needing a custom "newsletter" that uses URL links to FAQs instead of doclinks, so the non-Notes user can open them from a browser. Customers outside your organization can thus be serviced by this same automated email strategy.
• When new Rule documents are created, the action counts the number of existing rules and adds one to determine the next sort number for the new document. If multiple users are adding rules (and on multiple replicas of the database), it is very likely that duplicate sort numbers will result. While this doesn't really impact how the MailProcessor agent processes rules (uses the view order), it does prevent the move up/down actions from working. Use the Reset Sort action button in the MailBox view to manually reset the sort number of the selected rule.

Back to top

Summary of design elements

To use these features in your application, copy these design elements from the example database to your database:

• Forms: InfoDoc, ProblemReport, ProfileDoc, Rule
• Subform: SearchSupport (used in InfoDoc)
• Views: FAQs, Mail Box, Requests by\Posted Date
• Agent: MailProcessor
• Script Library: AutoProcess

It is up to you to configure your application so that database administrators can open the ProfileDoc form as a profile document. You can follow the example of the HomeFrameset frameset, HomeOutline outline, and HomePageOutline page design elements.

Back to top

Conclusion

This article described an application strategy that promotes responsiveness to user requests by automatically processing email requests and sending an automated reply. And because you are adding this feature to a Notes application using Domino Designer, it takes very little effort to quickly realize significant benefits.

Resources

• Read the first two articles in this series: "Notes application strategies: Rules-based task assignment" and "Notes application strategies: Interactive search."
  
  
• Download the Mail Processor database from the Sandbox.
  
  

About the author

Rate this article

Comments

Back to top

Table of contents

• Install the example database
• Configuration
• Test the application
• Hard-coded values you can change
• Caveats and workarounds
• Summary of design elements
• Conclusion
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into Lotus on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers