Introduction

Most people only think of SOAP over HTTP for a couple of reasons. First, it is the most common transport of the SOAP protocol, by far having the most services available on the web. Second, because of how the HTTP protocol works, it fits very well into a SOAP request/response structure. In HTTP, you send a message to the server. The format of an HTTP request is flexible enough that you can embed a SOAP request within the body. The HTTP protocol then specifies a response (to all HTTP requests) that is in turn flexible enough to embed a SOAP response. This allows for a very straightforward implementation of a SOAP service.

This is not the case with the SMTP protocol. When using SMTP, the request format is flexible enough that you can attach a SOAP request. However, the response structure is not very flexible. An SMTP response is as simple as "O.K." The SMTP Service Extensions (ESMTP) specification does add a bit more information to a response such as "Unknown User" but there is still not enough flexibility in the response format to put in an entire SOAP response. The only way to send a response with SMTP is with another e-mail message.

Because of this, developers of SMTP SOAP services need to add additional logic to be able to track incoming SOAP requests, then send the SOAP response over a separate SMTP message to the recipient. This adds a fair amount of complexity (as we will see shortly) to SOAP services that use SMTP.

So, why would you want to use SOAP over SMTP? The most common answer is "I cannot use HTTP." One of the more common reasons for this is firewalls. If you are behind a firewall, odds are that you have no control over where HTTP requests are processed. However, there is a very good chance that you receive e-mail. Another reason would be that the request/response messaging model may not be the desired model for your application. SMTP makes perfect sense for publish/subscribe or a one-way messaging model. As a final reason, the service you are writing is not real-time. If your service has to perform complex queries, or complex calculations that can take over 300 seconds (common HTTP request time out) then you need an asynchronous approach, such as SMTP.

Back to top

Processing SOAP messages

There are three possible ways to process SOAP requests and responses over SMTP. The first approach is dependent on the abilities of your SMTP Server. With this approach, you write your SOAP processing application, then connect it to an e-mail address. Most e-mail servers will allow you to pipe messages that are received on a specific address to an application. Your application would then read in the SMTP message (most likely from standard process input), process the request, and send off the response.

The second approach assumes that you have access to the mailbox file. With this approach, you would regularly scan the mail file for SOAP requests, remove these messages from the mail file, process the requests, and send out the responses.

The last approach, and the one we will use here as the other two are heavily dependent on local configuration options, is to write an SMTP Server. With this approach, you listen for SMTP requests on a port, process the requests as they are received, and send out the responses when processing is done.

Back to top

Linking requests and responses

No matter how you decide to handle SOAP over SMTP, you need a mechanism to link SOAP requests and SOAP responses. The SOAP experimental e-mail binding (see Resources) recommends using the "Message-ID" and "In-Reply-To" SMTP headers. The client, puts a unique identifier into the "Message-ID" header when the message is sent, then the server uses that same identifier in the "In-Reply-To" header for the response message. This allows the client to match a received response with the appropriate request.

Back to top

Example dependencies

The example for this column is written using Python 2.2.1 and uses some features, mainly the new server architecture. We will also be using ZSI 1.2 to process our SOAP messages. Installation of ZSI is fairly painless as it uses standard python distutils for its distribution. See the Resources section at the end of the column for download information.

Back to top

A SOAP SMTP server

Using Python's builtin smtpd library, writing an SMTP server is relatively painless. As with all of Python's server architectures, you inherit from a base class, smtpd.SMTPServer in this case (see Listing 1), and override certain methods based on the functionality that you desire. In this case, we override the process_message method which is called whenever a new message arrived. To start the server we use Python 2.2's new asyncore to start handling requests.

		
class OurServer(smtpd.SMTPServer):

    #A place to store the current ID of the message that we are processing.
    currentID = None

    #This is the callback from SMTPServer that all SMPTServers
    #must implement to handle messages
    def process_message(self, peer, mailfrom, rcpttos, data):
       #
       # Snip out code for process_message
       # It will be discussed below
       #

#This is the 2.2 way of running asynchronous servers
server = OurServer(("localhost", 8023),
                   (None, 0))
try:
    asyncore.loop()
except KeyboardInterrupt:
    pass

Once a message is received, we can do some simple tests to see if it is indeed a SOAP request. If so, then we can pass it off to ZSI to dispatch the message. As the ZSI _Dispatch method does not allow for additional parameters, we need to store the current message ID (so we can set it in the results) on the instance of the server. Of course, in a high traffic server this solution will never work as you can expect more than one incoming request at a time. To get around this we would either need to somehow pass this information through ZSI, or create a separate instance of the server for every request.

Once we have the results, whether they are a valid response, or a fault, we send the results off in a e-mail message. You'll notice that when we are sending an e-mail message, we use the common method common.SendMessage (see Listing 2). This helper function sends a message to a specified server using a different thread. This is needed so that the sending process is not blocked by waiting for a server connection, network delays, etc. In a production system, this function would have to be more complex and handle issues of network failure, a server's going down, etc. with some sort of queuing mechanism.

		
    def process_message(self, peer, mailfrom, rcpttos, data):
        #Parse the message into an email.Message instance
        p = Parser.Parser()
        m = p.parsestr(data)
        print "Received Message"

        #See if it is a SOAP request
        if m.has_key('To') and m['To'] == 'calendar@localhost':
            self.process_soap(m)
        else :
            #In normal circumstances, this would probably
            #forward the email message to another SMTP Server
            print "Unknown Email message"
            print m

    def process_soap(self,message):

        #Parse the SOAP Message
        ps = parse.ParsedSoap(message.get_payload(decode=1))

        #Store the current ID
        self.currentID = message['Message-Id']
        print "Processing Message: " + self.currentID

        #Use ZSI's dispatcher to call the correct function based on the message.
        dispatch._Dispatch(ps,
                           [self],
                           self.send_xml,
                           self.send_fault)

    #ZSI Callback to send an SOAP(non-Fault) response.
    def send_xml(self,xml):
        self.return_soap(xml)

    #ZSI callback to send a fault.
    def send_fault(self,fault):
        sys.stderr.write("FAULT While processing request:\n");
        s = cStringIO.StringIO()
        fault.serialize(s)
        st = s.getvalue()
        print st
        #Serialize the fault and send it to the client
        self.return_soap(st)

    #Called by our code to send result XML.
    def return_soap(self,st):
        msg = MIMEText.MIMEText(st)

        msg['Subject'] = "Test Message"
        msg['To'] = 'calendar@localhost'
        msg['From'] = 'Mike.Olson@Fourthought.com'
        msg['Message-Id'] = "2"
        msg['In-Reply-To'] = self.currentID or 0

        print "Sending Reply"
        common.SendMessage("127.0.0.1",8024,"me@fourthought.com",
        ["Mike.Olson@Fourthought.com"],msg)
                         
    #Implementation of our SOAP Service.
    def getMonth(self,year,month):
        print "Request for %d,%d" % (year,month)
        return calendar.month(year, month)

Back to top

A SOAP SMTP client

An SMTP SOAP client needs to listen for incoming mail messages that represent replies, so it looks a lot like a server implementation. We again override smtpd.SMTPServer to create a listener (on a different port) to listen for replies (see Listing 3). One of the big differences between the server and the client listener is that we start the listener off in a separate thread. This allows us to have one thread that is handling user input, and a second, our listener, that is listening and processing responses.

Another difference is that the listener contains a dictionary of "responses" that map "In-Reply-To" IDs to call-back methods. Using this approach, the main input thread can shoot off as many requests as it would like as long as it registers each with the listener. The main thread can then continue on its merry way then, whenever a responses comes in, the listener invokes the call-back to handle the response.

		
class ClientServer(smtpd.SMTPServer):
    #A simple server to receive our SOAP responses.
    #The responses dictionary is a mapping from
    #Message ID to a callback to handle the response.
    responses = {}

    #this is the method we must override in to handle SMTP messages
    def process_message(self, peer, mailfrom, rcpttos, data):
        #Parse the message into a email.Message instance
        p = Parser.Parser()
        m = p.parsestr(data)
        #See if this is a reply that we were waiting for.
        if m.has_key('In-Reply-To') and self.responses.has_key(m['In-Reply-To']):
            mID = m['In-Reply-To']
            #Invoke the response callback with the parsed SOAP.
            self.responses[mID](mID,parse.ParsedSoap(m.get_payload(decode=1)))
            del self.responses[mID]
        else:
            #In a product server, this would probably forward the message to another
            #SMTP Server.
            print "Unknown Email message"
            print m
            
    #method used to register that we are expecting a response from the server.
    def expectResponse(self,mId,callback):
        self.responses[str(mId)] = callback

User input is gathered in the HandleInput method (see Listing 4). This asks the user to enter in a month and a year. Then it creates a SOAP request and sends it to the server. It also registers this request with the listener so that the listener knows to expect a response. In this example, all of the requests are registered with the DisplayResults call-back. This method simply displays the results to the screen.

		
def DisplayResults(ID,ps):
    #This method is the generic callback used by all requests.
    #It uses the parsed SOAP to print out the results.
    print "\nResults for ID: " + ID
    tc = TC.String()
    data = _child_elements(ps.body_root)
    if len(data) == 0: print None
    print tc.parse(data[0], ps)

def HandleInput(server):
    #This method is used to query the user for a year and a month.
    #When one is received, then a new message is sent, and the server
    #is told to expect the results
    done = 0
    lastID = 1
    while not done:
        year = raw_input("Year of request(Return to exit): ")
        if not year: done = 1
        else:
            year = int(year)
            month = int(raw_input("Month of request: "))

            lastID += 1
            mID = lastID
            
            msg = MIMEText.MIMEText(BODY_TEMPLATE%(year,month))

            msg['Subject'] = "Test Message"
            msg['To'] = 'calendar@localhost'
            msg['From'] = 'Mike.Olson@Fourthought.com'
            msg['Message-Id'] = str(mID)

            server.expectResponse(mID,DisplayResults)
            print "Sending out message ID: " + str(mID)
            common.SendMessage("127.0.0.1",8023,"me@fourthought.com",
             ["Mike.Olson@Fourthought.com"],msg)

def StartServer():
    #Start up our response server in another thread.
    server = ClientServer(("localhost", 8024),
                          (None, 0))
    def run():
        try:
            asyncore.loop()
        except KeyboardInterrupt:
            pass
    print "Starting Client Server"
    t = threading.Thread(None,run)
    t.start()
    return server

if __name__ == '__main__':
    server = StartServer()
    HandleInput(server)

Back to top

Running the example

The client application queries the user for a month and a date. It will let you make as many queries as you'd like. Once a valid reply is returned, it will be printed to the screen. You'll notice that you might not get all the replies back in the same order that they were sent. The output is shown in Listing 5.

		
[molson@penny src]$ python client.py
Starting Client Server
Year of request(Return to exit): 2003
Month of request: 1
Sending out message ID: 2
Year of request(Return to exit): 2004
Month of request: 1
Sending out message ID: 3
Year of request(Return to exit): 2005
Month of request: 1
Sending out message ID: 4
Year of request(Return to exit):
Results for ID: 2
January 2003
Mo Tu We Th Fr Sa Su
       1  2  3  4  5
 6  7  8  9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31

Results for ID: 3
January 2004
Mo Tu We Th Fr Sa Su
          1  2  3  4
 5  6  7  8  9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31

Results for ID: 4
January 2005
Mo Tu We Th Fr Sa Su
                1  2
 3  4  5  6  7  8  9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31

Back to top

Next column

In the next installment of this column, we will look into what is required to interact with Google's SOAP API.

Back to top

Download

Information about download methods

Resources

• Participate in the discussion forum.
  
  
• Download the zip file with the examples used in this article.
  
  
• Read the previous column of this series on RSS for Python or check out all the Python web services developer columns in this series.
  
  
• Visit the ZSI download page.
  
  
• Get more information on the SOAP Email Binding.
  
  
• Read about RFC 821 (SMTP).
  
  
• Learn more about about RFC 1869 (ESMTP).
  
  

About the authors

Mike Olson is a consultant and co-founder of Fourthought Inc., a software vendor and consultancy specializing in XML solutions for enterprise knowledge management applications. Fourthought develops 4Suite, and 4Suite Server, open source platforms for XML middleware. You can contact Mr. Olson at mike.olson@fourthought.com.

Uche Ogbuji is a consultant and co-founder of Fourthought Inc., a software vendor and consultancy specializing in XML solutions for enterprise knowledge management applications. Fourthought develops 4Suite, and 4Suite Server, open source platforms for XML middleware. Mr. Ogbuji is a Computer Engineer and writer born in Nigeria, living and working in Boulder, Colorado, USA. You can contact Mr. Ogbuji at uche@ogbuji.net.

Rate this article

Comments

Back to top

Table of contents

• Introduction
• Processing SOAP messages
• Linking requests and responses
• Example dependencies
• A SOAP SMTP server
• A SOAP SMTP client
• Running the example
• Next column
• Download
• Resources
• About the authors
• Comments