Simple server-side 2G phone apps

Build applications that work with billions of phones around the world with a little server-side scripting

Mobile phones are transforming economies and societies all over the world, but often with phones that might be considered out-of-date by gadget geeks in more developed nations. The good news is that applications that work with these phones can be very simple to write, and they give your application a huge potential user base. In this article, learn how to write programs that respond to specialized requests for information from 2G phones.

Share:

Bob DuCharme, Solutions Architect, TopQuadrant

Photo of Bob DuCharmeBob DuCharme is a solutions architect at TopQuadrant, a provider of software for modeling, developing, and deploying Semantic Web applications. He's written four books and nearly one hundred online and print articles about information technology without using the word "functionality" in any of them. See http://www.snee.com/bobdc.blog for his weblog.



25 May 2010

Also available in Chinese Russian Japanese

These days, tech news is full of stories about the latest iPhone, Droid, and Palm Pre™ apps, but news outlets with a wider perspective have taken notice of how simpler mobile phones are providing new communication and entrepreneurship opportunities around the world, often in places with no steady supply of electricity. This is the only exposure that people in many of these places have ever had to the Internet, and they're using their phones for more than just talking, as SMS text messaging lets them exchange information that helps them to run their often tiny businesses.

Frequently used acronyms

  • SMS: Short Message Service
  • XML: Extensible Markup Language

Closer to home, how many of your friends and family don't have a color, hi-res, touch screen phone with the ability to browse Web sites and install a wide variety of specialized apps? They're using 2G, or second generation phones. When these were introduced in the early 1990s, they were different from their first generation predecessors because they were digital and could send text messages. For people on a budget, 2G phones and their monthly fees are much easier on the wallet, and for most people in the world, 3G isn't even an option yet. Last year, Apple sold 25 million iPhones, which sounds like a lot, but when you look at the International Telecommunication Union's recent estimate that 5 billion of the world's 6.8 billion people will have mobile phone subscriptions by the end of 2010, it gives you an idea of the relatively low worldwide use of 3G phones for the next few years.

2G phones can send text messages to e-mail addresses, and writing a script to automate a response to an e-mail message based on its content is not difficult—especially when you can be confident that your script will be responding to messages of 160 characters or less. Put these together and you'll see how you can write applications that will look, to most 2G phone owners, like specialized sources of information that can process their requests. If you, as the developer, think of those phones as little terminals passing arguments to functions that you write, you'll see how simple it is to provide information services to owners of simple, inexpensive phones.

As an example, look at a service that accepts a text message consisting of a three-digit U.S. area code and returns information about that area code. To use it, suppose that the "Missed Calls" list on my mobile phone shows that someone with an area code of 407 tried to call me. If I wonder where that area code is, I send the SMS text message "407" to my Area Code Information service from my 2G phone and it sends the following text message back: Florida (Orlando, Florida, St. Cloud and central eastern Florida). (For the purposes of this article, the service's e-mail address is acinfo@snee.com, but in the actual working app, which you can try yourself, the e-mail address is "aci" instead of "acinfo".)

The basic steps of the application, all of which use simple scripting, are these:

  1. Check all incoming e-mail and, when something comes for acinfo@snee.com, route it to the Python script aci.py which performs the remaining steps.
  2. Search a list of area code information for the text in the body of the received e-mail.
  3. If it's in the list, set the return message to the stored information about it (in the above example, Florida (Orlando, Florida, St. Cloud and central eastern Florida)).
  4. If it's not in the list, set the return message to say that no information was found about that incoming message.
  5. Send the return message back to the address that sent the original message and log it.

My application did the information lookup by searching through a simple text file, but for your application, you're only limited by your imagination and the sources of data that are accessible to your script.

Checking incoming e-mail and routing it to the right processing program: procmail

The key to automating the response to the incoming message is a grand old UNIX® utility called procmail. Many of the earliest systems that scanned for spam and automatically sorted e-mail into specific folders based on the information in the e-mail header were built on procmail, and it's still there to build on. If you have an account with a host provider that uses a Linux®-based system and provides shell access, you should be able to create a procmail configuration file for your account that scans for patterns in incoming e-mails and performs actions based on what it finds.

For the e-mail to be routed through this configuration file, which is called .procmailrc, another step or two might be necessary. In the old days, you'd create a .forward file that routed the e-mail there, but now it's common for your host provider to have a Web form for you to fill out to tell their system to check your .procmailrc file when e-mail arrives.

Once I configured my account at my host provider to do this, I added a rule to the .procmailrc file with the following three lines:

:0
* ^To: <?acinfo@snee.com>?
| /usr/home/bobd/aci/aci.py

The first line indicates that this is the beginning of a procmail recipe. The second line begins with an asterisk to show that you're specifying a condition, and the rest of that line has a regular expression that indicates what to search for in the e-mail: "To: acinfo@snee.com" starting right at the beginning of a line, with optional angle brackets around the e-mail address. (The fact that these angle brackets might or might not be there is only the first example of the inconsistency you must plan for when processing e-mail that might come from a variety of e-mail clients and phones.) I created this e-mail address to be used only for area code information requests, so the rule applies to all messages sent to that address.

The third line of a .procmailrc rule can name a mailbox where the message should be forwarded, but this recipe does something more interesting. The pipe symbol indicates that the message's contents should be sent as input for the indicated program: a Python script named aci.py.


Parsing the input and choosing a scripting language

Before you look at the aci.py program, look at the input that it must process. An SMS text message shows up as an e-mail with a sender address based on the phone number and a domain name used by the phone company. Listing 1 shows a sample SMS e-mail that showed up when I sent the area code 407 as a text message from an LG env2™ phone on the Verizon network, with the originating phone number changed to (434) 000-0000 in the listing.

Listing 1. E-mail version of an SMS text message
From 4340000000@vtext.com Wed Mar 10 00:50:01 2010
Return-Path: <4340000000@vtext.com>
Delivered-To: bobd-snee:com-acinfo@snee.com
X-Envelope-To: acinfo@snee.com
Received: (qmail 21729 invoked from network); 10 Mar 2010 00:50:00 -0000
Received: from mailwash38.pair.com (66.39.2.38)
         by oomur.pair.com with SMTP; 10 Mar 2010 00:50:00 -0000
Received: from localhost (localhost [127.0.0.1])
         by mailwash38.pair.com (Postfix) with SMTP id 021054142C
         for <acinfo@snee.com>; Tue,  9 Mar 2010 19:50:00 -0500 (EST)
X-Spam-Check-By: mailwash38.pair.com
X-Spam-Status: No, hits=2.9 required=4.0 tests=BAYES_00, FROM_STARTS_WITH_NUMS,
         MISSING_SUBJECT, TVD_SPACE_RATIO autolearn=no version=3.002005
X-Spam-Flag: NO
X-Spam-Level: **
X-Spam-Filtered: a7b240700a36d5e6c2608f9ce43a92c9
Received: from lrx5634xmtasa.alltel.net (lrx5634xmtasa.alltel.net
         [205.142.19.193])
         by mailwash38.pair.com (Postfix) with ESMTP id 3FF774142F
         for <acinfo@snee.com>; Tue,  9 Mar 2010 19:49:59 -0500 (EST)
X-Policy: RELAYLIST-$RELAYED
Received: from unknown (HELO ifs2006qwigfe) ([10.135.9.57])
         by lrx5634xmtasa.alltel.net with ESMTP; 09 Mar 2010 18:49:58 -0600
Message-ID: <26597005.1268182198841.JavaMail.root@ifs2006qwigfe>
From: 4340000000@vtext.com
To: acinfo@snee.com
Subject: 
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Date: Tue,  9 Mar 2010 19:49:59 -0500 (EST)

407

There's a lot there, but the script only needs two pieces of information: the e-mail address of the device sending the message (4340000000@vtext.com) and the message that it sent (407, on the last line). Perl is usually the first language I go to for simple text processing scripts, and it wasn't much trouble to write a Perl script that pulled the e-mail address and message out of the e-mail in Listing 1, looked up the message in an area code list, and sent the requested information back to the e-mail address that represents the sending phone.

This Perl script worked fine on messages from my phone, but as I tested it with more phones, I found that e-mail messages sent by phones aren't as consistent as I'd hoped. I've already mentioned that the .procmailrc file must account for e-mail addresses that may or may not be enclosed in angle brackets, which is simple enough for Perl to handle. It turned out that the rest of the e-mail message structure had other potential differences to account for.

Listing 2 shows a more complex message, which brings the message "305" as a multipart MIME message from an iPhone. (This is certainly not a 2G phone, but I thought it would be a good idea to test with it.) Don't look too closely for a "305" in there, because it's encoded. Finding the right part of the message to decode and then doing so was making my Perl script longer and longer, and it had already grown from accommodating several other phones.

Listing 2. More complex e-mail representation of an SMS message
From 6170000000@mms.att.net Sun Feb 28 21:00:03 2010
Return-Path: <6170000000@mms.att.net>
Delivered-To: bobd-snee:com-acinfo@snee.com
X-Envelope-To: acinfo@snee.com
Received: (qmail 18219 invoked from network); 28 Feb 2010 21:00:03 -0000
Received: from mailwash38.pair.com (66.39.2.38)
         by oomur.pair.com with SMTP; 28 Feb 2010 21:00:03 -0000
Received: from localhost (localhost [127.0.0.1])
         by mailwash38.pair.com (Postfix) with SMTP id B1D8A41430
         for <acinfo@snee.com>; Sun, 28 Feb 2010 16:00:02 -0500 (EST)
X-Spam-Check-By: mailwash38.pair.com
X-Spam-Status: No, hits=3.0 required=4.0 tests=BAYES_20, FROM_STARTS_WITH_NUMS,
         TVD_SPACE_RATIO autolearn=no version=3.002005
X-Spam-Flag: NO
X-Spam-Level: ***
X-Spam-Filtered: a7b240700a36d5e6c2608f9ce43a92c9
Received: from schemailmta08.cingularme.com (schemailmta08.cingularme.com
         [209.183.37.70])
         by mailwash38.pair.com (Postfix) with ESMTP id F35394142C
         for <acinfo@snee.com>; Sun, 28 Feb 2010 16:00:00 -0500 (EST)
X-Mms-MMS-Version: 18
Date: Sun, 28 Feb 2010 15:13:10 -0600
X-Nokia-Ag-Internal: ; smiltype=false; internaldate=1267391590642
Content-Type: multipart/mixed;
         boundary="----=_Part_9705244_14454315.1267391590647"
Received: from schagw01 ([172.16.130.170]) by schemailmta08.cingularme.com
         (InterMail vM.6.01.04.00 201-2131-118-20041027) with ESMTP id
         <20100228210001.QHEZ5910.schemailmta08.cingularme.com@schagw01>
         for <acinfo@snee.com>; Sun, 28 Feb 2010 15:00:01 -0600
X-Mms-Transaction-ID: 1267390700-6
From: <6170000000@mms.att.net>
To: acinfo@snee.com
Mime-Version: 1.0
Message-ID: <33144584.1267391590647.JavaMail.wluser@schagw01>
X-Mms-Message-Type: 0
Subject: Multimedia message
X-Nokia-Ag-Version: 2.0

------=_Part_9705244_14454315.1267391590647
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: base64
Content-Disposition: inline

MzA1
------=_Part_9705244_14454315.1267391590647--

At this point I remembered that an important criterion for picking a programming language is what libraries are available to take care of the more mundane tasks in your application, and parsing the e-mail header, finding the part with the message, and decoding the message if necessary are definitely mundane tasks. On CPAN, I found a Perl module to parse through a variety of e-mail message headers so that I could just retrieve the information that I needed with function calls regardless of the vagaries of the potentially different header formats. However, this library depended on other Perl libraries, and my host provider had an outdated version of one of those, so I investigated Python's offerings in this area. I found the Python email package, wrote a few simple tests, and decided to redo my program in Python (see Resources for a link to the e-mail package). Some quick Web searches found similar libraries available for Ruby, Java™, PHP, and other programming languages, so you're certainly not limited to Perl or Python if you want to write an e-mail autoresponder script.


The autoresponder script

Listing 3 shows the aci.py script; note how the import statements at the beginning pull in the email.Parser library along with several other popular Python libraries. The __main__ section at the bottom holds the program's basic logic: parse the incoming message, pull the message sender e-mail address and message body (stored in the variable areaCode) out of it, search for information about that particular area code using the areaCodeInfo function defined elsewhere in the script, send this information as a reply, and then log the message.

Listing 3. aci.py script
#!/usr/local/bin/python

# aci.pl: (area code information) read e-mail message to find area
# code, then send information about that area code.
# Bob DuCharme 2010-01 no warranty expressed or implied

import os
import email.Parser
import sys
import datetime
import re

def multipartBody(msg):
# following code from 
# http://docs.python.org/library/email-examples.html

  partCounter=1

  for part in msg.walk():
    if part.get_content_maintype()=="multipart":
      continue
    name=part.get_param("name")
    if name==None:
      name="part-%i" % partCounter
    partCounter+=1
    msgText = part.get_payload(decode=1)

  msgSender = msgSenderText
  return msgText.strip()      # strip whitespace


def areaCodeInfo(areaCode):

  # Look for data about that area code in areacodes.txt.
  # First initialize values that should get overridden.

  response = "No information available for area code " + areaCode + "."
  foundAreaCode = False
  line = "dummy"
  acFile = open(aciPath + "areacodes.txt")
  while ((not foundAreaCode) and line):
    line = acFile.readline()
    if (line[0:5] == areaCode + ": "):   # e.g. "212: " 
      response = line
      foundAreaCode = True

  return response


def sendReply(msgSender,response):

  f = os.popen("%s -t" % SENDMAIL, "w")
  f.write("To: " + msgSender + "\n")
  f.write("From: area code information <acinfo@snee.com>\n")
  f.write("Return-Path: area code information <acinfo@snee.com>\n")
  f.write("Content-type: text/plain\n\n")
  f.write(response)
  sts = f.close()

def  logIt(msgSenderText,areaCode):

  timestamp = datetime.datetime.today().isoformat()[0:19] 
  log = open(aciPath + "log.txt",'a')
  log.write(timestamp + " " + msgSenderText + " " + areaCode + "\n") 
  log.close()


if __name__ == "__main__":

  SENDMAIL = "/usr/sbin/sendmail" # sendmail location
  aciPath = "/usr/home/bobd/aci/"
  keepFullLog = False # For debugging. More detailed than log.txt.
  response = ""

  # Parse the standard input to find the message and sender value
  mailFile=sys.stdin
  p=email.Parser.Parser()
  msg=p.parse(mailFile)
  mailFile.close()

  msgSenderText = msg['From']  # text showing msg sender's name
  msgSender = msgSenderText    # save msgSenderText for log

  # If msgSender has the form "Some Guy <someguy@example.com>" 
  # then we just want someguy@example.com
  emailAddrRegEx = re.compile(r"\<(?P<emailAddr>.+)\>")
  result = emailAddrRegEx.search(msgSender)
  if result != None:
      msgSender = result.group('emailAddr') 

  if keepFullLog:
    output = open(aciPath + "fulllog.txt",'a')
    output.write(str(msg) + "\n-- end of mail msg --\n\n")
    output.close()

  if msg.has_key("X-Mailer") and \
        msg["X-Mailer"][0:24] == "Microsoft Office Outlook":
    response = "Microsoft Outlook format is not supported."
    areaCode = ""
  else:
    areaCode = multipartBody(msg)
    response = areaCodeInfo(areaCode)

  sendReply(msgSender,response)
  logIt(msgSenderText,areaCode)

Getting a Common Short Code address for your app

You might have noticed that some SMS-based apps have their users send messages to a special five-digit phone number such as 46645 (GOOGL) for Google or 40404 to post a message to Twitter. Wouldn't it be great if your users could send their text messages to one of these Common Short Codes (CSCs) instead of an e-mail address? It's possible, but they're expensive to register, especially if you want to pick your own five-digit number. These are administered by the Common Short Code Administration (CSCA), an agent for the International Association for the Wireless Telecommunications Industry. Once you enter into a contract to pay the monthly fee for the use of a particular CSC, you must still negotiate with each wireless carrier to activate that short code, and U.S. ones don't work outside of the U.S. The CSCA's Web site has more information (see Resources).

A few notes:

  • The logIt function saves a single line to the text file log.txt. This line records when it was logged, the e-mail address of the mobile unit that requested the information, and what area code the unit wanted to know about. A sample line could look like this: 2010-03-19T19:50:02 4340000000@vtext.com 407. If the boolean variable keepFullLog is set to True, the program saves the entire incoming message in a different log file. This gets pretty verbose, but was invaluable for debugging.
  • The e-mail message format sent by Microsoft® Outlook® was beyond the comprehension of the email.Parser library, but no 2G phone is going to be sending that. The script sends an appropriate response when it detects an Outlook message.
  • To send the e-mail message, the script uses another venerable UNIX utility called sendmail. This isn't some special library with functions to call, but operates more in keeping with the UNIX philosophy of assembling application components by piping information from one component to another. The Python script treats sendmail this way by opening it as if it were a text file, writing the appropriate information to it, and then "closing" it.
  • The areaCodeInfo function searches for information in the file areacodes.txt. An excerpt of this file, which I based on a Wikipedia page, is in Listing 4.

Listing 4. Excerpt from areacodes.txt
210: Texas (San Antonio area)
211: Community Services Hotline (e.g., crisis line, United Way, etc.)
212: New York (Manhattan except for Marble Hill)
213: California (central Los Angeles)
214: Texas (Dallas area)

An application that searches a text file for a string and returns the line where it found that string is a very, very simple application. When you develop your own application to respond to a message from a 2G phone, this is where you can get more creative: Your program can do database queries on any combination of local and remote stores, cross-reference the information it finds, and perform all kinds of logic to return something useful to the phone that sent the query, as long as it fits in the 160-character limit of an SMS message.

Think of the mobile phone as a client running a command-line interface to your application, and remember that your application can be a simple script that lets other libraries do the difficult and complicated work.You'll see that server-side application development for the billions of phones used around the world can be surprisingly easy.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into XML on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=XML, Open source, Web development
ArticleID=491163
ArticleTitle=Simple server-side 2G phone apps
publish-date=05252010