Creating mashups on the Google App Engine using Eclipse, Part 3: Using RESTful Web services

Michael Galpin (mike.sr@gmail.com), Software architect, eBay

Michael Galpin has been developing Java software professionally since 1998. He currently works for eBay. He holds a degree in mathematics from the California Institute of Technology.

Summary: Social networks are making it easier to take data and mash it up to create innovative Web applications. You still, however, must deal with all the usual issues with creating a scalable Web application. Now the Google App Engine (GAE) makes that easier for you. With it, you can forget all about managing pools of application servers, and, instead, you can concentrate on creating a great mashup. In this article, the last of a three-part "Creating mashups on the Google App Engine using Eclipse" series, we will take the application built in the first two parts and further enhance it. We will add the ability to view other users of the app and subscribe to their aggregate feeds. We will then complete the mashup circle by exposing the app as a Web service that can be used by other mashups.

View more content in this series

Tag this!

Date: 19 Aug 2008
Level: Intermediate
Activity: 1260 views

About this series

In this series, we look at how to get started with the Google App Engine (GAE). In Part 1, we look at how to get a development environment set up so you can start creating an application that will run on the GAE. We also saw how to use Eclipse to make developing and debugging your application easier. In Part 2, we enhance the application by adding some Ajax features. We also saw how to monitor the application once it was deployed to the GAE. Here in Part 3, we will give back to the ecosystem by creating RESTful Web services to our application, so other folks can use it to create their own mashups.

The GAE is a platform for creating Web applications. The biggest prerequisite for it is knowledge of Python, as this is the programming language used on it (currently, Python V2.5.2). For this series, it would be helpful to have some typical Web development skills (e.g., knowledge of HTML, JavaScript, and CSS). To develop for the App Engine, you will need to download the App Engine SDK (see Resources). In this series, we also use Eclipse V3.3.2 to aid in our GAE development (see Resources). And you'll need the PyDev plug-in to turn Eclipse into a Python IDE.

Subscriptions

So far, our application, aggroGator, allows a user to mash up several popular Web services and create an aggregate feed of those services. Now, to make things a little more interesting, we want to allow users to subscribe to other users' feeds (where each user's feed is quite possibly an aggregate of feeds itself.) For example, let's say we want to set up an account to subscribe to ourself on Twitter, last.fm, and del.icio.us, so friends can then subscribe to our aggroGator feed to see all of the activity on those services. To handle this, we need to once again revisit our data model.

Modeling

To enable subscriptions, we need to allow one user (account) to subscribe to a list of other accounts. One approach we could take would be to add a list of users to each account. Each user adds a subscription we could add to this list. The code for this would look something like Listing 1.

Listing 1. Account model with user list


class Account(db.Model):
    user = db.UserProperty(required=True)
    subscriptions = db.ListProperty(Account)

There are some advantages to this approach. When we retrieve an account, we get all of the other accounts that it is subscribed to. This is a common tactic to use with nonrelational data stores like the GAE's Bigtable: Keep all relevant data together and do not worry about things like normalization. However, there is a disadvantage to this approach. What if we want to show who all is subscribed to a particular user. The only way to do this would be to retrieve all Account models, look at all the subscriptions, and see if the given user is in the list. Alternatively, we could keep two lists in each Account model — one for the subscriptions and one for the subscribers. Instead of taking this approach, we will use a more traditional many-to-many model, as shown in Listing 2.

Listing 2. Subscribe model


class Subscribe(db.Model):
    subscriber = db.ReferenceProperty(Account, required=True, 
collection_name='subscriptions')
    subscribee = db.ReferenceProperty(Account, required=True, 
collection_name='subscribers')

As you can see, this is similar to a join table you would use with a relational database. Just because the GAE uses a nonrelational data store (Bigtable) does not mean that you cannot leverage techniques you have used with relational databases. Now that we have the data model in place, let's take a top-down look at how these many-to-many relationships will be created from the end user's perspective.

Subscription management

Our application can store subscriptions, so we just need some way for users to create subscriptions. To do this, we will create a page for users to add subscriptions (see Listing 3).

Listing 3. Accounts page


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/
    xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8" />
    <title>Aggrogator Accounts</title>
    <link rel="stylesheet" href="/css/aggrogator.css" type="text/css" />
    <script type="text/javascript" src="/js/prototype.js"></script>
    <script type="text/javascript" src="/js/builder.js"></script>
    <script type="text/javascript" src="/js/effects.js"></script>
    <script type="text/javascript" src="/js/aggrogator.js"></script>
  </head>
  <body>

    <img id="spinner" alt="spinner" src="/gfx/spinner.gif" style="display: none; 
position: fixed;" />

    <div id="logout">
      {{ account.user.nickname }}
      <a href="{{ logout_url }}">Logout</a>
    </div>

    <div class="clearboth"></div>

    <ol>
      {% for acc in all_accounts %}
        <li>
          <a href="" onclick="subscribe('{{ acc.user.email }}'); return false;">
{{ acc.user.email }}</a>
        </li>
      {% endfor %}
    </ol>

  </body>
</html>

As you can see, this page simply displays all of the accounts in the system. The user then picks an account to subscribe by clicking on it. You could imagine a more sophisticated interface. For example, this would become unwieldy with a large number of users, so a search-based system would be better. Or perhaps a system allowing the user to import his address book or use APIs from something like OpenSocial to find existing friends who are already part of the application. The template above needs a list of users, so let's take a quick look at the controller that creates the model for the page (see Listing 4).

Listing 4. Accounts page controller


#Accounts Module
class MainPage(webapp.RequestHandler):
    def get(self):
        # get the current user
        user = users.get_current_user()

        # is user an admin?
        admin = users.is_current_user_admin();

        # create user account if haven't already
        account = aggrogator.DB.getAccount(user)
        if account is None:
            account = aggrogator.Account(user=user)
            account.put()

        # create logout url
        logout_url = users.create_logout_url(self.request.uri)

        all_accounts = aggrogator.Account.all()

        template_values = {
            'account': account,
            'admin': admin,
            'logout_url': logout_url,
            'all_accounts': all_accounts,
            }
        path = os.path.join(os.path.dirname(__file__), 'accounts.html')
        self.response.out.write(template.render(path, template_values))

The controller gets all the data ready for display on the accounts page. Back on the accounts page in Listing 3, we see that there is JavaScript called when an account is clicked on.

Listing 5. Subscription JavaScript


function subscribe(email) {
  new Ajax.Request("/accounts/subscribe", {
    method: "post",
    parameters: {'email': email},
    onSuccess: alert('subscribed to ' + email)
  });
}

This JavaScript once again uses the Prototype library to make an Ajax request to the server. We call the URL /accounts/subscribe. Where is that URL mapped to? The code that creates the mapping is in the main function of the new accounts module, as shown below.

Listing 6. URL mappings for accounts module


def main():
    app = webapp.WSGIApplication([
        ('/accounts/', MainPage),
        ('/accounts/subscribe', Subscribe),
        ], debug=True)
    util.run_wsgi_app(app)

if __name__ == '__main__':
    main()

As you can see from the main function, a call to /accounts/subscribe is handled by a Subscribe controller class. That class is shown in Listing 7.

Listing 7. Subscribe controller class


class Subscribe(webapp.RequestHandler):
    def post(self):
        # get the current user
        user = users.get_current_user()
        email = self.request.get('email')

        aggrogator.DB.create_subscription(user, email)

This controller is simple. It gets the current user (the subscriber) and the e-mail address of the subscription being added. It then calls a new method in the DB utility class we have used previously. That class handles all our Bigtable-related calls. The new create_subscription function is shown below.

Listing 8. DB function for create_subscription


class DB:
    @staticmethod
    def create_subscription(user, email):
        subscriber = DB.getAccount(user)
        subscribee = DB.getAccountForEmail(email)
        subscription = Subscribe.gql("WHERE subscriber = :1 AND subscribee = :2", 
subscriber, subscribee).get()
        if subscription is None:
            Subscribe(subscriber=subscriber, subscribee=subscribee).put()

    @classmethod
    def getAccountForEmail(cls, email):
        user = users.User(email)
        return cls.getAccount(user)

This function first looks up the Account models for the user and subscription e-mail. For the latter, it uses the new getAccountForEmail function. This makes use of the GAE's user's API to look up the User object based on the e-mail, then querying Bigtable for the account. Once we have both accounts, we check to see if the subscription already exists. If it does not, we create a new one.

Of course, now that we have subscriptions, we want to make use of them in the main application. Instead of just showing the current user's services, we want to show the aggregate feed (the user's services and the services from his subscriptions, as well). To do this, we make a small change in the GetUserServices controller in the main module developed in previous articles. This is shown below.

Listing 9. Modified GetUserServices controller


class GetUserServices(webapp.RequestHandler):
    def get(self):
        user = users.get_current_user()

        # get the user's services from the cache
        #userServices = aggrogator.Cache.getUserServices(user)
        userServices = aggrogator.Aggrogator.get_services(user.email())

        stats = memcache.get_stats()
        self.response.headers['content-type']  = 'application/json'
        self.response.out.write(simplejson.dumps({'stats': stats, 'userServices': 
userServices}))

All we did here was call a new library class: the appropriately named aggrogator class, to get the aggregate services instead of just the user's. This library code is shown below.

Listing 10. The aggrogator Library: Retrieve aggregate services


class aggrogator:
    @staticmethod
    def get_services(username):
        accounts = []
        
        primary = DB.getAccountForEmail(username)
        accounts.append(primary)

        for subscription in primary.subscriptions:
            accounts.append(subscription.subscribee)
        services = []
        for account in accounts:
            services.extend(Cache.getUserServices(account.user))   
        return services

Here is where we can again see our new Subscribe model in action. In the code, we get the account for the username (by using the getAccountForEmail function we saw earlier), then call its subscriptions property. In this case, we only use this to get all of the services from the cache. Later, we will see these services used to create the aggregate feed.

We are almost ready to test the new accounts page. We have to make one last change: We need to configure our application to send certain URL requests to the new accounts module. To do this, we edit the app.yaml file and add a new section.

Listing 11. Addition to app.yaml


- url: /accounts/.*
  script: accounts.py
  login: required

This is just a new section of the file. It maps any request that has /accounts/ to the accounts module. This should appear before the catch-all handler used previously (url: /.*) so that it takes precedence. Now we can test the application just as before, using Eclipse and PyDev, and by going to http://localhost:8080/acounts/. Make sure you create multiple accounts so your testing can be interesting.

The aggroGator Web service

Social Web services make it possible to create interesting applications like aggroGator very easily. The GAE allows us to create such mashups that are also very scalable. So of course it makes sense to create an API/Web service around our mashup so others can use it to create their own interesting mashups. This turns out to be quite easy, as well.

For our Web service, we will start by making it a read-only service. The service will simply provide the aggregate feed for a user (i.e., the same thing one would see in the aggroGator UI). We will use a simple REST-style URL for this, such as /api?username=my@email.address. This time, we will start bottom-up. To handle such a URL, we once again add a section to our app.yaml file.

Listing 12. Addition to app.yaml


- url: /api
  script: main.py

Notice that we are still sending the /api requests to the main module. Why did we need a new mapping in app.yaml? We do not want to require authentication for the aggroGator API. That is the only reason we need the new rule in app.yaml. Since we leverage the main module, it needs to be modified.

Listing 13. New touting rule for main module


def main():
    app = webapp.WSGIApplication([
        ('/', MainPage),
        ('/addService', AddService),
        ('/getEntries', GetEntries),
        ('/api', AggroWebService),
        ('/getUserServices', GetUserServices),
        ], debug=True)
    util.run_wsgi_app(app)

if __name__ == '__main__':
    main()

All we have done to this function is add one entry to the list of mappings. We are mapping /api to the AggroWebService controller class. That class is shown below.

Listing 14. The AggroWebService controller class


class AggroWebService(webapp.RequestHandler):
    def get(self):
        self.response.headers['content-type'] = 'text/xml'
        username = self.request.get('username')
        entries = aggrogator.Aggrogator.get_feed(username)
        str = u"""<?xml version="1.0" encoding="utf-8"?><entries>"""
        for entry in entries:
            str += entry.to_xml()
        str += "</entries>"
        self.response.out.write(str)

The service starts off by retrieving the username request parameter. It then uses the aggroGator library we saw earlier, but uses a different method, get_feed, to get the aggregate entries. The code for that library function is shown below.

Listing 15. aggroGator get_feed


class Aggrogator:
    @staticmethod
    def get_feed(username):
        services = Aggrogator.get_services(username)
        entries = []
        for svc_tuple in set((svc['service'], svc['username']) for svc in services):
            entries.extend(Cache.getEntries(*svc_tuple))
        entries.sort(key=operator.attrgetter('timestamp'), reverse=True)
        return entries

This library function uses the get_services function we saw in Listing 10 to retrieve the aggregate services. It then iterates over the services. The code uses a set to make sure that the services are unique (i.e., if a user had subscribed to two other users who each used the same service). Because we used a set, we have to use tuple as we can only use an immutable object. Finally, we sort all of the entries by their timestamp in descending order (latest entries listed first).

Going back to Listing 14, once we have the list of entries, we then use some simple string concatenation to create an XML document. We use a to_xml() method on each Entry instance. This is a new method, shown below.

Listing 16. The Entry class


class Entry:
    def __init__(self, service=None, username=None, title=None, 
link=None, content=None, timestamp=None):
        self.service = service
        self.username = username
        self.title = title
        self.link = link
        self.content = content
        self.timestamp = timestamp
    def to_dict(self):
        return self.__dict__
    def to_xml(self):
        str = """<entry>
                    <service>%s</service>
                    <username>%s</username>
                    <title>%s</title>
                    <link>%s</link>
                    <content><![CDATA[%s]]></content>
                    <timestamp>%s</timestamp>
                </entry>"""
        return str % (self.service, self.username, self.title, self.link, self.content, 
self.timestamp)

As you can see, the to_xml() method simply uses a string template and string substitution to create an XML node. Going back to Listing 14, after we create the XML document as a string, we set the response header for the content type and send the XML string back to requester. That is all we have to do, and we have created a Web service that other mashups can now use.

Summary

This concludes Part 3 of the "Creating mashups on the Google App Engine using Eclipse" series on the Google App Engine. In this article, we added subscriptions and a UI for creating them. We modified the existing application to make use of subscriptions, and we created REST-style Web services to allow other mashups to build from aggroGator. There are a lot more things we can do from here. We could add comments to the Entry class and a UI to allow users to comment on entries. We could provide a subscription view and a personal view. We could extend our Web service so it could allow users to add to their feeds directly. All of these things are made easier, courtesy of the Google App Engine and using tools like Eclipse and PyDev in conjunction with the Google App Engine.

Download

Description	Name	Size	Download method
Sample code	os-eclipse-mashup-google-pt3.zip	238KB	HTTP

Information about download methods

Resources

Learn

To get started in this series, read "Creating mashups on the Google App Engine using Eclipse, Part 1: Creating the application."
Be sure to read "Creating mashups on the Google App Engine using Eclipse, Part 2: Building the Ajax mashup."
Read "Charming Python: Python elegance and warts, Part 1" for information about the latest goodies in Python.
Read more of the "Charming Python" articles on developerWorks.
The SDK uses the Web app framework that is very similar to Django. You can actually use Django, so you might want to learn about Django in "Python Web frameworks, Part 1: Develop for the Web with Django and Python."
Read "Get started with open source CMS, Part 6: Build a Python WebDAV client for Jakarta Slide" to read more PyDev in action.
Check out Bigtable: A Distributed Storage System for Structured Data to read all about Google's Bigtable in this research paper.
With a dynamic language like Python, it is always good to have the official Python documentation handy.
"Develop Ajax applications like the pros, Part 1" and Part 2 to learn about Prototype and script.aculo.us.
The App Engine's Memcache API is inspired by memcached. Read "Make PHP apps fast, faster, and fastest, Part 3" to see how this is commonly used to improve performance on PHP.
Doing Web development with Eclipse? You might want to read "Discover the Ajax Toolkit Framework for Eclipse."
Interested in what's happening in the Eclipse community? Check out PlanetEclipse.
Check out the available Eclipse plug-ins at Eclipse Plug-in Central.
Visit script.aculo.us for information about its JavaScript libraries.
To learn about the Prototype Framework, visit PrototypeJS.org.
Check out EclipseLive for webinars featuring various Eclipse technologies.
Check out the "Recommended Eclipse reading list."
Browse all the Eclipse content on developerWorks.
New to Eclipse? Read the developerWorks article "Get started with Eclipse Platform" to learn its origin and architecture, and how to extend Eclipse with plug-ins.
Expand your Eclipse skills by checking out IBM developerWorks' Eclipse project resources.
To listen to interesting interviews and discussions for software developers, check out developerWorks podcasts.
Stay current with developerWorks' Technical events and webcasts.
Watch and learn about IBM and open source technologies and product functions with the no-cost developerWorks On demand demos.
Check out upcoming conferences, trade shows, webcasts, and other Events around the world that are of interest to IBM open source developers.
Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.

Get products and technologies

Download the App Engine SDK and read official documentation from the Google App Engine site.
The application created in this article used Mark Pilgrim's Universal Feed Parser. This awesome library can parse RSS, Atom, you name it.
DjangoProject.com is the home page for the Django framework.
This article uses Eclipse Classic V3.3.2.
The PyDev plugin is available from http://pydev.sourceforge.net/updates/. It can be installed from within Eclipse using this update site.
Check out the latest Eclipse technology downloads at IBM alphaWorks.
Download Eclipse Platform and other projects from the Eclipse Foundation.
Download IBM product evaluation versions, and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
Innovate your next open source development project with IBM trial software, available for download or on DVD.

Discuss

The Eclipse Platform newsgroups should be your first stop to discuss questions regarding Eclipse. (Selecting this will launch your default Usenet news reader application and open eclipse.platform.)
The Eclipse newsgroups has many resources for people interested in using and extending Eclipse.
Participate in developerWorks blogs and get involved in the developerWorks community.

About the author

Michael Galpin has been developing Java software professionally since 1998. He currently works for eBay. He holds a degree in mathematics from the California Institute of Technology.

Comments (Undergoing maintenance)

Trademarks | My developerWorks terms and conditions

Close [x]

Help: Update or add to My dW interests

What's this?

This little timesaver lets you update your My developerWorks profile with just one click! The general subject of this content (AIX and UNIX, Information Management, Lotus, Rational, Tivoli, WebSphere, Java, Linux, Open source, SOA and Web services, Web development, or XML) will be added to the interests section of your profile, if it's not there already. You only need to be logged in to My developerWorks.

And what's the point of adding your interests to your profile? That's how you find other users with the same interests as yours, and see what they're reading and contributing to the community. Your interests also help us recommend relevant developerWorks content to you.

View your My developerWorks profile

Return from help

Close [x]

Help: Remove from My dW interests

What's this?

Removing this interest does not alter your profile, but rather removes this piece of content from a list of all content for which you've indicated interest. In a future enhancement to My developerWorks, you'll be able to see a record of that content.

View your My developerWorks profile

Return from help

static.content.url=http://www.ibm.com/developerworks/js/artrating/

SITE_ID=1

Zone=Open source

ArticleID=331171

ArticleTitle=Creating mashups on the Google App Engine using Eclipse, Part 3: Using RESTful Web services

publish-date=08192008

author1-email=mike.sr@gmail.com

author1-email-cc=