Creating mashups on the Google App Engine using Eclipse, Part 3: Using RESTful Web services

Social networks are making it easier to take data and mash it up to create innovative Web applications. You still, however, must deal with all the usual issues with creating a scalable Web application. Now the Google App Engine (GAE) makes that easier for you. With it, you can forget all about managing pools of application servers, and, instead, you can concentrate on creating a great mashup. In this article, the last of a three-part "Creating mashups on the Google App Engine using Eclipse" series, we will take the application built in the first two parts and further enhance it. We will add the ability to view other users of the app and subscribe to their aggregate feeds. We will then complete the mashup circle by exposing the app as a Web service that can be used by other mashups.

Share:

Michael Galpin (mike.sr@gmail.com), Software architect, eBay

Michael Galpin's photoMichael Galpin has been developing Java software professionally since 1998. He currently works for eBay. He holds a degree in mathematics from the California Institute of Technology.



19 August 2008

About this series

In this series, we look at how to get started with the Google App Engine (GAE). In Part 1, we look at how to get a development environment set up so you can start creating an application that will run on the GAE. We also saw how to use Eclipse to make developing and debugging your application easier. In Part 2, we enhance the application by adding some Ajax features. We also saw how to monitor the application once it was deployed to the GAE. Here in Part 3, we will give back to the ecosystem by creating RESTful Web services to our application, so other folks can use it to create their own mashups.

The GAE is a platform for creating Web applications. The biggest prerequisite for it is knowledge of Python, as this is the programming language used on it (currently, Python V2.5.2). For this series, it would be helpful to have some typical Web development skills (e.g., knowledge of HTML, JavaScript, and CSS). To develop for the App Engine, you will need to download the App Engine SDK (see Resources). In this series, we also use Eclipse V3.3.2 to aid in our GAE development (see Resources). And you'll need the PyDev plug-in to turn Eclipse into a Python IDE.


Subscriptions

So far, our application, aggroGator, allows a user to mash up several popular Web services and create an aggregate feed of those services. Now, to make things a little more interesting, we want to allow users to subscribe to other users' feeds (where each user's feed is quite possibly an aggregate of feeds itself.) For example, let's say we want to set up an account to subscribe to ourself on Twitter, last.fm, and del.icio.us, so friends can then subscribe to our aggroGator feed to see all of the activity on those services. To handle this, we need to once again revisit our data model.

Modeling

To enable subscriptions, we need to allow one user (account) to subscribe to a list of other accounts. One approach we could take would be to add a list of users to each account. Each user adds a subscription we could add to this list. The code for this would look something like Listing 1.

Listing 1. Account model with user list
class Account(db.Model):
    user = db.UserProperty(required=True)
    subscriptions = db.ListProperty(Account)

There are some advantages to this approach. When we retrieve an account, we get all of the other accounts that it is subscribed to. This is a common tactic to use with nonrelational data stores like the GAE's Bigtable: Keep all relevant data together and do not worry about things like normalization. However, there is a disadvantage to this approach. What if we want to show who all is subscribed to a particular user. The only way to do this would be to retrieve all Account models, look at all the subscriptions, and see if the given user is in the list. Alternatively, we could keep two lists in each Account model — one for the subscriptions and one for the subscribers. Instead of taking this approach, we will use a more traditional many-to-many model, as shown in Listing 2.

Listing 2. Subscribe model
class Subscribe(db.Model):
    subscriber = db.ReferenceProperty(Account, required=True, 
collection_name='subscriptions')
    subscribee = db.ReferenceProperty(Account, required=True, 
collection_name='subscribers')

As you can see, this is similar to a join table you would use with a relational database. Just because the GAE uses a nonrelational data store (Bigtable) does not mean that you cannot leverage techniques you have used with relational databases. Now that we have the data model in place, let's take a top-down look at how these many-to-many relationships will be created from the end user's perspective.

Subscription management

Our application can store subscriptions, so we just need some way for users to create subscriptions. To do this, we will create a page for users to add subscriptions (see Listing 3).

Listing 3. Accounts page
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/
    xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8" />
    <title>Aggrogator Accounts</title>
    <link rel="stylesheet" href="/css/aggrogator.css" type="text/css" />
    <script type="text/javascript" src="/js/prototype.js"></script>
    <script type="text/javascript" src="/js/builder.js"></script>
    <script type="text/javascript" src="/js/effects.js"></script>
    <script type="text/javascript" src="/js/aggrogator.js"></script>
  </head>
  <body>

    <img id="spinner" alt="spinner" src="/gfx/spinner.gif" style="display: none; 
position: fixed;" />

    <div id="logout">
      {{ account.user.nickname }}
      <a href="{{ logout_url }}">Logout</a>
    </div>

    <div class="clearboth"></div>

    <ol>
      {% for acc in all_accounts %}
        <li>
          <a href="" onclick="subscribe('{{ acc.user.email }}'); return false;">
{{ acc.user.email }}</a>
        </li>
      {% endfor %}
    </ol>

  </body>
</html>

As you can see, this page simply displays all of the accounts in the system. The user then picks an account to subscribe by clicking on it. You could imagine a more sophisticated interface. For example, this would become unwieldy with a large number of users, so a search-based system would be better. Or perhaps a system allowing the user to import his address book or use APIs from something like OpenSocial to find existing friends who are already part of the application. The template above needs a list of users, so let's take a quick look at the controller that creates the model for the page (see Listing 4).

Listing 4. Accounts page controller
#Accounts Module
class MainPage(webapp.RequestHandler):
    def get(self):
        # get the current user
        user = users.get_current_user()

        # is user an admin?
        admin = users.is_current_user_admin();

        # create user account if haven't already
        account = aggrogator.DB.getAccount(user)
        if account is None:
            account = aggrogator.Account(user=user)
            account.put()

        # create logout url
        logout_url = users.create_logout_url(self.request.uri)

        all_accounts = aggrogator.Account.all()

        template_values = {
            'account': account,
            'admin': admin,
            'logout_url': logout_url,
            'all_accounts': all_accounts,
            }
        path = os.path.join(os.path.dirname(__file__), 'accounts.html')
        self.response.out.write(template.render(path, template_values))

The controller gets all the data ready for display on the accounts page. Back on the accounts page in Listing 3, we see that there is JavaScript called when an account is clicked on.

Listing 5. Subscription JavaScript
function subscribe(email) {
  new Ajax.Request("/accounts/subscribe", {
    method: "post",
    parameters: {'email': email},
    onSuccess: alert('subscribed to ' + email)
  });
}

This JavaScript once again uses the Prototype library to make an Ajax request to the server. We call the URL /accounts/subscribe. Where is that URL mapped to? The code that creates the mapping is in the main function of the new accounts module, as shown below.

Listing 6. URL mappings for accounts module
def main():
    app = webapp.WSGIApplication([
        ('/accounts/', MainPage),
        ('/accounts/subscribe', Subscribe),
        ], debug=True)
    util.run_wsgi_app(app)

if __name__ == '__main__':
    main()

As you can see from the main function, a call to /accounts/subscribe is handled by a Subscribe controller class. That class is shown in Listing 7.

Listing 7. Subscribe controller class
class Subscribe(webapp.RequestHandler):
    def post(self):
        # get the current user
        user = users.get_current_user()
        email = self.request.get('email')

        aggrogator.DB.create_subscription(user, email)

This controller is simple. It gets the current user (the subscriber) and the e-mail address of the subscription being added. It then calls a new method in the DB utility class we have used previously. That class handles all our Bigtable-related calls. The new create_subscription function is shown below.

Listing 8. DB function for create_subscription
class DB:
    @staticmethod
    def create_subscription(user, email):
        subscriber = DB.getAccount(user)
        subscribee = DB.getAccountForEmail(email)
        subscription = Subscribe.gql("WHERE subscriber = :1 AND subscribee = :2", 
subscriber, subscribee).get()
        if subscription is None:
            Subscribe(subscriber=subscriber, subscribee=subscribee).put()

    @classmethod
    def getAccountForEmail(cls, email):
        user = users.User(email)
        return cls.getAccount(user)

This function first looks up the Account models for the user and subscription e-mail. For the latter, it uses the new getAccountForEmail function. This makes use of the GAE's user's API to look up the User object based on the e-mail, then querying Bigtable for the account. Once we have both accounts, we check to see if the subscription already exists. If it does not, we create a new one.

Of course, now that we have subscriptions, we want to make use of them in the main application. Instead of just showing the current user's services, we want to show the aggregate feed (the user's services and the services from his subscriptions, as well). To do this, we make a small change in the GetUserServices controller in the main module developed in previous articles. This is shown below.

Listing 9. Modified GetUserServices controller
class GetUserServices(webapp.RequestHandler):
    def get(self):
        user = users.get_current_user()

        # get the user's services from the cache
        #userServices = aggrogator.Cache.getUserServices(user)
        userServices = aggrogator.Aggrogator.get_services(user.email())

        stats = memcache.get_stats()
        self.response.headers['content-type']  = 'application/json'
        self.response.out.write(simplejson.dumps({'stats': stats, 'userServices': 
userServices}))

All we did here was call a new library class: the appropriately named aggrogator class, to get the aggregate services instead of just the user's. This library code is shown below.

Listing 10. The aggrogator Library: Retrieve aggregate services
class aggrogator:
    @staticmethod
    def get_services(username):
        accounts = []
        
        primary = DB.getAccountForEmail(username)
        accounts.append(primary)

        for subscription in primary.subscriptions:
            accounts.append(subscription.subscribee)
        services = []
        for account in accounts:
            services.extend(Cache.getUserServices(account.user))   
        return services

Here is where we can again see our new Subscribe model in action. In the code, we get the account for the username (by using the getAccountForEmail function we saw earlier), then call its subscriptions property. In this case, we only use this to get all of the services from the cache. Later, we will see these services used to create the aggregate feed.

We are almost ready to test the new accounts page. We have to make one last change: We need to configure our application to send certain URL requests to the new accounts module. To do this, we edit the app.yaml file and add a new section.

Listing 11. Addition to app.yaml
- url: /accounts/.*
  script: accounts.py
  login: required

This is just a new section of the file. It maps any request that has /accounts/ to the accounts module. This should appear before the catch-all handler used previously (url: /.*) so that it takes precedence. Now we can test the application just as before, using Eclipse and PyDev, and by going to http://localhost:8080/acounts/. Make sure you create multiple accounts so your testing can be interesting.


The aggroGator Web service

Social Web services make it possible to create interesting applications like aggroGator very easily. The GAE allows us to create such mashups that are also very scalable. So of course it makes sense to create an API/Web service around our mashup so others can use it to create their own interesting mashups. This turns out to be quite easy, as well.

For our Web service, we will start by making it a read-only service. The service will simply provide the aggregate feed for a user (i.e., the same thing one would see in the aggroGator UI). We will use a simple REST-style URL for this, such as /api?username=my@email.address. This time, we will start bottom-up. To handle such a URL, we once again add a section to our app.yaml file.

Listing 12. Addition to app.yaml
- url: /api
  script: main.py

Notice that we are still sending the /api requests to the main module. Why did we need a new mapping in app.yaml? We do not want to require authentication for the aggroGator API. That is the only reason we need the new rule in app.yaml. Since we leverage the main module, it needs to be modified.

Listing 13. New touting rule for main module
def main():
    app = webapp.WSGIApplication([
        ('/', MainPage),
        ('/addService', AddService),
        ('/getEntries', GetEntries),
        ('/api', AggroWebService),
        ('/getUserServices', GetUserServices),
        ], debug=True)
    util.run_wsgi_app(app)

if __name__ == '__main__':
    main()

All we have done to this function is add one entry to the list of mappings. We are mapping /api to the AggroWebService controller class. That class is shown below.

Listing 14. The AggroWebService controller class
class AggroWebService(webapp.RequestHandler):
    def get(self):
        self.response.headers['content-type'] = 'text/xml'
        username = self.request.get('username')
        entries = aggrogator.Aggrogator.get_feed(username)
        str = u"""<?xml version="1.0" encoding="utf-8"?><entries>"""
        for entry in entries:
            str += entry.to_xml()
        str += "</entries>"
        self.response.out.write(str)

The service starts off by retrieving the username request parameter. It then uses the aggroGator library we saw earlier, but uses a different method, get_feed, to get the aggregate entries. The code for that library function is shown below.

Listing 15. aggroGator get_feed
class Aggrogator:
    @staticmethod
    def get_feed(username):
        services = Aggrogator.get_services(username)
        entries = []
        for svc_tuple in set((svc['service'], svc['username']) for svc in services):
            entries.extend(Cache.getEntries(*svc_tuple))
        entries.sort(key=operator.attrgetter('timestamp'), reverse=True)
        return entries

This library function uses the get_services function we saw in Listing 10 to retrieve the aggregate services. It then iterates over the services. The code uses a set to make sure that the services are unique (i.e., if a user had subscribed to two other users who each used the same service). Because we used a set, we have to use tuple as we can only use an immutable object. Finally, we sort all of the entries by their timestamp in descending order (latest entries listed first).

Going back to Listing 14, once we have the list of entries, we then use some simple string concatenation to create an XML document. We use a to_xml() method on each Entry instance. This is a new method, shown below.

Listing 16. The Entry class
class Entry:
    def __init__(self, service=None, username=None, title=None, 
link=None, content=None, timestamp=None):
        self.service = service
        self.username = username
        self.title = title
        self.link = link
        self.content = content
        self.timestamp = timestamp
    def to_dict(self):
        return self.__dict__
    def to_xml(self):
        str = """<entry>
                    <service>%s</service>
                    <username>%s</username>
                    <title>%s</title>
                    <link>%s</link>
                    <content><![CDATA[%s]]></content>
                    <timestamp>%s</timestamp>
                </entry>"""
        return str % (self.service, self.username, self.title, self.link, self.content, 
self.timestamp)

As you can see, the to_xml() method simply uses a string template and string substitution to create an XML node. Going back to Listing 14, after we create the XML document as a string, we set the response header for the content type and send the XML string back to requester. That is all we have to do, and we have created a Web service that other mashups can now use.


Summary

This concludes Part 3 of the "Creating mashups on the Google App Engine using Eclipse" series on the Google App Engine. In this article, we added subscriptions and a UI for creating them. We modified the existing application to make use of subscriptions, and we created REST-style Web services to allow other mashups to build from aggroGator. There are a lot more things we can do from here. We could add comments to the Entry class and a UI to allow users to comment on entries. We could provide a subscription view and a personal view. We could extend our Web service so it could allow users to add to their feeds directly. All of these things are made easier, courtesy of the Google App Engine and using tools like Eclipse and PyDev in conjunction with the Google App Engine.


Download

DescriptionNameSize
Sample codeos-eclipse-mashup-google-pt3.zip238KB

Resources

Learn

Get products and technologies

Discuss

  • The Eclipse Platform newsgroups should be your first stop to discuss questions regarding Eclipse. (Selecting this will launch your default Usenet news reader application and open eclipse.platform.)
  • The Eclipse newsgroups has many resources for people interested in using and extending Eclipse.
  • Participate in developerWorks blogs and get involved in the developerWorks community.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Open source on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Open source
ArticleID=331171
ArticleTitle=Creating mashups on the Google App Engine using Eclipse, Part 3: Using RESTful Web services
publish-date=08192008