 | Level: Intermediate Michael Galpin (mike.sr@gmail.com), Software architect, eBay
19 Aug 2008 Social networks are making it easier to take data and mash it up to create
innovative Web applications. You still, however, must deal with all the usual issues
with creating a scalable Web application. Now the Google App Engine (GAE) makes that
easier for you. With it, you can forget all about managing pools of application servers,
and, instead, you can concentrate on creating a great mashup. In this article, the
last of a three-part "Creating
mashups on the Google App Engine using Eclipse" series, we will take the
application built in the first two parts and further enhance it. We will add the
ability to view other users of the app and subscribe to their aggregate feeds. We
will then complete the mashup circle by exposing the app as a Web service that can
be used by other mashups.
About this series
In this series, we look at how to get started with the Google App Engine (GAE). In
Part
1, we look at how to get a development environment set up so you can start
creating an application that will run on the GAE. We also saw how to use Eclipse to
make developing and debugging your application easier. In Part
2, we enhance the application by adding some Ajax features. We also saw how to
monitor the application once it was deployed to the GAE. Here in Part 3, we will give
back to the ecosystem by creating RESTful Web services to our application, so other folks can use it to create their own mashups.
The GAE is a platform for creating Web applications. The biggest prerequisite for it is
knowledge of Python, as this is the programming language used on it (currently, Python
V2.5.2). For this series, it would be helpful to have some typical Web development
skills (e.g., knowledge of HTML, JavaScript, and CSS). To develop for
the App Engine, you will need to download the App Engine SDK (see Resources). In this series, we also use Eclipse
V3.3.2 to aid in our GAE development (see Resources).
And you'll need the PyDev plug-in to turn Eclipse into a Python IDE.
Subscriptions
So far, our application, aggroGator, allows a user to mash up several popular Web
services and create an aggregate feed of those services. Now, to make things a little
more interesting, we want to allow users to subscribe to other users' feeds (where each
user's feed is quite possibly an aggregate of feeds itself.) For example, let's say we
want to set up an account to subscribe to ourself on Twitter, last.fm, and del.icio.us,
so friends can then subscribe to our aggroGator feed to see all of the activity on
those services. To handle this, we need to once again revisit our data model.
Modeling
To enable subscriptions, we need to allow one user (account) to subscribe to a list of
other accounts. One approach we could take would be to add a list of users to each
account. Each user adds a subscription we could add to this list. The code for this would look something like Listing 1.
Listing 1. Account model with user list
class Account(db.Model):
user = db.UserProperty(required=True)
subscriptions = db.ListProperty(Account)
|
There are some advantages to this approach. When we retrieve an account, we get all of
the other accounts that it is subscribed to. This is a common tactic to use with
nonrelational data stores like the GAE's Bigtable: Keep all relevant data together and
do not worry about things like normalization. However, there is a disadvantage to this
approach. What if we want to show who all is subscribed to a particular user. The only
way to do this would be to retrieve all Account models, look
at all the subscriptions, and see if the given user is in the list. Alternatively, we
could keep two lists in each Account model —
one for the subscriptions and one for the subscribers. Instead of taking this approach, we will use a more
traditional many-to-many model, as shown in Listing 2.
Listing 2. Subscribe model
class Subscribe(db.Model):
subscriber = db.ReferenceProperty(Account, required=True,
collection_name='subscriptions')
subscribee = db.ReferenceProperty(Account, required=True,
collection_name='subscribers')
|
As you can see, this is similar to a join table you would use with a relational
database. Just because the GAE uses a nonrelational data store (Bigtable) does not mean
that you cannot leverage techniques you have used with relational databases. Now that
we have the data model in place, let's take a top-down look at how these many-to-many
relationships will be created from the end user's perspective.
Subscription management
Our application can store subscriptions, so we just need some way for users to create
subscriptions. To do this, we will create a page for users to add subscriptions (see
Listing 3).
Listing 3. Accounts page
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/
xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>Aggrogator Accounts</title>
<link rel="stylesheet" href="/css/aggrogator.css" type="text/css" />
<script type="text/javascript" src="/js/prototype.js"></script>
<script type="text/javascript" src="/js/builder.js"></script>
<script type="text/javascript" src="/js/effects.js"></script>
<script type="text/javascript" src="/js/aggrogator.js"></script>
</head>
<body>
<img id="spinner" alt="spinner" src="/gfx/spinner.gif" style="display: none;
position: fixed;" />
<div id="logout">
{{ account.user.nickname }}
<a href="{{ logout_url }}">Logout</a>
</div>
<div class="clearboth"></div>
<ol>
{% for acc in all_accounts %}
<li>
<a href="" onclick="subscribe('{{ acc.user.email }}'); return false;">
{{ acc.user.email }}</a>
</li>
{% endfor %}
</ol>
</body>
</html>
|
As you can see, this page simply displays all of the accounts in the system. The user
then picks an account to subscribe by clicking on it. You could imagine a more
sophisticated interface. For example, this would become unwieldy with a large number of
users, so a search-based system would be better. Or perhaps a system allowing the user
to import his address book or use APIs from something like OpenSocial to find existing
friends who are already part of the application. The template above needs a list of
users, so let's take a quick look at the controller that creates the model for the page
(see Listing 4).
Listing 4. Accounts page controller
#Accounts Module
class MainPage(webapp.RequestHandler):
def get(self):
# get the current user
user = users.get_current_user()
# is user an admin?
admin = users.is_current_user_admin();
# create user account if haven't already
account = aggrogator.DB.getAccount(user)
if account is None:
account = aggrogator.Account(user=user)
account.put()
# create logout url
logout_url = users.create_logout_url(self.request.uri)
all_accounts = aggrogator.Account.all()
template_values = {
'account': account,
'admin': admin,
'logout_url': logout_url,
'all_accounts': all_accounts,
}
path = os.path.join(os.path.dirname(__file__), 'accounts.html')
self.response.out.write(template.render(path, template_values))
|
The controller gets all the data ready for display on the accounts page. Back on the
accounts page in Listing 3, we see that there is JavaScript called when an account is
clicked on.
Listing 5. Subscription JavaScript
function subscribe(email) {
new Ajax.Request("/accounts/subscribe", {
method: "post",
parameters: {'email': email},
onSuccess: alert('subscribed to ' + email)
});
}
|
This JavaScript once again uses the Prototype library to make an Ajax request to the
server. We call the URL /accounts/subscribe. Where is that URL mapped to? The code that
creates the mapping is in the main function of the new accounts module, as shown below.
Listing 6. URL mappings for accounts module
def main():
app = webapp.WSGIApplication([
('/accounts/', MainPage),
('/accounts/subscribe', Subscribe),
], debug=True)
util.run_wsgi_app(app)
if __name__ == '__main__':
main()
|
As you can see from the main function, a call to /accounts/subscribe is handled by a
Subscribe controller class. That class is shown in Listing 7.
Listing 7. Subscribe controller class
class Subscribe(webapp.RequestHandler):
def post(self):
# get the current user
user = users.get_current_user()
email = self.request.get('email')
aggrogator.DB.create_subscription(user, email)
|
This controller is simple. It gets the current user (the subscriber) and the e-mail
address of the subscription being added. It then calls a new method in the DB utility
class we have used previously. That class handles all our Bigtable-related calls. The
new create_subscription function is shown below.
Listing 8. DB function for create_subscription
class DB:
@staticmethod
def create_subscription(user, email):
subscriber = DB.getAccount(user)
subscribee = DB.getAccountForEmail(email)
subscription = Subscribe.gql("WHERE subscriber = :1 AND subscribee = :2",
subscriber, subscribee).get()
if subscription is None:
Subscribe(subscriber=subscriber, subscribee=subscribee).put()
@classmethod
def getAccountForEmail(cls, email):
user = users.User(email)
return cls.getAccount(user)
|
This function first looks up the Account models for the user
and subscription e-mail. For the latter, it uses the new getAccountForEmail function. This makes use of the GAE's user's API
to look up the User object based on the e-mail, then
querying Bigtable for the account. Once we have both accounts, we check to see if the
subscription already exists. If it does not, we create a new one.
Of course, now that we have subscriptions, we want to make use of them in the main
application. Instead of just showing the current user's services, we want to show the
aggregate feed (the user's services and the services from his subscriptions, as
well). To do this, we make a small change in the GetUserServices controller in the main module developed in previous
articles. This is shown below.
Listing 9. Modified GetUserServices controller
class GetUserServices(webapp.RequestHandler):
def get(self):
user = users.get_current_user()
# get the user's services from the cache
#userServices = aggrogator.Cache.getUserServices(user)
userServices = aggrogator.Aggrogator.get_services(user.email())
stats = memcache.get_stats()
self.response.headers['content-type'] = 'application/json'
self.response.out.write(simplejson.dumps({'stats': stats, 'userServices':
userServices}))
|
All we did here was call a new library class: the appropriately named aggrogator class, to get the aggregate services instead of just the
user's. This library code is shown below.
Listing 10. The aggrogator Library: Retrieve aggregate services
class aggrogator:
@staticmethod
def get_services(username):
accounts = []
primary = DB.getAccountForEmail(username)
accounts.append(primary)
for subscription in primary.subscriptions:
accounts.append(subscription.subscribee)
services = []
for account in accounts:
services.extend(Cache.getUserServices(account.user))
return services
|
Here is where we can again see our new Subscribe model in
action. In the code, we get the account for the username (by using the getAccountForEmail function we saw earlier), then call its
subscriptions property. In this case, we only use this to get all of the services from
the cache. Later, we will see these services used to create the aggregate feed.
We are almost ready to test the new accounts page. We have to make one last change: We
need to configure our application to send certain URL requests to the new accounts
module. To do this, we edit the app.yaml file and add a new section.
Listing 11. Addition to app.yaml
- url: /accounts/.*
script: accounts.py
login: required
|
This is just a new section of the file. It maps any request that has /accounts/ to the
accounts module. This should appear before the catch-all handler used previously (url:
/.*) so that it takes precedence. Now we can test the application just as before, using
Eclipse and PyDev, and by going to http://localhost:8080/acounts/. Make sure you create
multiple accounts so your testing can be interesting.
The aggroGator Web service
Social Web services make it possible to create interesting applications like aggroGator
very easily. The GAE allows us to create such mashups that are also very scalable. So
of course it makes sense to create an API/Web service around our mashup so others can
use it to create their own interesting mashups. This turns out to be quite easy, as well.
For our Web service, we will start by making it a read-only service. The service will
simply provide the aggregate feed for a user (i.e., the same thing one would see in the
aggroGator UI). We will use a simple REST-style URL for this, such as
/api?username=my@email.address. This time, we will start bottom-up. To handle such a
URL, we once again add a section to our app.yaml file.
Listing 12. Addition to app.yaml
- url: /api
script: main.py
|
Notice that we are still sending the /api requests to the main module. Why did we need
a new mapping in app.yaml? We do not want to require authentication for the aggroGator
API. That is the only reason we need the new rule in app.yaml. Since we leverage the
main module, it needs to be modified.
Listing 13. New touting rule for main module
def main():
app = webapp.WSGIApplication([
('/', MainPage),
('/addService', AddService),
('/getEntries', GetEntries),
('/api', AggroWebService),
('/getUserServices', GetUserServices),
], debug=True)
util.run_wsgi_app(app)
if __name__ == '__main__':
main()
|
All we have done to this function is add one entry to the list of mappings. We are
mapping /api to the AggroWebService controller class. That class is shown below.
Listing 14. The AggroWebService controller class
class AggroWebService(webapp.RequestHandler):
def get(self):
self.response.headers['content-type'] = 'text/xml'
username = self.request.get('username')
entries = aggrogator.Aggrogator.get_feed(username)
str = u"""<?xml version="1.0" encoding="utf-8"?><entries>"""
for entry in entries:
str += entry.to_xml()
str += "</entries>"
self.response.out.write(str)
|
The service starts off by retrieving the username request
parameter. It then uses the aggroGator library we saw earlier, but uses a different
method, get_feed, to get the aggregate entries. The code for that library function is shown below.
Listing 15. aggroGator get_feed
class Aggrogator:
@staticmethod
def get_feed(username):
services = Aggrogator.get_services(username)
entries = []
for svc_tuple in set((svc['service'], svc['username']) for svc in services):
entries.extend(Cache.getEntries(*svc_tuple))
entries.sort(key=operator.attrgetter('timestamp'), reverse=True)
return entries
|
This library function uses the get_services function we saw in Listing 10 to retrieve the aggregate
services. It then iterates over the services. The code uses a set to make sure that the
services are unique (i.e., if a user had subscribed to two other users who each used
the same service). Because we used a set, we have to use tuple as we can only use an
immutable object. Finally, we sort all of the entries by their timestamp in descending
order (latest entries listed first).
Going back to Listing 14, once we have the list of entries, we then use some simple
string concatenation to create an XML document. We use a to_xml() method on each Entry instance.
This is a new method, shown below.
Listing 16. The Entry class
class Entry:
def __init__(self, service=None, username=None, title=None,
link=None, content=None, timestamp=None):
self.service = service
self.username = username
self.title = title
self.link = link
self.content = content
self.timestamp = timestamp
def to_dict(self):
return self.__dict__
def to_xml(self):
str = """<entry>
<service>%s</service>
<username>%s</username>
<title>%s</title>
<link>%s</link>
<content><![CDATA[%s]]></content>
<timestamp>%s</timestamp>
</entry>"""
return str % (self.service, self.username, self.title, self.link, self.content,
self.timestamp)
|
As you can see, the to_xml() method simply uses a string
template and string substitution to create an XML node. Going back to Listing 14, after
we create the XML document as a string, we set the response header for the content type
and send the XML string back to requester. That is all we have to do, and we have
created a Web service that other mashups can now use.
Summary
This concludes Part 3 of the "Creating
mashups on the Google App Engine using Eclipse" series on the Google App Engine. In
this article, we added subscriptions and a UI for creating them. We modified the
existing application to make use of subscriptions, and we created REST-style Web
services to allow other mashups to build from aggroGator. There are a lot more things
we can do from here. We could add comments to the Entry
class and a UI to allow users to comment on entries. We could provide a subscription
view and a personal view. We could extend our Web service so it could allow users to add
to their feeds directly. All of these things are made easier, courtesy of the Google App
Engine and using tools like Eclipse and PyDev in conjunction with the Google App Engine.
Download | Description | Name | Size | Download method |
|---|
| Sample code | os-eclipse-mashup-google-pt3.zip | 238KB | HTTP |
|---|
Resources Learn
-
To get started in this series, read "Creating
mashups on the Google App Engine using Eclipse, Part 1: Creating the application."
-
Be sure to read "Creating
mashups on the Google App Engine using Eclipse, Part 2: Building the Ajax mashup."
-
Read "Charming
Python: Python elegance and warts, Part 1" for information about the latest goodies in Python.
-
Read more of the "Charming
Python" articles on developerWorks.
-
The SDK uses the Web app framework that is very similar to Django. You can actually use
Django, so you might want to learn about Django in "Python Web frameworks,
Part 1: Develop for the Web with Django and Python."
-
Read "Get started with open source CMS, Part 6: Build a
Python WebDAV client for Jakarta Slide" to read more PyDev in action.
-
Check out Bigtable: A Distributed Storage
System for Structured Data to read all about Google's Bigtable in this research paper.
-
With a dynamic language like Python, it is always good to have the official Python documentation handy.
-
"Develop Ajax
applications like the pros, Part 1" and Part 2 to learn about Prototype and script.aculo.us.
-
The App Engine's Memcache API is inspired by memcached. Read "Make
PHP apps fast, faster, and fastest, Part 3" to see how this is commonly used to improve performance on PHP.
-
Doing Web development with Eclipse? You might want to read "Discover the Ajax
Toolkit Framework for Eclipse."
-
Interested in what's happening in the Eclipse community? Check out PlanetEclipse.
-
Check out the available Eclipse plug-ins at Eclipse Plug-in Central.
-
Visit script.aculo.us for information about its JavaScript libraries.
-
To learn about the Prototype Framework, visit PrototypeJS.org.
-
Check out EclipseLive for webinars featuring various Eclipse technologies.
-
Check out the "Recommended Eclipse reading list."
-
Browse all the Eclipse content on developerWorks.
-
New to Eclipse? Read the developerWorks article "Get started with Eclipse Platform" to learn its origin and architecture, and how to extend Eclipse with plug-ins.
-
Expand your Eclipse skills by checking out IBM developerWorks' Eclipse project resources.
-
To listen to interesting interviews and discussions for software developers, check out developerWorks podcasts.
-
Stay current with developerWorks' Technical events and webcasts.
-
Watch and learn about IBM and open source technologies and product functions with the no-cost developerWorks On demand demos.
-
Check out upcoming conferences, trade shows, webcasts, and other Events around the world that are of interest to IBM open source developers.
-
Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.
Get products and technologies
Discuss
-
The Eclipse Platform newsgroups should be your first stop to discuss questions regarding Eclipse. (Selecting this will launch your default Usenet news reader application and open eclipse.platform.)
-
The Eclipse newsgroups has many resources for people interested in using and extending Eclipse.
-
Participate in developerWorks blogs and get involved in the developerWorks community.
About the author | 
| | Michael Galpin has been developing Java software professionally since 1998. He currently works for eBay. He holds a degree in mathematics from the California Institute of Technology. |
Rate this page
| |
