Real Web 2.0: Bookmarks? Tagging? Delicious!

Learn how real-world developers and users gain value from a classic Web 2.0 site

Uche Ogbuji (uche@ogbuji.net), Principal Consultant, Fourthought, Inc.

Uche Ogbuji is a consultant and founder of Uli, LLC, a software vendor and consultancy specializing in solutions integrating XML and Web technologies with enterprise data processing systems. Mr. Ogbuji co-develops 4Suite, an open source platform for XML, RDF, and knowledge-management applications. He is a computer engineer and writer born in Nigeria, now living and working in Boulder, Colorado, USA. You can find more about Mr. Ogbuji at his Weblog, Copia.

Summary: In this article, you'll learn how to work with del.icio.us, one of the classic Web 2.0 sites, using Web XML feeds and JSON, in Python and ECMAScript. When you think of Web 2.0 technology, you might think of the latest Ajax tricks, but that is just a small part of the picture. More fundamental concerns are open data, simple APIs, and features that encourage users to form social networks. These are also what make Web 2.0 a compelling problem for Web architects. This column will look more than skin deep at important real-world Web 2.0 sites and demonstrate how Web architects can incorporate the best from the Web into their own Web sites.

View more content in this series

Tag this!

Date: 26 Oct 2006
Level: Introductory
Activity: 4180 views

When people list classic Web 2.0 sites, one that never fails to come up is del.icio.us (see Resources for a link). This site, despite its relative youth, is the best established social bookmarking site. It allows people to post links and add tags to those links. It provides Web feeds for link collections, including automatically aggregated tags such as the special popular tag, for links that have been posted by many users. It provides a simple and open API. del.icio.us also serves as a demonstration of how fuzzy a term Web 2.0 is. You might be expecting a slick, Ajax-driven Web page with Web boxes winking seductively at you. At one point, del.icio.us was nothing at all like that; it was very simple and functional. The front page has recently been dressed up quite a bit, but throughout most of its history it has had a very plain interface.

The idea of sharing browser bookmarks is not new. During the dot-com era, services such as Backflip emerged with which people could host and categorize their bookmarks online. Backflip was free of charge, but the breakthrough of sites such as del.icio.us is that the information is even more free than that: freely available for use in innumerable ways, and for sharing in innumerable patterns. Open APIs, Web feeds (such as those built on RSS and Atom), and tagging with automatic aggregation are the foundation of the more exciting developments on the Web, and these too often get lumped in with user interface tricks such as Ajax. These collaborative features are sometimes hampered by the typically limited interface of the Web, so sometimes a few fancy touches make it possible and attractive for more people to get involved with Web 2.0, and the value of such sites is often proportional to the number of participants.

The substance of an effective Web 2.0 site, and the points of interest for Web architects (as opposed to, say, Web designers), lie in how readily real developers and users can take advantage of open data features. From widgets that users can use to customize their bits of territory on a social site to mashups that developers can use to create offspring from Web 2.0 parents, there are ways to understand what leads to success for such sites, and how you can emulate such success in your own work. This column, Real Web 2.0, will cut through the hype to focus on the most valuable features of actual sites from the perspective of the Web architect. In this first installment, I'll begin with one of the ancestors of the genre, del.icio.us.

The front door

In this column, I'll refer to the main page of a Web 2.0 site as its façade. This is because, like the façade of a building, it's only the front face, and much of the actual business goes on in the less visible areas. I use del.icio.us almost every day, but I almost never bother with its façade. I use a Firefox plug-in to post interesting sites to del.icio.us. I follow items on the popular tag within my Web feed reader, and I use a simple Python script to automatically post my del.icio.us postings to my Weblog each day. This is what I mean when I say that much of the real business in Web 2.0 goes on behind the façade. Nevertheless, the front door is where you enter, so that's where I'll start.

The main page of del.icio.us tries to distill the major trends in tagging by users. It lists hot now Web pages that many people have recently bookmarked. It also lists some tags the editors deem particularly interesting. There is the mandatory Weblog by the site owners, and a recent entry celebrates the fact that the site now has 1 million registered users after just three years in business. del.icio.us is definitely an illustration of how having good architectural features for open data supports a community of contributors (developers of tools, plug-ins, mashups, and more), which brings in more users, which leads to more contributors, and so on in a virtuous circle.

Home sweet home

Registered users have a home page that is accessible using a very simple URL scheme: http://del.icio.us/<username>. A very sensible URL scheme is one of the site's enduring strengths, and leads to value beyond the façade, as I'll discuss later. Your home page includes your recently posted links and your tags. Figure 1 illustrates a portion of my own del.icio.us page.

Figure 1. del.icio.us user home page

As with most Web 2.0 pages, simple, categorizing keywords, or tags, are a basic feature of del.icio.us, and the site was a pioneer of the tag cloud concept. You can see the tag cloud on the right-hand side of Figure 1. It is a list of tag terms with some rendered in larger type or in boldface depending on their frequency of use. There has been a lot of discussion in the Web 2.0 world about the effectiveness of tag clouds as a visual metaphor, and in fact del.icio.us has de-emphasized this feature a bit lately.

You can also see some of the aggregation features of the site in the way it highlights some links that a number of other users linked to. From your user page, you can search your links and your social network (the list of people you specify whose links interest you). You can also access the Web user interface for posting and managing entries, although as I mentioned, many users will do such work in tools off the main page.

Home-grown recipes

You can only admire the façade for so long, and at some point with Web 2.0 sites, you'll seek tools that make the site's features available in ways even the site creators didn't foresee. There are many such tools for del.icio.us. The site's creators kicked things off by producing some dozen tools of their own, from badges that you can use to show posts from your network on other sites to browser toolbars. Also, well over a hundred third-party tools cover most uses you can imagine for del.icio.us -- most, but not all, which is why, when it comes down to it, the importance of Web 2.0 sites is that you can break out the compiler or interpreter and create your own features.

Web feeds: APIs on the cheap

An official API for del.icio.us uses HTTP with SSL and authentication. But if you just need read access, you have the option to use the even more official API of Web 2.0 as a whole: Web feeds. You can access Web feeds for a user (http://del.icio.us/rss/<username>), a tag (http://del.icio.us/rss/<tag>), or a combination of the two (http://del.icio.us/rss/<username>/<tag>). Then it's just a matter of parsing the Web feed to extract the desired information. Listing 1 provides an example. It's Python code that sends an e-mail containing del.icio.us feed entries from the previous day.

Listing 1. Code to e-mail del.icio.us postings from a previous day


import time
import smtplib
from email.MIMEText import MIMEText
from datetime import date, timedelta

import amara

#The base URI for all tags
TAGBASE = 'http://del.icio.us/tag/'

#Set FEEDS to customize which feeds you want to monitor
FEEDS = ['http://del.icio.us/rss/uche', 'http://del.icio.us/rss/popular']

FROM = 'del.icio.us@example.com'
TO = 'user@example.com'
SMTPHOST = 'localhost'

#Compute the date string for yesterday in ISO-8601 format
yesterday = (date(*time.gmtime()[:3]) - timedelta(1)).isoformat()

message_text = u''

#Using Amara. Easy to just grab the RSS feed
for feed in FEEDS:
    doc = amara.parse(feed)
    message_text += u'\n' + unicode(doc.RDF.channel.title) + u'\n\n'
    current_items = [ item for item in doc.RDF.item
                      if unicode(item.date).startswith(yesterday) ]
    for item in current_items:
        #Get the properties of the link, defaulting to empty string
        title = unicode(getattr(item, 'title', u''))
        href = unicode(getattr(item, 'link', u''))
        desc = unicode(getattr(item, 'description', u''))
        creator = unicode(getattr(item, 'creator', u''))
        message_text += u'<%s>--"%s" (from %s)\n'%(href, title, creator)
        message_text += desc + '\n'

#Be sure to handle Unicode by encoding to UTF-8
msg = MIMEText(message_text.encode('utf-8'))

#Set message metadata
msg['Subject'] = u'del.icio.us bookmarks for %s\n' % yesterday
msg['From'] = FROM
msg['To'] = TO

#Send the message via the specified SMTP server
s = smtplib.SMTP()
s.connect(SMTPHOST)
#s.login(SMTP_USERNAME, SMTP_PASSWORD) #If login is necessary
s.sendmail(FROM, [TO], msg.as_string())
s.close()

This code uses the Amara XML Toolkit (see Resources) to download and parse Web feeds (del.icio.us uses RSS 1.0 for its Web feed format). The two feeds in the example are the one for my own user, and the one for the popular tag. The for loop iterates over the feeds, finds entries for the desired day, and extracts a few fields to build into the accumulated message text, in message_text. The lines after the loop's body are concerned with actually sending the message.

The formal API

If you want to be able to edit entries, or to access data in any way that has not already been packaged into a Web feed for you, you have to use the official API, which is based on authenticated and encrypted HTTP. The secret to these fancier HTTP features is the Python module urllib2. I won't provide a full Python example using the official API because these are pretty easy to find on the Web. But I will mention that whenever you are using a Web 2.0 site's API, you always have to be careful about making queries too often. As the del.icio.us API page says:

Please wait AT LEAST ONE SECOND between queries, or you are likely to get automatically throttled. If you are releasing a library to access the API, you MUST do this.

This page contains other admonishments, and most of these apply to any free and open Web API, so it's well worth paying close attention to the practices that make you a good Web developer citizen.

Working del.icio.us into JavaScript

If you're using ECMAScript (sometimes known as JavaScript) to write tools using del.icio.us, you have another option, similar to Web feeds, for read access. Most Web developers are by now familiar with Asynchronous JavaScript with XML (Ajax). The XML part denotes the way information can be shuttled across the network, though XML isn't necessary (see Resources for more information). Another popular format is JavaScript Object Notation (JSON), which has the advantage of being much easier than XML to parse using ECMAScript. Some del.icio.us JSON feeds are similar to the Web feeds used earlier in this article. When you access URLs such as http://del.icio.us/feeds/json/<username> and http://del.icio.us/feeds/json/<tag>, you get back a JSON representation of recent entries. Listing 2 is a small example of a Web page that includes ECMAScript that loads a JSON feed from del.icio.us.

Listing 2. Code to display recent del.icio.us postings on a Web page


<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
  "http://www.w3.org/TR/html4/strict.dtd">
<html>
  <head>
    <meta content="text/html; charset=iso-8859-1" http-equiv="Content-Type">
    <title>Bookmarks from del.icio.us</title>
    <script type="text/javascript" 
            src="http://del.icio.us/feeds/json/uche?count=10">
      //Loads JSON from del.icio.us, which evaluates to an an array: Delicious.posts
    </script>

    <script type="text/javascript">
//Set up asynchronous display of images
function showImage(img){ return (function(){ img.style.display='inline'; }) };

function showBookmarks()
{
    var ul = document.createElement('ul');
    //Process each item in the del.icio.us JSON feed
    for (var i=0, post; post = Delicious.posts[i]; i++) {
        var li = document.createElement('li');
        var a = document.createElement('a');
        var img = document.createElement('img');
        a.style.marginLeft = '20px';
        img.style.position = 'absolute';
         //Don't display image at first: will be asynchronously turned on
        img.style.display = 'none';
        img.height = img.width = 16;
        //Raw string processing to get the URL of the bookmark icon
        img.src = post.u.split('/').splice(0,3).join('/')+'/favicon.ico';
        img.onload = showImage(img);
        a.setAttribute('href', post.u);
        a.appendChild(document.createTextNode(post.d));
        li.appendChild(img);
        li.appendChild(a);
        ul.appendChild(li);
    }
    //No bullet markings for list items
    ul.style.setProperty('list-style', 'none', 'important');
    //Replace a targeted portion of initial content with the constructed list
    var removeTarget = document.getElementById('removeTarget');
    var updateTarget = document.getElementById('updateTarget');
    updateTarget.removeChild(removeTarget);
    updateTarget.appendChild(ul);
}
    </script>
  </head>
  <body id='updateTarget'>
    <h2>Bookmarks from del.icio.us</h2>
    <p id='removeTarget'>
      <a href="javascript:showBookmarks()">Click here to load addresses</a>
    </p>
  </body>
</html>

The HTML includes two scripts. The first is just the del.icio.us link for retrieving JSON. The resulting object will now be available to other scripts on the page. The second script is triggered by a link on the main HTML page. It reads the JSON and uses DOM instructions to build a list for inclusion into the page. Figure 2 is a clipping from the Web page that is produced when I selected the Click here to load addresses link has been clicked.

Figure 2. Web page dynamically loaded with del.icio.us entries

In conclusion

The combination of open, flexible Web APIs and content uploaded by users and maintained in user networks is the true 2.0 of Web 2.0. The Internet and its users are migrating from a Web of tightly-controlled traffic flowing one way from publishers to readers, to a Web of information managed as a collaboration between the publisher and its community of users. Those of a more optimistic bent see in such a development a shift almost as fundamental as those that powered the great Renaissance in Europe. Whether or not that's a laughable exaggeration, it's important for every Web architect to understand the new techniques and practices, because the Web is certainly moving in a more collaborative direction. In this column, I shall be looking at these techniques and practices in the context of specific Web sites that are leading the vanguard of Web 2.0.

Resources

Learn

del.icio.us: Definitely one of the grandparents of the Web 2.0 movement. Check out the general API information page and the JSON feeds information page.
Social bookmarking: Learn more about the history of this phenomenon from Wikipedia.
ECMAScript: Wikipedia has a wealth of information on ECMAScript and the DOM; learn more by reading one of the many ECMAScript books and tutorials listed here.
"Build apps using Asynchronous JavaScript with XML (Ajax)"" (Naveen Balani and Rajeev Hathi, developerWorks, November 2005): Get into Ajax with this tutorial for a dynamic, asynchronous Web experience without the need for page refreshes.
"Mastering Ajax, Part 7: Using XML in requests and responses" (Brett McLaughlin, developerWorks, October 2006): Find more on the role XML does -- and doesn't -- play in Ajax.
"Cache in with JSON," Bakul L. Patel (developerWorks, October 2006): Learn more about the JavaScript Object Notation (JSON) technology.
developerWorks technical events and Webcasts: Stay current with these resources.
developerWorks Web development zone: Expand your site development skills with articles and tutorials that specialize in Web technologies.

Get products and technologies

JSON.org: Visit this home page that also serves as tutorial and specification for this very simple data format.
del.icio.us tools and add-ons: Tap into the vast ecosystem available for developers and end users alike.
Amara XML Toolkit: Check out this toolkit, an easy way to process XML in Python, used in Listing 1 of this article.
XBELicious: If you want to pull your del.icio.us feeds into your browser bookmarks, you can use this set of scripts. It integrates del.icio.us with the XBEL format, which is a popular way to store browser bookmarks.

Discuss

developerWorks blogs: Participate and get involved in the developerWorks community.

About the author

Comments (Undergoing maintenance)

Trademarks | My developerWorks terms and conditions

Close [x]

Help: Update or add to My dW interests

What's this?

This little timesaver lets you update your My developerWorks profile with just one click! The general subject of this content (AIX and UNIX, Information Management, Lotus, Rational, Tivoli, WebSphere, Java, Linux, Open source, SOA and Web services, Web development, or XML) will be added to the interests section of your profile, if it's not there already. You only need to be logged in to My developerWorks.

And what's the point of adding your interests to your profile? That's how you find other users with the same interests as yours, and see what they're reading and contributing to the community. Your interests also help us recommend relevant developerWorks content to you.

View your My developerWorks profile

Return from help

Close [x]

Help: Remove from My dW interests

What's this?

Removing this interest does not alter your profile, but rather removes this piece of content from a list of all content for which you've indicated interest. In a future enhancement to My developerWorks, you'll be able to see a record of that content.

View your My developerWorks profile

Return from help

static.content.url=http://www.ibm.com/developerworks/js/artrating/

SITE_ID=1

Zone=Web development, Open source, XML

ArticleID=170033

ArticleTitle=Real Web 2.0: Bookmarks? Tagging? Delicious!

publish-date=10262006

author1-email=uche@ogbuji.net

author1-email-cc=htc@us.ibm.com