Introduction

In Part 1 of this series, we introduced JSONP as an effective cross-domain communication technique, one that lets you bypass the same-origin policy limitations imposed by the current browsers. We showed how you can use jQuery's native JSONP support to gather JSON-formatted content from third-party services, which we call JSONP services. This article introduces Yahoo! Query Language (YQL) as a single endpoint JSONP service that lets you query, filter, and combine data from different data sources.

In addition, this article shows you how to build a jQuery implementation of a mashup application that collects stock quotes, The New York Times RSS feed, and weather forecast data to present in a single Web page.

Web as a database

There is a lot of structured data on the Web that is exposed to developers through various Web services or APIs. These services require you to access and query them, and then parse results in non-standard ways. Each one of them has its own URL for access as well as documents describing how to query and understand the results. You can view YQL as a unified way to query data served up by different services at different URLs. YQL provides a single endpoint service that lets you query, filter, and combine data across the Web without having to deal with each service's peculiarities.

YQL is basically a query language based on SQL that lets you query and retrieve structured data from multiple services in a standard way. Yet YQL is also a query engine, which is hosted by Yahoo!, that is exposed at a REST endpoint. YQL uses the familiar relational database model of tables and rows. However, data is interpreted and structured as XML internally. If the underlying data source does not provide XML data, YQL does the conversion. It can return responses as either XML or JSON (which you can specify while calling the service). Moreover, you can also specify a callback function if you request JSON data from the service. This makes the YQL service a JSONP service, which we think is very significant: It brings automatic JSONP support to many Web services.

YQL understands and supports data sources like RSS, Atom, JSON, XML, CSV, HTML, Flickr, Yahoo! Finance, Weather, and so on. Because YQL also supports external data sources (external to Yahoo!), the possibilities of building mashup applications are endless. We could say that it connects the Web to your applications in a single interface.

As the article already mentioned, YQL is a SQL-like language to query structured data in the Web. Hence, SELECT is the main verb in YQL statements. The syntax is also very similar to SQL:

SELECT fields FROM source_table WHERE filter ..

The following listing shows a simple query to retrieve the headlines from The New York Times RSS feed:

select title from rss where url="http://www.nytimes.com/services/xml/rss/nyt/HomePage.xml"

A more involved example of a YQL query to retrieve vegetarian restaurants in New York, limiting the number to 3, would be:

select Title, Address, City, Rating.AverageRating from local.search 
  where query="vegetarian" and location="New York, NY" limit 3

The above query would return a JSON response like what is shown in Listing 1 (omitting returned metadata about the query).

"results": {
  "Result": [
    {
      "Title": "World of Vegetarian Incorporated",
      "Address": "24 Pell St",
      "City": "New York",
      "Rating": {
        "AverageRating": "5"
      }
    },
    {
      "Title": "Counter",
      "Address": "105 1st Ave",
      "City": "New York",
      "Rating": {
        "AverageRating": "4"
      }
    },
    {
      "Title": "Red Bamboo",
      "Address": "140 W 4th St",
      "City": "New York",
      "Rating": {
        "AverageRating": "4.5"
      }
    }
  ]
}

You can even query the Web through YQL:

select title,abstract,url from search.web where query="Bart Simpson"

A good way to learn and improve your understanding of YQL is to use the YQL console, which is provided by Yahoo! at the following address: http://developer.yahoo.com/yql/console/ . (See Figure 2 for a screen capture.)

Try to copy and paste any YQL statements you see in this article into the YQL console to examine the resulting JSONP responses (don't forget to check the radio button for getting JSON output format instead of XML).

Back to top

Querying YQL with jQuery

A typical YQL GET request looks like this:

http://query.yahooapis.com/v1/public?q=[command]&[query parameter]

where command is the YQL command and query parameter represents optional parameters to the service. Because you are interested in YQL's JSONP support, your requests must always include two optional parameters in order to get JSONP responses from the YQL service. So your requests will use the following format:

http://query.yahooapis.com/v1/public?q=[command]&format=json&callback=?

where format is a request parameter defining the requested response format and callback is the name of your callback function in your Web application, which will be supplied by jQuery in this case. Note that we put a ? as the callback function name, instead of a real function name, to let jQuery know that it should generate a function for us.

A typical call to the YQL service in jQuery looks like the code in Listing 2.

jQuery.getJSON(yqlUrl, function(data) {
// data is the JSON response
// process the response here
}); 

Back to top

Building the mashup

Now that you know how to query and get JSONP responses from YQL, you can start building your mashup Web page. We are now going to develop a Web page with three widgets, each of which gets its content from a different part of the Web, but through the same unified service, which is YQL.

The first widget is a stock quote widget that queries and retrieves the latest stock quotes for IBM®, Yahoo!, Google and Microsoft®. The data source for this query is Yahoo! Finance CSV files. Listing 3 shows the YQL for the first widget.

select symbol, price from csv 
  where url='http://download.finance.yahoo.com/d/quotes.csv?
  s=IBM,YHOO,GOOG,MSFT&f=sl1d1t1c1ohgv&e=.csv' and 
  columns='symbol,price,date,time,change,col1,high,low,col2'

The second widget shows the headlines from The New York Times RSS feed, which gets its data with the YQL shown in Listing 4.

select title, link from rss 
  where url="http://www.nytimes.com/services/xml/rss/nyt/HomePage.xml"

The third and the last widget, shown in Listing 5, is a weather forecast widget, which shows the forecast for the postcode 10504 (Armonk, NY):

select * from weather.forecast where location=10504

Each widget has the skeletal code shown in Listing 6 in the Web page.

<div class="widget">
  <div class="widget-head">[WIDGET HEADER]</div>
  <div class="widget-content" id="[CONTENT ID]">[WIDGET CONTENT]</div>
</div>

Each widget is a div element of class type widget, which has a header (widget-head) and content area (widget-content). The content area of a widget will be filled by your JavaScript code that will pull in JSONP data and append it to the content section of the widget.

You start building your mashup Web page by defining three widget placeholders: one for stock quotes, one for NYT news, and one for the weather forecast. These are shown in Listing 7.

<div class="widget">
  <div class="widget-head">Lastest stock quotes</div>
  <div class="widget-content" id="quotes"></div>
</div>   
<div class="widget">
  <div class="widget-head">NYT news headlines</div>
  <div class="widget-content" id="headlines"></div>
</div>   
<div class="widget">
  <div class="widget-head">Weather for Armonk NY</div>
  <div class="widget-content" id="weather"> </div>
</div>   

Note that you leave the content areas empty because you're going to use JSONP to fill them with dynamic data. As you will shortly see, jQuery makes it very easy to generate HTML fragments on-the-fly. Each one of these areas will be filled by the skeletal jQuery code shown in Listing 8.

$.getJSON(yqlUrl, function(data){
  // loop through the items
  $.each(data.query.results.[ITEM NAME], function(index, item){
    // process each item here
    // generate HTML to append to the widget's content area
  });
});

The code in Listing 8 issues a GET request to the yqlUrl (which you must provide) and pulls in the JSONP response. After the JSON response arrives (which we named data), you get the chance to loop through the result items and process them one by one. [ITEM NAME] is a place holder for the array of items that may be named differently depending on your data source. Use the previously mentioned YQL console to see the resulting responses and to determine the exact response structure.

First widget

Listing 9 shows the first jQuery fragment to fill the stock quotes widget content:

var yqlUrl1= "http://query.yahooapis.com/v1/public/yql?q=
  select%20symbol%2C%20price%20from%20csv%20
  where%20url%3D'http%3A%2F%2Fdownload.finance.yahoo.com%2Fd%2Fquotes.csv%3F
  s%3DIBM%2CYHOO%2CGOOG%2CMSFT%26f%3Dsl1d1t1c1ohgv%26e%3D.csv'%20and%20
  columns%3D'symbol%2Cprice%2Cdate%2Ctime%2Cchange%2Ccol1%2Chigh%2Clow%2Ccol2'
  &format=json&callback=?";
$.getJSON(yqlUrl1, function(data){
  $.each(data.query.results.row, function(index, item){
    $('#quotes')
    .append(
      $('<p/>')
        .append($('<span class="left"/>').text(item.symbol))
        .append($('<span class="right"/>').text('$'+item.price))
      );
    });
});

How did we get the properly escaped URL for the first widget? When you look at the YQL console, you see a section with the title "The Rest query" on the upper right corner of the screen. After successfully testing your YQL statement, you should copy the query string from there and use it as the first parameter for the getJSON() function.

The stock quotes widget loops through the items and appends a p element that contains two span elements into the div element identified by the id #quotes. The resulting HTML code for the content area looks like what is shown in Listing 10.

 <div id="quotes" class="widget-content">
  <p>
    <span class="left">"IBM"</span>
    <span class="right">$91.51</span>
  </p>
  <p>
    <span class="left">"YHOO"</span>
    <span class="right">$12.22</span>
  </p>
  <p>
    <span class="left">"GOOG"</span>
    <span class="right">$353.11</span>
  </p>
  <p>
    <span class="left">"MSFT"</span>
    <span class="right">$18.12</span>
  </p>
</div>

Second widget

After finishing the first widget, we'll proceed to the second one.

var yqlUrl2= "http://query.yahooapis.com/v1/public/yql?q=
  select%20title%2C%20link%20from%20rss%20
  where%20url%3D%22http%3A%2F%2Fwww.nytimes.com%2Fservices%2Fxml%2Frss%2Fnyt%2F
  HomePage.xml%22&format=json&callback=?";
$.getJSON(yqlUrl2, function(data){
  $.each(data.query.results.item, function(index, item){
    $("<a href='" + item.link + "' target="_blank\"/>")
    .html(item.title)
    .appendTo($('#headlines'))
    .wrap('<p/>');
  });
});
			

The code in Listing 11 first pulls in the title and link elements of the NYT RSS feed. It then appends p elements, containing a link to the main article, into the widget's content area. The resulting HTML code for the content area would look like the code in Listing 12.

<div id="headlines" class="widget-content">
  <p>
    <a target="_blank" href="http://www.nytimes.com/2009/02/19/business/19housing.html
    ?partner=rss&emc=rss">Obama Unveils $75 Billion Plan to Fight Home Foreclosures</a>
  </p>
  <p>
    <a target="_blank" href="http://www.nytimes.com/2009/02/19/business/economy/19fed.html
    ?partner=rss&emc=rss">Fed Offers Bleak Economic Outlook</a>
  </p>
  ...
</div>
				

Third widget

The third widget works very similar to the previous two. However, this time we append the item.description into the content area directly, because it already contains HTML formatted data.

var yqlUrl3= "http://query.yahooapis.com/v1/public/yql?q=
  select%20*%20from%20weather.forecast%20where%20location%3D10504&
  format=json&callback=?";
$.getJSON(yqlUrl3, function(data){
  $.each(data.query.results.channel, function(index, item){
    $('#weather')
    .append($('<p/>').html(item.description));
  });	
});
            

For the sake of completeness, Listing 14 combines everything in one place (except the CSS we used to style the widgets).

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <script type="text/javascript" src="jquery-1.3.1.min.js"/>
    <style type="text/css">
    ...
    </style>
    <title>JSONP Mashup</title>
  </head>
  <body>
    <div class="widget">
      <div class="widget-head">Latest stock quotes</div>
      <div class="widget-content" id="quotes"/>
    </div>
    <div class="widget">
      <div class="widget-head">NYT news headlines</div>
      <div class="widget-content" id="headlines"/>
    </div>
    <div class="widget">
      <div class="widget-head">Weather for Armonk, NY</div>
      <div class="widget-content" id="weather"> </div>
    </div>
    <script type="text/javascript">
      var yqlUrl1="http://query.yahooapis.com/v1/public/yql?q=
        select%20symbol%2C%20price%20from%20csv%20
        where%20url%3D'http%3A%2F%2Fdownload.finance.yahoo.com%2Fd%2Fquotes.csv%3F
        s%3DIBM%2CYHOO%2CGOOG%2CMSFT%26f%3Dsl1d1t1c1ohgv%26e%3D.csv'%20and%20
        columns%3D'symbol%2Cprice%2Cdate%2Ctime%2Cchange%2Ccol1%2Chigh%2Clow%2Ccol2'
        &format=json&callback=?";
      $.getJSON(yqlUrl1, function(data){
        $.each(data.query.results.row, function(index, item){
          $('#quotes')
          .append(
            $('<p/>')
            .append($('<span class="left"/>').text(item.symbol))
            .append($('<span class="right"/>').text('$'+item.price))
          );
        });
      });

      var yqlUrl2="http://query.yahooapis.com/v1/public/yql?q=
        select%20title%2C%20link%20from%20rss%20
        where%20url%3D%22http%3A%2F%2Fwww.nytimes.com%2Fservices%2Fxml%2Frss%2Fnyt%2F
        HomePage.xml%22&format=json&callback=?";
      $.getJSON(yqlUrl2, function(data){
        $.each(data.query.results.item, function(index, item){
          $("<a href='" + item.link + "' target="_blank\"/>")
          .html(item.title)
          .appendTo($('#headlines'))
          .wrap('<p/>');
        });
      });

      var yqlUrl3= "http://query.yahooapis.com/v1/public/yql?q=
        select%20*%20from%20weather.forecast%20where%20location%3D10504&
        format=json&callback=?";
      $.getJSON(yqlUrl3, function(data){
        $.each(data.query.results.channel, function(index, item){
          $('#weather')
          .append($('<p/>').html(item.description));
        });	
      });
    </script>
  </body>
</html>
            

Figure 3 shows a screenshot of the resulting Web page when this article was written.

Back to top

Conclusion

YQL is a powerful service that makes client-side mashups possible without using server-side proxies. Coupled with its JSONP support and jQuery, YQL lets you access structured data across the Web through a single unified interface. In this article, you learned how to use jQuery and YQL to build up a mashup Web page with few lines of code. You can use this as a beginning and improve the code to make it more sophisticated. Here are some tips:

• Put an image link into the widget header that refreshes the content area when clicked.
• Provide an input field for the user to enter stock symbols and fetch the quotes for the entered symbols only.
• Show not only RSS items' title and link, but also their descriptions.

Resources

• "Cross Domain Communications with JSONP: Part 1" (developerWorks, February 2009): Read the first part of this article series, which introduces JSONP.
  
  
• Yahoo! Query Language: Get to know YQL from its inventors.
  
  
• YQL Console: Try the YQL console to learn about and get practice with the YQL.
  
  
• "Working with jQuery, Part 1: Bringing desktop applications to the browser" (developerWorks, September 2008): Get to know jQuery in this three-part article series.
  
  

About the authors

Seda Özses is a member of the ibm.com corporate webmaster team. Her main focus is on the ibm.com corporate Web portals, where she manages the XSL work and the requirements for her team. You can contact her at seda.ozses@at.ibm.com.

Salih Ergül is a self-employed IT architect who specializes in large, mission-critical Enterprise Application Integration projects in the telecom and banking industries. He has authored several articles on Java programming and currently works in the banking industry.

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Introduction
• Web as a database
• Querying YQL with jQuery
• Building the mashup
• Conclusion
• Resources
• About the authors
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into Web development on developerWorks

Calling all developers

Download no charge Lotus Domino Designer

Special offers