Skip to main content

By clicking Submit, you agree to the developerWorks terms of use.

The first time you sign into developerWorks, a profile is created for you. Select information in your profile (name, country/region, and company) is displayed to the public and will accompany any content you post. You may update your IBM account at any time.

All information submitted is secure.

  • Close [x]

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerworks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

By clicking Submit, you agree to the developerWorks terms of use.

All information submitted is secure.

  • Close [x]

The ultimate mashup -- Web services and the semantic Web, Part 1: Use and combine Web services

Explore mashup concepts and build a simple mashup

Nicholas Chase (ibmquestions@nicholaschase.com), Freelance writer, Backstop Media
Nicholas Chase has been involved in Web site development for companies such as Lucent Technologies, Sun Microsystems, Oracle, and the Tampa Bay Buccaneers. Nick has been a high school physics teacher, a low-level radioactive waste facility manager, an online science fiction magazine editor, a multimedia engineer, an Oracle instructor, and the Chief Technology Officer of an interactive communications company. He is the author of several books, including XML Primer Plus (Sams).

Summary:  As Web services grow in popularity, enterprising Web and application developers create new and innovative applications with their data. In addition to single-service applications, developers are creating mashups, applications that combine data from multiple services to create something new. This series chronicles the creation of the ultimate mashup, an application that not only stores data from different mashups but uses semantic technology to enable users to create their own mashups by swapping services, or even by picking and choosing data. It uses Java™ programming and a combination of servlets, JSP, software from the open source Jena project, and DB2's new native XML capabilities. In this part, Nicholas Chase introduces the concept of mashups, shows you how they work and how to build a simple version of one.

View more content in this series

Date:  08 Mar 2007 (Published 22 Aug 2006)
Level:  Intermediate PDF:  A4 and Letter (416 KB | 41 pages)Get Adobe® Reader®

Activity:  36728 views
Comments:  

Moving the application to the Web

At this point the application is complete, but it still needs to be transferred to the Web. To do that, you need to add the code to a servlet. The servlet has two main methods: doGet() and doPost(). When the browser requests the servlet using the GET method, it displays a simple form that enables the user to enter a search query. When the user submits the form, it submits it through the POST method, and the servlet processes the data as before.

Creating the servlet

First, create the actual servlet, as in Listing 21.


Listing 21. Creating the actual servlet
                    
import java.io.IOException;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class MashupClientServlet 
                       extends javax.servlet.http.HttpServlet 
                       implements javax.servlet.Servlet {

   public MashupClientServlet() {
      super();
   }      
   

   protected void doGet(HttpServletRequest request, 
                        HttpServletResponse response) 
                           throws ServletException, IOException {
      response.getWriter().print("<html>");
      response.getWriter().print("<head></head>");
      response.getWriter().print("<body>");
      
      response.getWriter().print("<h1>The Mashup</h1>");
      response.getWriter().print("<p>Enter a keyword:</p>");
      response.getWriter().print(
              "<form action='MashupClientServlet' method='post'>");
      response.getWriter().print(
                             "<input type='text' name='query' />");
      response.getWriter().print("<input type='submit' />");
      response.getWriter().print("</form>");
      response.getWriter().print("</body>");
      response.getWriter().print("</html>");
   }     
   
   protected void doPost(HttpServletRequest request, 
                         HttpServletResponse response) 
                          throws ServletException, IOException {
      // Actual processing goes here
   }               
}

This is just a standard servlet. If you save it and deploy it to your server, and then call it from the browser, you will see something similar to Figure 1.


Figure 1. The doGet() method
The doget() method

Adjusting the Service information

Because you now pull keywords from the form, you need to make a slight change so that the keywords are no longer embedded in the URL that represents the REST requests, as show in Listing 22.


Listing 22. Adjusting the Service information
                    
...
      thisService.name = "Yahoo! News Search";
      thisService.baseURL = 
"http://api.search.yahoo.com/NewsSearchService/V1/newsSearch?appid=ma
shupid&type=all&query=";
      thisService.template =
 "<p><b><value/></b>:<value/></p>";
...
      thisService2.name = "YouTube";
      thisService2.baseURL = 
"http://www.youtube.com/api2_rest?method=youtube.videos.list_by_tag&de
v_id=s2gNEM-7qoU&tag=";
      thisService2.template =
 "<p><value/>:<value/></p>";
...

Otherwise, the information is unchanged. Add the keywords back on before you call the service.


The complete servlet

Completing the servlet is a straightforward matter of transferring the functionality you've already built to the doPost() method, as you can see in Listing 23.


Listing 23. The full servlet
                    
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathFactory;

import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
import java.io.IOException;
import java.io.StringReader;

public class MashupClientServlet extends javax.servlet.http.HttpServlet
 implements javax.servlet.Servlet {

   public MashupClientServlet() {
      super();
   }      
   

   protected void doGet(HttpServletRequest request, 
                        HttpServletResponse response) 
                        throws ServletException, IOException {
      response.getWriter().print("<html>");
...
      response.getWriter().print("</html>");
   }     
   
   protected void doPost(HttpServletRequest request, 
                         HttpServletResponse response) 
                              throws ServletException, IOException {
      try {

         String query = request.getParameter("query").toString();
         
         Service[] svcs = Service.getServices();
         DocumentBuilder builder = 
           DocumentBuilderFactory.newInstance().newDocumentBuilder();

         Document hostDoc = builder.parse(
                   new InputSource(new StringReader("<div />")));
         Node hostRoot = hostDoc.getDocumentElement();         
         
         for (int s=0; s < svcs.length; s++){
   
            Service svc = svcs[s];
            Document document = builder.parse(svc.baseURL+query);
      
            XPath xpath = XPathFactory.newInstance().newXPath();
...

         }   
            
         DOMSource source = new DOMSource(hostDoc);
         StreamResult result = new StreamResult(response.getWriter());
         TransformerFactory transFactory = 
                           TransformerFactory.newInstance();
         Transformer transformer = transFactory.newTransformer();
         transformer.transform(source, result);   
         
      } catch (Exception e){
      
         e.printStackTrace();
      }

   }               
}

All you really did here is to take the contents of the main() method of the standalone class and add it as the contents of the doPost() method, with three small changes. First, you retrieved the query entered by the user, and then you added the query to the end of the URLs to be requested.

Finally, rather than output the results to the command line, you output them to the Web page.

The results should look something like Figure 2.


Figure 2. Results of running the servlet
results of running the servlet

Adding attribute templates

If you scroll down the page, you'll find that YouTube results are also on the page, but they don't look very nice, because they don't have any images. Similarly, it would be nice to be able to provide links to the news items Yahoo! presents. To do that, however, you need to provide a way to specify attributes in your templates, and not just elements.

To do that, you can add a new attribute to the Service class, and alter the templates somewhat, as you can see in Listing 24.


Listing 24. Adding attributes to the Service class
                    
public class Service {

   String name = "";
   String baseURL = "";
   String template = "";
   String[] elementValues = {"title", 
                       "desc"};
   String[] attributeValues = {"src"};
   String recordExp = ""; 


public static Service[] getServices(){
      Service[] services = new Service[2];
          
      Service thisService = new Service();
      thisService.name = "Yahoo! Search";
      thisService.baseURL = 
"http://api.search.yahoo.com/NewsSearchService/V1/newsSearch?appid=ma
shupid&type=all&query=";
      thisService.template = 
           "<p><b><a
 value='href'><value/></a></b>:<value/>
</p>";
      thisService.elementValues[0] = "Title";
      thisService.elementValues[1] = "Summary";
      thisService.attributeValues[0] = "ClickUrl";
      thisService.recordExp = "/ResultSet/Result";   
      services[0] = thisService;

      Service thisService2 = new Service();
      thisService2.name = "YouTube";
      thisService2.baseURL = 
"http://www.youtube.com/api2_rest?method=youtube.videos.list_by_tag&de
v_id=s2gNEM-7qoU&tag=";
      thisService2.template = 
                  "<p><img value='src' />
 <value/>:<value/></p>";
      thisService2.elementValues[0] = "title";
      thisService2.elementValues[1] = "description";
      thisService2.attributeValues[0] = "thumbnail_url";
      thisService2.recordExp = "/ut_response/video_list/video";
      services[1] = thisService2;
      
      return services;
      
   }
}

Here you added an attributeValues array for each object, but you also added new attributes to the templates. The form is slightly different from the value elements. The value attribute determines the element to carry the attribute, as well as the name of the attribute to appear in the final document. For example, the template element <img value='src' /> gets translated into an element of <img src="(contents of the thumbnail_url)" />.

Let's see how that works.


Processing the attribute templates

Processing the attribute values is similar to processing the element values, as you can see in Listing 25.


Listing 25. Processing attribute templates
                    
...
                     String replacementValue = 
                                 valueNode.getTextContent();
                     Node replacementNode = 
                      templateDoc.createTextNode(replacementValue);
                     parentNode.replaceChild(replacementNode, 
                                             thisTemplateNode);

                  }
               }

               expression = "//@value";
               templateNodes = (NodeList) xpath.evaluate(
                   expression, templateDoc, XPathConstants.NODESET);
               for (int j=0; j < templateNodes.getLength(); j++){
            
                  Attr thisNode = (Attr) templateNodes.item(j);
                  Element parentElement =  
                                 (Element)thisNode.getOwnerElement();
                  
                  parentElement.removeAttribute("value");
                  if ( (Boolean)xpath.evaluate(svc.attributeValues[j], 
                                               recordNodes.item(i), 
                                             XPathConstants.BOOLEAN)) {
                     Node valueNode = (Node)xpath.evaluate(
                                               svc.attributeValues[j],  
                                               recordNodes.item(i), 
                                               XPathConstants.NODE);
                     parentElement.setAttribute(
                                            thisNode.getNodeValue(), 
                                         valueNode.getTextContent());   

                  } 
               }
               
               Node importedNode = hostDoc.importNode(
                         templateDoc.getDocumentElement(), true);
               hostRoot.appendChild(importedNode);
               hostRoot.appendChild(hostDoc.createTextNode("\n"));
            
            }

         }   
...

First, define the XPath expression for attributes called value, and get a list of Nodes to which it applies. For each one, get a reference to the Node itself, and then the element carrying it. Once you have that, you can remove the old attribute and if the relevant information is present, add a new one with the replacement value.

The response is a page that includes links and images, as you can see in Figure 3.


Figure 3. Links and images
links and images

6 of 11 | Previous | Next

Comments



static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=XML, Information Management, Java technology, Open source, Web development, SOA and Web services
ArticleID=154966
TutorialTitle=The ultimate mashup -- Web services and the semantic Web, Part 1: Use and combine Web services
publish-date=03082007
author1-email=ibmquestions@nicholaschase.com
author1-email-cc=dwxed@us.ibm.com