Working with web services using the EXPath HTTP client

HTTP testing with EXPath

EXPath is a community-driven effort whose goal is to define common extensions for XML technologies, making it easier for developers to create portable applications using well-defined function libraries. The EXPath HTTP Client module defines a set of functions that enable the sending of HTTP and HTTPS requests. Learn to use this library to interact with HTTP from both XSLT and XQuery, and find out how to test HTTP requests using a primitive HTTP tester utility built with XQuery and the EXPath HTTP Client module.

James R. Fuller (jim.fuller@webcomposite.com), Technical Director, Webcomposite s.r.o.

Photo of Jim FullerJames Fuller has been a professional developer for more than 15 years, working with several software blue-chip companies in both his native USA and the UK. He has co-written a few technology-related books and regularly speaks and writes articles focusing on XML technologies. He is a founding committee member for XML Prague and was in the gang responsible for EXSLT. He spends most of his time working with XML databases and XQuery. You can reach James at jim.fuller@webcomposite.com.



15 February 2011

Also available in Chinese Japanese

The EXPath suite of specifications is an attempt at harmonizing the use of common functionality across the spectrum of XML technologies. EXPath is open to anyone to contribute to and represents an alternate approach toward defining useful standards that can positively affect how you use XML standards—in much shorter time scales (over those defined by official standards organizations).

Frequently used acronyms

  • HTTP: Hypertext Transfer Protocol
  • HTTPS: HTTP over Secure Sockets Layer
  • MIME: Multipurpose Internet Mail Extensions
  • REST: Representational State Transfer
  • SOAP: Simple Object Access Protocol
  • SQL: Structured Query Language
  • URI: Uniform Resource Indicator
  • URL: Uniform Resource Locator
  • XML: Extensible Markup Language
  • XSLT: Extensible Stylesheet Language Transformation

EXPath, which is maintained by XML luminary Florent Georges, is based on the premise that XPath, out of all the XML technologies, provides a lot of the power beneath the XML stack of technologies. By providing specifications that enhance and extend XPath, it is possible to integrate new functionality in a consistent and rigorous manner, set within the context of the XPath data model.

EXPath is, first and foremost, an attempt to define functional specifications. It is then up to the community to build implementations based on these specifications.

Now is the perfect time to join the debate and contribute your views on the functionality you would like to see in the final versions of these EXPath modules. See Resources for several links to both the website and related mailing lists.

The EXPath HTTP Client module

The EXPath HTTP Client module defines a single http:send-request() function that allows you to create and send HTTP requests. The http prefix is bound to the http://expath.org/ns/http-client namespace URI. The err prefix, which defines error elements, is bound to the http://expath.org/ns/error namespace URI.

The http:send-request function is used to generate an HTTP request with any of the HTTP verbs—HTTP GET, HTTP POST, HTTP PUT, and HTTP DELETE. When the function is invoked, it sends an HTTP request to the specified server and emits the HTTP response, along with all metadata associated with the HTTP connection.

A number of different signatures are available for http:send-request: I prefer to use the following one, where you pass in an http:element that represents the HTTP request that you want to make:

http:send-request($request as element(http:request))

EXPath modules

The specifications of many EXPath modules already come with sufficiently stable implementations. Those undergoing the most active development are:

  • HTTP Client. This module defines a set of functions for sending HTTP requests and handling responses.
  • Zip. This module provides a set of functions for reading the structure of a ZIP file, reading entries out of a ZIP file, updating entries in an existing ZIP file, or creating new files.
  • File Handling. This module defines a set of functions for reading and modifying file system information. It allows you to read files, list directories, create them, and get information about files.
  • Packaging. This is a standard packaging system for several core XML technologies, including XSLT, XQuery, XProc, and XML Schema. After the technologies are packaged, you can deploy libraries on every supported processor.
  • Webapp container. This is a standard web application container and framework for several core XML technologies.

HTTP request

The http:request element is a wrapper around the body of the request and contains several attributes needed to send an HTTP request:

  • method. HTTP get, post, put, delete, or any of the other HTTP verbs.
  • href. The URI the request has to be sent to.
  • status-only. How the response appears. (If it is true, only the status code and the headers are returned.)
  • auth-method. Defines the Basic or Digest authorization method, used when accessing secured URLs.
  • send-authorization. Must be set to True when using an auth-method.
  • username. The user name used for authentication.
  • password. The password used for authentication.
  • override-media-type. Overrides the Content-Type header that the server returns.
  • follow-redirect. Controls whether an HTTP redirect is automatically followed.
  • timeout. The maximum number of seconds to wait for the server to respond.

Listing 1 shows how an http:request element is constructed and optionally can contain other elements, such as http:header elements and http:body (or, if multipart, http:multipart). You use the http:body element to send data (POST or PUT) to the server. If you send just an HTTP GET, then you omit the http:body element.

Listing 1. The http:request element
<http:request
     method = ncname 
     href? = uri 
     status-only? = boolean 
     username? = string 
     password? = string 
     auth-method? = string 
     send-authorization? = boolean 
     override-media-type? = string 
     follow-redirect? = boolean 
     timeout? = integer> 
     (http:header*,
     (http:multipart| http:body)?) 
</http:request>

You use the http:header element to define custom HTTP headers you want to send with. For example, to associate your HTTP request with a user agent, you can define <http:header name="User-Agent" value="My HTTP Tester/1.0"/>:

<http:header name = string value = string/>

The http:request element can contain an http:body element, as in Listing 2. This element contains the content you want to send to a server. For example, to send an HTTP POST, you place the name=value pairs within this element.

Listing 2. The http:body element
<http:body
     media-type = string 
     src? = uri 
     method? = "xml" | "html" | "xhtml" | "text" | "binary" | qname-but-not-ncname 
     byte-order-mark? = "yes" | "no"
     cdata-section-elements? = qnames 
     doctype-public? = string 
     doctype-system? = string
     encoding? = string 
     escape-uri-attributes? = "yes" | "no" indent? = "yes" | "no"
     normalization-form? = "NFC" | "NFD" | "NFKC" | "NFKD" | "fully-normalized" 
        | "none" | nmtoken 
     omit-xml-declaration? = "yes" | "no" 
     standalone? = "yes" | "no" | "omit"
     suppress-indentation? = qnames 
     undeclare-prefixes? = "yes" | "no" 
     output-version? = nmtoken> any* 
</http:body>

The http:body element, used with HTTP POST or PUT, specifies attributes that define how the HTTP body is to be processed and sent:

  • media-type. The MIME media type of the body part.
  • src. Instead of inline, you can specify the contents of the body as another URI.
  • method. The format of the body (xml, html, xhtml, text, binary).
  • cdata-section-elements. Specifies elements that will be encoded as CDATA sections.
  • doctype-public. Defines the body public doctype.
  • doctype-system. Defines the body system doctype.
  • encoding. The character encoding of the body.
  • indent. Specifies whether the body is automatically indented before sending.
  • omit-xml-declaration. Specifies whether to exclude the XML declaration in the body.
  • suppress-indentation. Specifies elements that are not indented when the indent attribute is set to True.

If you read the EXPath HTTP specification, you might find more attributes. Be aware that some of these attributes are still under development, so I include only those that are clearly defined.

The code shown so far demonstrated all the components that make up a complete HTTP request. The HTTP protocol also provides functionality for sending files with HTTP requests. In general, HTTP treats a multipart message body no differently than any other media type, as Listing 3 shows: strictly as payload (separate HTTP request). The element for defining EXPath HTTP is the http:multipart element.

Listing 3. The http:multipart element
<http:multipart media-type = string
     boundary? = string>
     (http:header*,
     http:body)+
     </http:multipart>

As the element defines a separate HTTP request (subrequest), you use the media-type to define the media type for this request. The boundary attribute defines the boundary marker used to separate different parts of the multipart message contained within the nested http:body.

HTTP response

After the http:send-request() function is invoked, it returns a sequence, the first item being the http:response element and the following items containing the body of the HTTP response (see Listing 4).

Listing 4. An http:response element example
<http:response 
status = integer 
message = string> 
(http:header*)
</http:response> 

.... HTTP Response body ....

The status attribute indicates the HTTP RESPONSE STATUS code (for example, 200) sent back from the server. The http:response element contains HTTP response headers, multipart, and body sent back from the server.


Getting started

The EXPath effort is mainly focused on defining specifications, not necessarily providing implementations of EXPath module functionality. Luckily, a few reference implementations of the EXPath HTTP Client are available from the EXPath HTTP Client module page for use today:

  • Saxon. Installed as an EXPath package (.xar).
  • eXist. Installed as an EXPath package or uses the XQuery library directly.
  • Zorba. An implementation built into the Zorba XQuery processor.
  • MarkLogic Server. Used as an XQuery library.
  • Microsoft® .NET Framework. Provided through the myxsl.net ASP.net extension.

As the standards are still evolving, implementations can be volatile. Make sure that you check the basic operation before using any of these modules in a production setting.

An XSLT example

Put what you've learned about the EXPath HTTP Client module into action by invoking an HTTP request from within an XSLT transformation. You use the SAXON XSLT and XQuery processor from Michael Kay (Saxon version 9.2) as your XSLT processor.

To install the EXPath HTTP Client implementation, you need to use another useful EXPath module, the EXPath Packaging System, which installs the module automatically. To install the client, perform the following steps:

  1. Download the EXPath Packaging module for SAXON.
  2. Download the EXPath HTTP Client module for SAXON.
  3. Install the EXPath Packaging module (follow the README file).
  4. Using the EXPath Packaging module, deploy the EXPath HTTP Client XAR package.

Your SAXON processor should be able to access new functionality simply by defining the EXPath HTTP Client namespace and invoking the http:send-request function, as in Listing 5.

Listing 5. An HTTP GET example called in XSLT
<?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:http="http://expath.org/ns/http-client" version="2.0"> <xsl:template
name="main"> <xsl:variable name="req" as="element()"> <http:request
     href="http://neo.jpl.nasa.gov/risk/" method="get"/>
</xsl:variable> <xsl:copy-of
select="http:send-request($req)[2]"/> </xsl:template>
</xsl:stylesheet>

This code makes an HTTP GET request on the URL, http://neo.jpl.nasa.gov/risk/, which incidentally is a website dedicated to assessing the impact risk from near-earth-orbiting asteroids. This HTTP GET request is equivalent to what happens when you request a web page from your browser. You could change the href attribute to any valid (and accessible) web URL.

The http:response element that is returned from the function contains the HTTP response element, which in turn contains http:header elements, followed by the actual requested web page content. Listing 6 shows the markup.

Listing 6. HTTP response from http://neo.jpl.nasa.gov/risk/
<http:response
xmlns:http="http://expath.org/ns/http-client" 
status="200"> 
<http:header name="Date" value="Sun, 14 Nov 2010 10:37:34 GMT"/> 
<http:header name="Server" value="Apache"/> <http:header 
          name="Accept-Ranges" value="bytes"/> 
<http:header name="Transfer-Encoding" value="chunked"/>
<http:header name="Content-Type" value="text/html"/> 
<http:body media-type="text/html"/> 
</http:response> 
... actual web page ....

As you are interested in returning the actual web page, you used a predicate on the http:send-request function that selects the second item in the returned sequence:

http:send-request($req)[2]

An XQuery example

Invoke your function from within the eXist XML database by referring to the EXPath HTTP Client xquery library implementation—http-client.xqm—bundled with the example download for the article.

You need to run Listing 7 in an instance of the eXist XML database loaded as an XQuery file. But don't worry if you can't install the database: I provide an "install free" method later using the Zorba XQuery processor.

Listing 7. An XQuery example
import module namespace http = "http://expath.org/ns/http-client" at "http-client.xqm"; 
let $url:= 'http://neo.jpl.nasa.gov/risk/' 
let $req := <http:request href="{$url}" method="get"/> 
return 
     http:send-request($req)[1]

Importing XQuery libraries is more familiar for developers (EXPath packaging being a newer standard), and I use this method for most of the examples in the article.

Instead of returning the web page, as with the XSLT example, opt to return just the first item in the sequence: the http:response element. Listing 8 shows this element.

Listing 8. The http:response element
<http:response xmlns:http="http://expath.org/ns/http-client" status="200"> 
     <http:header name="Date" value="Sun, 14 Nov 2010 10:37:34 GMT"/> 
     <http:header name="Server" value="Apache"/> 
     <http:header name="Accept-Ranges" value="bytes"/> 
     <http:header name="Transfer-Encoding" value="chunked"/>
     <http:header name="Content-Type" value="text/html"/> 
     <http:body media-type="text/html"/> 
</http:response>

Now, look at this same example running in the Zorba XQuery processor. For this, simply use the Zorba XQuery processor online live demo, so you don't have to install anything. Point your browser to http://try.zorba-xquery.com, then paste the code in Listing 9 into the area provided on the web page.

Listing 9. Zorba example
import module namespace http = "http://expath.org/ns/http-client"; 
let $url := 'http://neo.jpl.nasa.gov/risk/'
let $req := <http:request href="{$url}" method="get"/>
return 
     http:send-request($req)[1]

The only change you have to make in the code is to the import module declaration, as the EXPath HTTP Client is built into Zorba. Figure 1 shows how this should look in your browser. (View a larger version of Figure 1.)

Figure 1. Trying the EXPath HTTP client example in Zorba
Screen capture of the EXPath HTTP client example in a Zorba live demo

Click Execute to run the script through an online Zorba XQuery processor. You get the same http:response as in the eXist XML database example.

This consistency across processors is not just about avoiding dreaded "vendor lock-in": It's also about spending less time learning yet another new functional signature and more time building great web applications.


HTTP tester

Now that you have the basics of the EXPath HTTP Client module, use it to build a generic web-based HTTP tester that you can use to test sending HTTP requests. In building the HTTP tester, I set a challenge of what I could build in 15 minutes of programming. Although this tester wins no awards for design, I was surprised at how easy and quick it was to create a primitive HTTP tester. I won't go into the details of the implementation: Suffice it to say that there's lots of room for improvement although it can serve as a starting point for your own investigations.

As you might expect, the heart of the tester is the http:send-request function. Everything else is just assisting you in composing the http:request element. To install the tester, upload the following files to your eXist XML database:

  • expath-http.xqm. The EXPath HTTP Client XQuery library for the eXist XML database.
  • index.xqy. The simple HTTP tester.

The HTTP tester is built to be used within the eXist XML database: To install it, upload it to the database. You can access the HTTP tester by using the eXist XML database RESTful interface and pointing your browser to index.xqy.

http://localhost:8080/exist/rest/db/expath-http/index.xqy

If all is well, you see something similar to Figure 2. (View a larger version of Figure 2.)

Figure 2. The HTTP tester
Screen capture of the HTTP tester with both an HTTP request and an HTTP response

Testing web services

Now that you have a generic tester, try out all the important HTTP verbs and define headers and content that you want to send against any web service you might find. I summarized the URL method along with the expected http:request and http:response elements:

  • MarkMail. This service provides a useful query interface to an archive of popular email lists. MarkMail exposes a variety of feeds, one of which you can access. That feed represents the list of the most recent emails coming into the archive. Atom is a good example of a well-implemented RESTful web service, and this example shows how the EXPath HTTP client works easily with it (see Listing 10).
    Listing 10. MarkMail
    Method: HTTP GET 
    URL: http://markmail.markmail.org/atom/ 
    
    <http:request xmlns:http="http://expath.org/ns/http-client" method="get" 
    href="http://markmail.markmail.org/atom/" 
    username="" password="" auth-method="" 
    send-authorization="false" 
    follow-redirect="true" timeout="30"/>
    
    <http:response xmlns:http="http://expath.org/ns/http-client" status="200">
    <http:header name="Content-Type" value="application/atom+xml"/>
    <http:header name="Date" value="Sun, 14 Nov 2010 12:00:52"/>
    <http:header name="Expires" value="Sun, 14 Nov 2010 12:57:52"/>
    <http:header name="Cache-Control" value="public"/>
    <http:header name="Server" value="MarkLogic"/>
    <http:header name="Content-Length" value="209931"/>
    <http:header name="X-Cache" value="MISS from cache-2.a.markmail.biz"/>
    <http:header name="Via" value="1.0 cache-2.a.markmail.biz:80 (squid)"/>
    <http:header name="Connection" value="close"/>
    <http:body content-type="application/atom+xml"/>
    </http:response>
  • Yahoo! YQL. Yahoo! provides a SQL-like SELECT syntax, called Yahoo! Query Language (YQL), for querying all sorts of data hosted at Yahoo!. YQL lets you query, filter, and join data across web services. You can use EXPath to directly access this data (see Listing 11).
    Listing 11. Yahoo! YQL
    Method: HTTP GET 
    URL: http://query.yahooapis.com/v1/public/yql?q=select * from 
      weather.forecast where location=90210&diagnostics=true
    
    <http:request xmlns:http="http://expath.org/ns/http-client" method="get" 
         href="http://query.yahooapis.com/v1/public/yql?
               q=select%20*%20from%20weather.
               forecast%20where%20location%3D90210&diagnostics=true" 
         username="" password="" auth-method="" send-authorization="false" 
         follow-redirect="true" timeout="30"/>
    
    <http:response xmlns:http="http://expath.org/ns/http-client" status="200">
         <http:header name="Date" value="Sun, 14 Nov 2010 13:45:51 GMT"/>
         <http:header name="P3P" value="policyref="http://info.yahoo.com/w3c/p3p.xml", 
             CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi 
             OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT 
             DEM CNT STA POL HEA PRE LOC GOV""/>
         <http:header name="Access-Control-Allow-Origin" value="*"/>
         <http:header name="Cache-Control" value="public, max-age=1199"/>
         <http:header name="Vary" value="Accept-Encoding"/>
         <http:header name="Content-Type" value="text/xml;charset=utf-8"/>
         <http:header name="Age" value="0"/>
         <http:header name="Transfer-Encoding" value="chunked"/>
         <http:header name="Connection" value="keep-alive"/>
         <http:header name="Server" value="YTS/1.19.4"/>
         <http:body content-type="text/xml;charset=utf-8"/>
    </http:response>
  • London gold price. This service provides a SOAP interface for getting current gold prices. Working with SOAP typically means that you need to add specific HTTP headers (such as SOAPaction), and this example shows how easy it is to work with SOAP from the EXPath HTTP Client module. See Listing 12.
    Listing 12. London gold price
    Method: HTTP POST 
    URL: http://www.webservicex.net/LondonGoldFix.asmx
    HEADER:  SOAPAction = http://www.webservicex.net/GetLondonGoldAndSilverFix
    BODY: <soap:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
       xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <soap:Body>
    <GetLondonGoldAndSilverFix xmlns="http://www.webservicex.net"/>
    </soap:Body>
    </soap:Envelope>
    
    <http:request xmlns:http="http://expath.org/ns/http-client" 
      method="post" href="http://www.webservicex.net/LondonGoldFix.asmx" 
    username="" password="" auth-method="" send-authorization="false" 
      follow-redirect="true" timeout="30">
    <http:header name="SOAPAction" 
      value="http://www.webservicex.net/GetLondonGoldAndSilverFix"/>
    <http:body content-type="text/xml">
    <soap:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
      xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" 
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <soap:Body>
    <GetLondonGoldAndSilverFix xmlns="http://www.webservicex.net"/>
    </soap:Body>
    </soap:Envelope>
    </http:body>
    </http:request>

Summary

As this is an introductory article, I focused on showing the basics of what is possible with the EXPath HTTP client as opposed to investigating all the dark corners lurking in the HTTP protocol. The number of optional attributes on the http:request and http:body elements might give you a hint of this complexity; in common practice, though,these attributes are rarely used.

Many XQuery and XPath processors have their own extensions for making HTTP requests. Using the EXPath HTTP Client module can make your code more portable across such processors.

It's still early in the lifetime of the EXPath effort, but with modules like the EXPath HTTP Client module already showing good promise, I recommend that you try the modules, join the mailing list, and get involved in the debate.


Download

DescriptionNameSize
Example download filesrc.zip7KB

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into XML on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=XML, SOA and web services
ArticleID=626511
ArticleTitle=Working with web services using the EXPath HTTP client
publish-date=02152011