The EXPath suite of specifications is an attempt at harmonizing the use of common functionality across the spectrum of XML technologies. EXPath is open to anyone to contribute to and represents an alternate approach toward defining useful standards that can positively affect how you use XML standards—in much shorter time scales (over those defined by official standards organizations).

EXPath, which is maintained by XML luminary Florent Georges, is based on the premise that XPath, out of all the XML technologies, provides a lot of the power beneath the XML stack of technologies. By providing specifications that enhance and extend XPath, it is possible to integrate new functionality in a consistent and rigorous manner, set within the context of the XPath data model.

EXPath is, first and foremost, an attempt to define functional specifications. It is then up to the community to build implementations based on these specifications.

Now is the perfect time to join the debate and contribute your views on the functionality you would like to see in the final versions of these EXPath modules. See Resources for several links to both the website and related mailing lists.

The EXPath HTTP Client module

The EXPath HTTP Client module defines a single http:send-request() function that allows you to create and send HTTP requests. The http prefix is bound to the http://expath.org/ns/http-client namespace URI. The err prefix, which defines error elements, is bound to the http://expath.org/ns/error namespace URI.

The http:send-request function is used to generate an HTTP request with any of the HTTP verbs—HTTP GET, HTTP POST, HTTP PUT, and HTTP DELETE. When the function is invoked, it sends an HTTP request to the specified server and emits the HTTP response, along with all metadata associated with the HTTP connection.

A number of different signatures are available for http:send-request: I prefer to use the following one, where you pass in an http:element that represents the HTTP request that you want to make:

http:send-request($request as element(http:request))

HTTP request

The http:request element is a wrapper around the body of the request and contains several attributes needed to send an HTTP request:

• method. HTTP get, post, put, delete, or any of the other HTTP verbs.
• href. The URI the request has to be sent to.
• status-only. How the response appears. (If it is true, only the status code and the headers are returned.)
• auth-method. Defines the Basic or Digest authorization method, used when accessing secured URLs.
• send-authorization. Must be set to True when using an auth-method.
• username. The user name used for authentication.
• password. The password used for authentication.
• override-media-type. Overrides the Content-Type header that the server returns.
• follow-redirect. Controls whether an HTTP redirect is automatically followed.
• timeout. The maximum number of seconds to wait for the server to respond.

Listing 1 shows how an http:request element is constructed and optionally can contain other elements, such as http:header elements and http:body (or, if multipart, http:multipart). You use the http:body element to send data (POST or PUT) to the server. If you send just an HTTP GET, then you omit the http:body element.

 
<http:request
     method = ncname 
     href? = uri 
     status-only? = boolean 
     username? = string 
     password? = string 
     auth-method? = string 
     send-authorization? = boolean 
     override-media-type? = string 
     follow-redirect? = boolean 
     timeout? = integer> 
     (http:header*,
     (http:multipart| http:body)?) 
</http:request>

You use the http:header element to define custom HTTP headers you want to send with. For example, to associate your HTTP request with a user agent, you can define <http:header name="User-Agent" value="My HTTP Tester/1.0"/>:

<http:header name = string value = string/> 

The http:request element can contain an http:body element, as in Listing 2. This element contains the content you want to send to a server. For example, to send an HTTP POST, you place the name=value pairs within this element.

 
<http:body
     media-type = string 
     src? = uri 
     method? = "xml" | "html" | "xhtml" | "text" | "binary" | qname-but-not-ncname 
     byte-order-mark? = "yes" | "no"
     cdata-section-elements? = qnames 
     doctype-public? = string 
     doctype-system? = string
     encoding? = string 
     escape-uri-attributes? = "yes" | "no" indent? = "yes" | "no"
     normalization-form? = "NFC" | "NFD" | "NFKC" | "NFKD" | "fully-normalized" 
        | "none" | nmtoken 
     omit-xml-declaration? = "yes" | "no" 
     standalone? = "yes" | "no" | "omit"
     suppress-indentation? = qnames 
     undeclare-prefixes? = "yes" | "no" 
     output-version? = nmtoken> any* 
</http:body>

The http:body element, used with HTTP POST or PUT, specifies attributes that define how the HTTP body is to be processed and sent:

• media-type. The MIME media type of the body part.
• src. Instead of inline, you can specify the contents of the body as another URI.
• method. The format of the body (xml, html, xhtml, text, binary).
• cdata-section-elements. Specifies elements that will be encoded as CDATA sections.
• doctype-public. Defines the body public doctype.
• doctype-system. Defines the body system doctype.
• encoding. The character encoding of the body.
• indent. Specifies whether the body is automatically indented before sending.
• omit-xml-declaration. Specifies whether to exclude the XML declaration in the body.
• suppress-indentation. Specifies elements that are not indented when the indent attribute is set to True.

If you read the EXPath HTTP specification, you might find more attributes. Be aware that some of these attributes are still under development, so I include only those that are clearly defined.

The code shown so far demonstrated all the components that make up a complete HTTP request. The HTTP protocol also provides functionality for sending files with HTTP requests. In general, HTTP treats a multipart message body no differently than any other media type, as Listing 3 shows: strictly as payload (separate HTTP request). The element for defining EXPath HTTP is the http:multipart element.

				
<http:multipart media-type = string
     boundary? = string>
     (http:header*,
     http:body)+
     </http:multipart>

As the element defines a separate HTTP request (subrequest), you use the media-type to define the media type for this request. The boundary attribute defines the boundary marker used to separate different parts of the multipart message contained within the nested http:body.

HTTP response

After the http:send-request() function is invoked, it returns a sequence, the first item being the http:response element and the following items containing the body of the HTTP response (see Listing 4).

<http:response 
status = integer 
message = string> 
(http:header*)
</http:response> 

.... HTTP Response body .... 

The status attribute indicates the HTTP RESPONSE STATUS code (for example, 200) sent back from the server. The http:response element contains HTTP response headers, multipart, and body sent back from the server.

Back to top

Getting started

The EXPath effort is mainly focused on defining specifications, not necessarily providing implementations of EXPath module functionality. Luckily, a few reference implementations of the EXPath HTTP Client are available from the EXPath HTTP Client module page for use today:

• Saxon. Installed as an EXPath package (.xar).
• eXist. Installed as an EXPath package or uses the XQuery library directly.
• Zorba. An implementation built into the Zorba XQuery processor.
• MarkLogic Server. Used as an XQuery library.
• Microsoft® .NET Framework. Provided through the myxsl.net ASP.net extension.

As the standards are still evolving, implementations can be volatile. Make sure that you check the basic operation before using any of these modules in a production setting.

An XSLT example

Put what you've learned about the EXPath HTTP Client module into action by invoking an HTTP request from within an XSLT transformation. You use the SAXON XSLT and XQuery processor from Michael Kay (Saxon version 9.2) as your XSLT processor.

To install the EXPath HTTP Client implementation, you need to use another useful EXPath module, the EXPath Packaging System, which installs the module automatically. To install the client, perform the following steps:

1. Download the EXPath Packaging module for SAXON.
2. Download the EXPath HTTP Client module for SAXON.
3. Install the EXPath Packaging module (follow the README file).
4. Using the EXPath Packaging module, deploy the EXPath HTTP Client XAR package.

Your SAXON processor should be able to access new functionality simply by defining the EXPath HTTP Client namespace and invoking the http:send-request function, as in Listing 5.

<?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:http="http://expath.org/ns/http-client" version="2.0"> <xsl:template
name="main"> <xsl:variable name="req" as="element()"> <http:request
     href="http://neo.jpl.nasa.gov/risk/" method="get"/>
</xsl:variable> <xsl:copy-of
select="http:send-request($req)[2]"/> </xsl:template>
</xsl:stylesheet>

This code makes an HTTP GET request on the URL, http://neo.jpl.nasa.gov/risk/, which incidentally is a website dedicated to assessing the impact risk from near-earth-orbiting asteroids. This HTTP GET request is equivalent to what happens when you request a web page from your browser. You could change the href attribute to any valid (and accessible) web URL.

The http:response element that is returned from the function contains the HTTP response element, which in turn contains http:header elements, followed by the actual requested web page content. Listing 6 shows the markup.

<http:response
xmlns:http="http://expath.org/ns/http-client" 
status="200"> 
<http:header name="Date" value="Sun, 14 Nov 2010 10:37:34 GMT"/> 
<http:header name="Server" value="Apache"/> <http:header 
          name="Accept-Ranges" value="bytes"/> 
<http:header name="Transfer-Encoding" value="chunked"/>
<http:header name="Content-Type" value="text/html"/> 
<http:body media-type="text/html"/> 
</http:response> 
... actual web page ....

As you are interested in returning the actual web page, you used a predicate on the http:send-request function that selects the second item in the returned sequence:

http:send-request($req)[2]

An XQuery example

Invoke your function from within the eXist XML database by referring to the EXPath HTTP Client xquery library implementation—http-client.xqm—bundled with the example download for the article.

You need to run Listing 7 in an instance of the eXist XML database loaded as an XQuery file. But don't worry if you can't install the database: I provide an "install free" method later using the Zorba XQuery processor.

import module namespace http = "http://expath.org/ns/http-client" at "http-client.xqm"; 
let $url:= 'http://neo.jpl.nasa.gov/risk/' 
let $req := <http:request href="{$url}" method="get"/> 
return 
     http:send-request($req)[1]

Importing XQuery libraries is more familiar for developers (EXPath packaging being a newer standard), and I use this method for most of the examples in the article.

Instead of returning the web page, as with the XSLT example, opt to return just the first item in the sequence: the http:response element. Listing 8 shows this element.

 
<http:response xmlns:http="http://expath.org/ns/http-client" status="200"> 
     <http:header name="Date" value="Sun, 14 Nov 2010 10:37:34 GMT"/> 
     <http:header name="Server" value="Apache"/> 
     <http:header name="Accept-Ranges" value="bytes"/> 
     <http:header name="Transfer-Encoding" value="chunked"/>
     <http:header name="Content-Type" value="text/html"/> 
     <http:body media-type="text/html"/> 
</http:response>

Now, look at this same example running in the Zorba XQuery processor. For this, simply use the Zorba XQuery processor online live demo, so you don't have to install anything. Point your browser to http://try.zorba-xquery.com, then paste the code in Listing 9 into the area provided on the web page.

import module namespace http = "http://expath.org/ns/http-client"; 
let $url := 'http://neo.jpl.nasa.gov/risk/'
let $req := <http:request href="{$url}" method="get"/>
return 
     http:send-request($req)[1]

The only change you have to make in the code is to the import module declaration, as the EXPath HTTP Client is built into Zorba. Figure 1 shows how this should look in your browser. (View a larger version of Figure 1.)

Click Execute to run the script through an online Zorba XQuery processor. You get the same http:response as in the eXist XML database example.

This consistency across processors is not just about avoiding dreaded "vendor lock-in": It's also about spending less time learning yet another new functional signature and more time building great web applications.

Back to top

HTTP tester

Now that you have the basics of the EXPath HTTP Client module, use it to build a generic web-based HTTP tester that you can use to test sending HTTP requests. In building the HTTP tester, I set a challenge of what I could build in 15 minutes of programming. Although this tester wins no awards for design, I was surprised at how easy and quick it was to create a primitive HTTP tester. I won't go into the details of the implementation: Suffice it to say that there's lots of room for improvement although it can serve as a starting point for your own investigations.

As you might expect, the heart of the tester is the http:send-request function. Everything else is just assisting you in composing the http:request element. To install the tester, upload the following files to your eXist XML database:

• expath-http.xqm. The EXPath HTTP Client XQuery library for the eXist XML database.
• index.xqy. The simple HTTP tester.

The HTTP tester is built to be used within the eXist XML database: To install it, upload it to the database. You can access the HTTP tester by using the eXist XML database RESTful interface and pointing your browser to index.xqy.

http://localhost:8080/exist/rest/db/expath-http/index.xqy

If all is well, you see something similar to Figure 2. (View a larger version of Figure 2.)

Testing web services

Now that you have a generic tester, try out all the important HTTP verbs and define headers and content that you want to send against any web service you might find. I summarized the URL method along with the expected http:request and http:response elements:

• MarkMail. This service provides a useful query interface to an archive of popular email lists. MarkMail exposes a variety of feeds, one of which you can access. That feed represents the list of the most recent emails coming into the archive. Atom is a good example of a well-implemented RESTful web service, and this example shows how the EXPath HTTP client works easily with it (see Listing 10).
  
  Listing 10. MarkMail
  

  Method: HTTP GET URL: http://markmail.markmail.org/atom/ <http:request xmlns:http="http://expath.org/ns/http-client" method="get" href="http://markmail.markmail.org/atom/" username="" password="" auth-method="" send-authorization="false" follow-redirect="true" timeout="30"/> <http:response xmlns:http="http://expath.org/ns/http-client" status="200"> <http:header name="Content-Type" value="application/atom+xml"/> <http:header name="Date" value="Sun, 14 Nov 2010 12:00:52"/> <http:header name="Expires" value="Sun, 14 Nov 2010 12:57:52"/> <http:header name="Cache-Control" value="public"/> <http:header name="Server" value="MarkLogic"/> <http:header name="Content-Length" value="209931"/> <http:header name="X-Cache" value="MISS from cache-2.a.markmail.biz"/> <http:header name="Via" value="1.0 cache-2.a.markmail.biz:80 (squid)"/> <http:header name="Connection" value="close"/> <http:body content-type="application/atom+xml"/> </http:response>

  
  
• Yahoo! YQL. Yahoo! provides a SQL-like SELECT syntax, called Yahoo! Query Language (YQL), for querying all sorts of data hosted at Yahoo!. YQL lets you query, filter, and join data across web services. You can use EXPath to directly access this data (see Listing 11).
  
  Listing 11. Yahoo! YQL
  

  Method: HTTP GET URL: http://query.yahooapis.com/v1/public/yql?q=select * from weather.forecast where location=90210&diagnostics=true <http:request xmlns:http="http://expath.org/ns/http-client" method="get" href="http://query.yahooapis.com/v1/public/yql? q=select%20*%20from%20weather. forecast%20where%20location%3D90210&diagnostics=true" username="" password="" auth-method="" send-authorization="false" follow-redirect="true" timeout="30"/> <http:response xmlns:http="http://expath.org/ns/http-client" status="200"> <http:header name="Date" value="Sun, 14 Nov 2010 13:45:51 GMT"/> <http:header name="P3P" value="policyref="http://info.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV""/> <http:header name="Access-Control-Allow-Origin" value="*"/> <http:header name="Cache-Control" value="public, max-age=1199"/> <http:header name="Vary" value="Accept-Encoding"/> <http:header name="Content-Type" value="text/xml;charset=utf-8"/> <http:header name="Age" value="0"/> <http:header name="Transfer-Encoding" value="chunked"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Server" value="YTS/1.19.4"/> <http:body content-type="text/xml;charset=utf-8"/> </http:response>

  
  
• London gold price. This service provides a SOAP interface for getting current gold prices. Working with SOAP typically means that you need to add specific HTTP headers (such as SOAPaction), and this example shows how easy it is to work with SOAP from the EXPath HTTP Client module. See Listing 12.
  
  Listing 12. London gold price
  

  Method: HTTP POST URL: http://www.webservicex.net/LondonGoldFix.asmx HEADER: SOAPAction = http://www.webservicex.net/GetLondonGoldAndSilverFix BODY: <soap:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soap:Body> <GetLondonGoldAndSilverFix xmlns="http://www.webservicex.net"/> </soap:Body> </soap:Envelope> <http:request xmlns:http="http://expath.org/ns/http-client" method="post" href="http://www.webservicex.net/LondonGoldFix.asmx" username="" password="" auth-method="" send-authorization="false" follow-redirect="true" timeout="30"> <http:header name="SOAPAction" value="http://www.webservicex.net/GetLondonGoldAndSilverFix"/> <http:body content-type="text/xml"> <soap:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soap:Body> <GetLondonGoldAndSilverFix xmlns="http://www.webservicex.net"/> </soap:Body> </soap:Envelope> </http:body> </http:request>

  
  

Back to top

Summary

As this is an introductory article, I focused on showing the basics of what is possible with the EXPath HTTP client as opposed to investigating all the dark corners lurking in the HTTP protocol. The number of optional attributes on the http:request and http:body elements might give you a hint of this complexity; in common practice, though,these attributes are rarely used.

Many XQuery and XPath processors have their own extensions for making HTTP requests. Using the EXPath HTTP Client module can make your code more portable across such processors.

It's still early in the lifetime of the EXPath effort, but with modules like the EXPath HTTP Client module already showing good promise, I recommend that you try the modules, join the mailing list, and get involved in the debate.

Back to top

Download

Information about download methods

Resources

Learn

• EXPath: Find the EXPath suite of specifications.
  
  
• EXPath mailing lists: Discuss and contribute to EXPath through the mailing lists.
  
  
• EXQuery: Discover the EXQuery suite of specifications.
  
  
• EXProc: Check out the EXProc suite of specifications.
  
  
• Atom processing with Zorba: Read this well-written article on using the EXPath HTTP Client module in the Zorba XQuery processor.
  
  
• More articles by this author (James R. Fuller, developerWorks, June 2008-current): Read articles about XProc, XQuery, Atom XML, Firefox XUL, and other technologies.
  
  
• XML area on developerWorks: Get the resources you need to advance your skills in the XML arena.
  
  
• New to web development: Start learning about dynamic web applications and more on developerWorks.
  
  
• My developerWorks: Personalize your developerWorks experience.
  
  
• IBM XML certification: Find out how you can become an IBM-Certified Developer in XML and related technologies.
  
  
• XML technical library: See the developerWorks XML Zone for a wide range of technical articles and tips, tutorials, standards, and IBM Redbooks. Also, read more XML tips.
  
  
• developerWorks technical events and webcasts: Stay current with technology in these sessions.
  
  
• developerWorks on Twitter: Join today to follow developerWorks tweets.
  
  
• developerWorks podcasts: Listen to interesting interviews and discussions for software developers.
  
  
• developerWorks on-demand demos: Watch demos ranging from product installation and setup for beginners to advanced functionality for experienced developers.
  
  

Get products and technologies

• EXPath HTTP Client module: Download and learn more about EXPath HTTP Client module implementations, examples. and the specification.
  
  
• Zorba XQuery Processor online demo: Play in the online query sandbox, and try out XQuery.
  
  
• EXPath HTTP Client module XSD (Contributed by Marcus Pilman): Try an XSD schema for the elements in the http:* namespace (request, response, and others).
  
  
• Microsoft .NET EXPath HTTP client implementation: Get the myxsl.net ASP.NET extension, which includes the EXPath HTTP client.
  
  
• IBM product evaluation versions: Download or explore the online trials in the IBM SOA Sandbox and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
  
  

Discuss

• Yahoo! Groups related to XML: Join the discussions.
  
  
• XML zone discussion forums: Participate in any of several XML-related discussions.
  
  
• The developerWorks community: Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.
  
  

About the author

James Fuller has been a professional developer for more than 15 years, working with several software blue-chip companies in both his native USA and the UK. He has co-written a few technology-related books and regularly speaks and writes articles focusing on XML technologies. He is a founding committee member for XML Pragueand was in the gang responsible for EXSLT. He spends most of his time working with XML databases and XQuery. You can reach James at jim.fuller@webcomposite.com.

Rate this article

Comments

Back to top

Table of contents

• The EXPath HTTP Client module
• Getting started
• HTTP tester
• Summary
• Download
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into SOA and Web services on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers