I've posted an 'article' comparing / constrasting the two, at my wiki, here: Amazon S3 vs RESTful Collections.[Read More]
Dan Jemiolo: What kind of client generation are you looking for?
I suppose I must not have gotten around to telling Dan my horror stories of using WSDL in the early, early days of Jazz.The low point was when it once took me two working days to getthe code working again, after we made some slight changes to the WSDL.Of course, we were doing some evil, evil things, like parsing Javacode with a JDT jar in an ant task, and replacing generated codefrom the WSDL generation process. But still, code generation ofthis ilk leaves a bad taste in my mouth.
Also see Dare's issues with bananas.
The best code generation is no code generation.
And that's what we changed in Jazz. Because we already had complete control over thedata typing story (EMF),we had no problem generating the XML and JSON we wanted, completely dynamically, by reflecting over the dataused in our services. But we had to do something about the service methods themselves.
So we rolled our own client and server side stack for this.
We kept the notion of defining the operations in a web service in a Java interface,because this makes a lot of sense to do in Java. We can reflect over it to look at themethods and signatures. On the server, you can write a class to implement the interface,and that's your service implementation. The low-level server interface (ie, Servlet for Java)can figure out what service to invoke, and then call the service implementation reflectively.And on the client, you can use Proxyand friends to build an object which implements the interface by making the HTTP requeston your behalf.
(Quick aside to note that Jazz services are largely RPC styled, though there are some that are more RESTy flavored - stay tuned; they've caught the REST bug.I think the 'client library' or invocation style is largely independent of the architectural style, so I think everything I'm saying here completely holds forREST as well as RPC, and everything in between.)
By having the client and server work completely reflectively, all we had to do wasmake sure the data classes and service interfaces were the same (or at least compatible)between the client and server. Understand, "all we had to do" can be a problem; butat least we didn't have the generated code to deal with as well, nor did we have a separate build step for it that can be an agility killer.
It goes without saying that you can get by with a lot less configuration in such a situation. Reflection over Configuration. Don't Repeat Yourself.
Looking back on this, I think this was a great trade-off in the time expendedto build the stacks. For instance, we were able to tweak the system in variousways that would have been impossible to do with a code-gen built system. I suspect this is going to be the case for any medium- to large-scaled system built using a number of services. You can either lock yourself into a system under which you have very limited control, and spend your time working around it and fighting it, or you can write code customized to yourneeds and spend time tweaking as you need.
Let's get back to your original question, but tweak it a bit:What should we do to make it easier for people to build clients?
If you can reflect on your service interfaces and data dynamically in the server,then you can generate all this meta-data reflectively as well.
panacea: A remedy for all diseases, evils, or difficulties; a cure-all.
In my post "That Darned Cat! - 1",I complained about Twitter performance, and peeked at some of their HTTP headers noticing they didn'tseem to respect ETags or Last-Modified header cache validator tests.
Since posting, Twitter performance is back on track. I haven't checked, but I'm guessing they didn'tadd ETag support. :-)
A number of people seemed to read into my post that ETags are a cause of Twitter's performance problems.I'd be the first to admit that such a proposition is a bit of a stretch. ETags are no panacea, and in fact you'll obviously have to write more code to handle them correctly. Harder even, if you'reusing some kind of high level framework for your app. This isn't easy stuff.
And in general, my 20+ years of programming have taught me that your first guess at whereyour performance problems in your code are, is dead wrong. You really need to break out somediagnostic tools, or write some, to figure out where your problems are. Since I don't havethe Twitter code, I'm of course at a complete loss to guess where their problems are, when theyhave them.
ETags and Last-Modified processing is something you ought to do, if you can afford it, because it does allow for some optimization in your client / server transactions. To be clear, the optimization is that the server doesn't have to send the contentit would have sent to the client, as the client has indicated it already hasthat 'version' of it cached. There is still a round-trip to the server involved. If you'relooking for an absolute killer optimization though, you should be looking atExpires and Cache-Control headers. See Mark Nottingham's recent post"Expires vs. max-age" forsome additional information, along with the link to his caching tutorial.
Expires and friends are killer, because they allow the ultimate in client / servertransaction optimization; the transaction is optimized away completely. The client can check the expiration data, and determine that the data they havecached has not 'expired', and thus they don't need to ask the server for it at all. Unfortunately,many applications won't be able to use these headers, if their data is designed to changerapidly; eg, Twitter.
Sam Ruby also blogged about anothergreat exampleof Expires headers. How often does the Google maps image data really change?
Here's another great example, applicable to our new web 2.0-ey future.Yahoo! is hosting their YUI libraryfor any application to use directly, without copying the toolkit to their own web site.Let's peek at the headers from one of their files:
$ curl -I http://yui.yahooapis.com/2.2.2/build/reset/reset-min.cssHTTP/1.1 200 OKLast-Modified: Wed, 18 Apr 2007 17:35:45 GMTX-Yahoo-Compressed: trueContent-Type: text/cssCache-Control: max-age=313977290Expires: Tue, 02 May 2017 04:08:44 GMTDate: Mon, 21 May 2007 04:13:54 GMTConnection: keep-alive
Good stuff to know, and take advantage of if you can.
For more specific information about our essential protocol, HTTP 1.1, see RFC 2616.It's available in multiple formats, here: http://www.faqs.org/rfcs/rfc2616.html.[Read More]
A few blog posts reverberating in my mind:
I just ran across a post by Joe Gregorio today, where he's comparing WS-* RPC vs. REST, specifically talking about the factthat WS-* RPC only uses the POST request method of HTTP:
"That POST of a generic media type gives no indication if the request is safe, or idempotent, nor is there any indication of the cachability of the response."
I also just read this morning apost by Leonard Richardson on Amazon's new FPS service, commenting on the 'REST'y ness of the service:
"its 'REST' interface is about as RESTful as Flickr's and del.icio.us's 'REST' interfaces"
Note, the Flickr and del.icio.us aren't considered terribly RESTy. :-)
Joe notes a handful of things that "just using POST in HTTP" breaks.Leonard notes that FPS isn't truly RESTy, but is in fact just as non-RESTyas two other fairly popular services.
And I'm left wondering: do we really need to do everything in pure REST?
Perhaps we can identify a small set of characteristics that get us most of the benefits of REST, that would be easier to implement than going'full' REST. And just focus on those. Because apparently, it's hard to do full REST. Or to be even more pessimistic, perhaps there are disadvantages to usingREST. Why would Amazon and Yahoo not use REST?
Most of the obvious advantages with REST revolve around the GET verb. Make sure youonly put safe, idempotent actions behind the logic of your GET processing.Use ETags, or even better, cache directives, in your HTTP responses onthe GETs to allow clients to intelligently cache things.
What else? Are there really other benefits? If there are, why are peoplestill not jumping on the RESTy bandwagon?
Here's my thought on the lack of adoption of REST: it's complicatedfor anything but the GETs. Namely, the mutating verbs POST, PUT, and DELETE.Not technically, but semantically. Check therest-discuss mailing listsometime. I've been having conversations with folks here recently regardingsome of the semantics of 'updates' as well; ask 5 people their thoughts, you'llget 10 different answers.
Martin Nally had told me a few months ago that he thought people shouldbe able to do 90-95% of their services in a RESTy style, and for thoseinterfaces that didn't fit the REST mold, you could do 'em in non-RESTy style(ie, some kind of RPC-ish non-GET mutation operation), butit should cost you a $1000. I think I'm ready to lower the price to $100.
Updated on 2007/08/03 to fix a syntax error in the image element.
One of the things we discovered in Jazz was that when youstart thinking of defining your product's remoteable APIs asservices, that you find yourself in a position where youwant your server to also be a client to these services.But you obviously don't want to be making a remote HTTP call out,from your server, back to yourself. That's not verylight.
I will tell you that infinite recursive callsin a remote-to-yourself scenario are quite exciting!
Fixing the remote-to-yourself problem is simple when youknow that other service you are calling is available somewherein your current address space. Because you can just 'call'it. There are issues; like thefact that remote calls are always copying the 'parameters', but in many programming languages, passing arguments willoccur by reference. Something to watch out for, whendesigning your services around existing programming paradigmslike function/method invocations, and objects.
Now let's make the story a little more interesting. Sayyou want to call some service, which isn't in your addressspace, but is on your local machine somewhere. Do you justgo ahead and pay the cost of issuing HTTP requests, likea real client, and let your existing server infrastructurehandle it like a remote call? For such local calls, tcp/ip overhead (to localhost) is likely to be neglible,since often this won't hit your ethernet interface at all.tcp/ip stacks aren't that stupid. But there's stillthe HTTP overhead, and that can be significant.
I'm reminded of the old Network Vehicleproject I worked on eons ago. We had this same problem. A Smalltalk programcontrolling all the devices. And a Java program running the user interface.Same box. Needed to chat. This was 1997, and while using HTTP wouldn'thave been out of the question, it was clearly overkill. So, we developed ourown 'protocol' over tcp/ip. Java could send us messages like:
and Smalltalk would receive the message and dial the phone. Smalltalk mightsend a message to Java, in reference to a voice recognition event:
Really, it was that easy. Of course, we had a finite set of messages tosend, so coming up with a simple protocol was quite feasible.
But it makes me wonder if what sort of economies might be available byspecifically avoiding HTTP, in the cases where it's not buying you verymuch to begin with.
As atonement for my sins,I've started a new mini-project calledgRESTbook.The idea is to actually stop spouting off about REST, and just waving the "it's easy and it's cacheable!" flag,and writes some code. An example. Of a really simple service, but complex enough to show all theinteresting generic challenges. Remember web Guestbooks? That's even easier than the old todo listthat we usually used to use as an example app.
I've been meaning to do this for a while. Actually, what I really want to play with is stuff like this (liftedfrom the project page):
But you have to crawl before you can walk.
And I also want to play with the client end as well, since there areplenty of unknownsin that space as well. Currently planning on using Flex for the client.
I'll sporadically post back here as significant function becomes available ...[Read More]
So if you're writing (or generating) contract/interface-level code which can't late-bind to all resources, everywhere, you're not doing REST ...
Is this "We don't need no stinkin contracts!" meme a reaction to the non-web-friendlyWS-* world, what with it's overly complex and verbose schemas? Because I think there'splenty of room for some people to apply contracts to parts of the web. I certainlydon't believe the entire web can be fully described using some all-encompassing schema language; but smallpieces? Sure.
I guess what I don't understand, is how you are supposed to describe your servicesto someone without some kind of meta-data describing the service. Every 'web api'I've ever seen has human readable text describingthe URIs, the data expected as input, and the data expected as output. (Admittedly, most of these 'web api's violate best practice HTTP principles somehow, but I think that's notan issue here; they could all be refactored to be be better HTTP citizens.) That human readable text is a contract; an interface. In English. Which is terrible. I'd rather have a machine readbleversion of that contract, so I can generate something human readable from it.And perhaps a validator. And some client stubs. Maybe some some test cases.Diagnostic tools. Etc.
What is the the alternative to describing your services?How is anyone going to write code to use these services, if they don't know whereto send requests, what verbs to use, what data to send, and what kind of data to expect?Instead of Flickr producing a description of their web services like this, they're simply supposedto say "Flickr is now fully REST-enabled. Start here, and have fun!" ??
As with data modelling,I don't feel like there is a single answer to what schema or contract languagebe used. I'm not initially sold on WADL (seems too verbose), and certainly wouldn't use it if therewas something else, better, for whatever project I was working on. The shape ofthe schema language isn't important, as long as it works for you.
So I guess contract-driven HTTP interfaces aren't REST. But this is an areaI'm interested in; what name should I use, so I can avoid be labelled as "not doing REST" while I'm optimizing my use of the web by being a goodHTTP citizen?[Read More]
Some thoughts on JSR 311 - JAX-RS: The Java™ API for RESTful Web Services.The description of the JSR, from the web page is:"This JSR will develop an API for providing support for RESTful (Representational State Transfer) Web Services in the Java Platform."
Here are a few related links:
Some of the positive aspects of the ongoing effort:
OK, now for the bad news.
The licensing issue is bad enough that it's really going to force an alternateimplementation to be developed. This might be something thatApache would typically do, but given theApache / Sun disagreement on JCP issues,it's not really clear to me that Apache will ever be interested in working onJSR implementations again.
Another interesting wrench to throw into the gears is Eclipse. As of Eclipse 3.3,the Equinox projecthas been shipping Jetty,along with some goop to allow you to use OSGi within servlets, orstart an HTTP server as an OSGi service.Adding RESTful service support to this story seems like a logicalnext step to me. Note that the existing JSR 311 non-goal ofsupport for <= Java 5 support violates an OSGi constraint ofrunning in their smaller (1.4.x-ish) environments.
Seems to me, if we're going to have to work on an alternate implementationanyway (to solve the licensing issues), we might as well solve some ofthe technical problems as well (J2ME / Java 1.4 support, service descriptions,client apis, etc).
Anyone up for a "Battle of the RESTful frameworks"?
When people think about links, with regard to tying together information on the web, the usual thoughts are of URLs. Either absolute URLs, or a URL relative tosome base (either implicitly the URL of the resource that contains the link,or explicitly via some kind of xml:base-like annotation).
But I wrestle with this.
Here's one issue. Let's say I have multiple representations of my resourcesavailable; today you see this typically as services exposing data as either JSONor XML. If that representation includes a link to other data that can be exposed as either JSON or XML, do you express that link as some kind of "platonic URL"? Or if you are doing content-negotiation via'file name extension' sort of processing, does your JSON link point to a.json URL, but your XML link point to a .xml URL?
Seea discussion in the JSR 311 mailing listfor some thoughts on this; I stole the term "platonic URL" from this discussion.
The godfather had something interesting to say in a recent presentation. In"The Rest of REST",on slide 22, Roy Fielding writes:
Hypertext does not need to be HTML on a browser
Where's the URL? Perhaps tying links to URLs is a constraint we can relax. Consider,as a complementary alternative, that just a piece of data could be considered a link.
Here's an example: let's say I have a banking system with a resource representinga person, that has a set of accounts associated with it. I might typically representthe location of the account resources as a URL. But if I happen to know, a priori,the layout of the URLs, I could just provide an account number (assuming that's thekey). With the account number, and knowledge of how to construct a URL toan account given that information (and perhaps MORE information), the URL to theaccount can easily be constructed.
The up-side is that the server doesn't have to calculate the URL, if all they have is theaccount number. They just provide the account number. The notion of content-type-specificURLs goes away; there is only the account number. The resources on the server can bea bit more independent of themselves; they don't have to know where the resourceactually resides, just to generate the URL.
Code-wise, on the server, this is nice. There's always some kind of translationstep on the server that's pulling your URLs apart, figuring out what kind of resourceyou're going after, and then invoking some code to process the request. "Routing". For that codeto also know how to generate URLs going back to other resources, means the code needsthe reverse information.
The down-side, of course, is that you can't use a dumb client anymore; yourclient now needs to know things like how to get to an account given just anaccount number.
And just generally, why put more work on the client, when you can do it onthe server? Well, server performance is something we're always trying to optimize -why NOT foist the work back to the client?
I realize that's a bad example for me to use; me being the guy who thinks browsers area relatively terrible application client, but what the heck; that's the way thingsare today.
For folks who just want the data, and not the client code, because they havetheir own client code, well, they'll need some kind of description of how everything'slaid out; the data within a resource representation, and the locations of theresources themselves. But the server already knows all that information, and couldeasily provide it in one of several formats (human- and machine-readable).
As an proof point of all of this, consider Google Maps. Think about how the maptiles are being downloaded, and how they might be being referenced as "links". Do youthink that when Google Maps first displays a page, all the map tiles for that first map vieware sent down as URLs? Think about what happens when you scroll the map area, and newtiles need to be downloaded. Does the client send a request to the server asking for theURLs for the new tiles? Or maybe those URLs were even sent down as part of the originalrequest.
For Google maps, it's easy to imagine programmatically generating the list oftiles based on location, map zoom level, and map window size. Assuming the tilesare all accessible via URLs that include location and zoom level somewhere in the URL.In that case, the client code for calculating the URLS of the tiles needed is just a math problem.Why make it more complex than that?
I think there are problem domains where dealing with 'links' as just data, insteadof explicit URLs make sense, as outlined with Google Maps. Remember what Roy wrote in his presentation:"machines can follow links when they understand the data format and relationship types".Of course, there's plenty of good reason to use continue to use URLs for links as well, especially with dumb-ish clients.
w/r/t the GPL vs CDDL licensing issue; sorry, I'm sure I saw CDDL on some of the pages,I should have included it in the blog post. I frankly am not all that familiar with theCDDL, but Dave Johnson got me curious about something, and sure enough, it looks like Apache allows CDDL licensed binary software to be shipped with Apache projects.Note that the link above is a proposal, but it sounds like Apache is already doing this in somecases with CDDL. Very cool.
As for the other issues Marc raised; it's silly to debate via blog posts; looks like firstname.lastname@example.org be the most appropriate place to have the discussion.
But I can't resist making two notes.
From Marc: "On the Java ME side, the fact that 311 is primarily a server side API ... means it is less suitable for the ME environment anyway."
To quote from the Java ME Sun Products overview,"As CDC is not limited to just the wireless market, and expands into other emerging markets like digital television, telematics, multi-function printers, ...".Those sound like platforms that could make use of a server supporting RESTful web services.
From Marc: "Jersey will automatically produce a WADL description"
Wonderful. The point isn't even necessarily that you should make this part of the spec,but to ensure that this is do-able within the current spec's design. Because you don't want to find out later, after the spec has been finalized, that the design preventssomething like generating a description. Sounds like you're doing that with the WADLgeneration. I'd like to see the same done with the client.
Dan Jemiolo posted today about his restdoc tool,which produces Gregorio Tablesfrom comments in REST service implementations designed to be runin Project Zero.He also posted to the Project Zero forum, and included some screen shots of the output of his tool, here.
Here is what's cool about this:
Now, for some rocks:
Finally, not to knock Dan at all, but I really have to wonder what'sgoing on if the best we can do is describe our services in human readable tables of text. Really? I mean, can't these multi-core,multi-gigahertz computers of ours really help out anymore thanby rendering some HTML to a display that we read while hand-coding our HTTP invocations?
That just isn't right.
Duncan Cragg's Business Functions | The REST Dialoguesjust blew my mind.
Here I was, all this time, thinking myself a RESTafarian. I had however, always believed there wasa place for RPC-styled invocations. This is largely, no doubt, from the work I did on my previousproject, Jazz, where we started with RPC-styled web services, and then added some REST ones. In mymind, I've been thinking all along that REST and RPC-styled services, together, are what it takesto feasibly build a complex web services based system.
But something about Duncan's arguments rings true with me.
One thing does cause me concern though. And that is dealing with concurrency of 'objects' on the server. Once your individual POST requests end up having to hit multiple objects, locking them, you needto start to be concerned about deadlocking. Not that this wasn't a concern before, but it's stilla concern here, though Duncan seems to brush it aside too casually.
I'm not sure I believe everything can be boiled down to state-transition tables, but certainlyit would be nice if they could. Makes me wonder if things like BPEL (well, a non-XML version thereof :-)make more sense than they otherwise seem to, to me, today.
My most interesting thought though, was what this means to programmers, in terms of the APIs they use to invoke these services and get the results back. RPC maps wonderfully to function/method callpatterns. Do we really end up living in a world where every web service has the same set of methodnames available (but different typing for POST parameters, and POST and GET results)? hmmm. How else might we able to make the protocol interaction more palatable to ordinary programmers used todefining, implementing, and calling methods and functions that are more RPC-ish? Or is a functional programming style, that this interaction pattern seems best suited for, just the obvious way to do it?
Anyhoo, got some back-reading to do this week ...
As an aside, I also love the dialog style of argument that Duncan used. Not sure why, maybe it's easier to read thanstraight through narrative. Steve O'Grady also often does this,and I've tried my hand at it as well.
Count me as someone who wants some typing in the REST world, based on the arguments made in thepost by Aristotle Pagaltzislast week.
We're talking about contracts here. Contracts need to be formalized, somehow.English is not the best language to use, especially since we have much more preciselanguages available to us.
My thoughts here are really just an extension to my thoughts on data serialization.Services are just the next level of thing that need to be meta-described.
Several folks have pointed out WADL (Web Application Description Language)as a potential answer, but it has at least one hole:it doesn't have a way of describing non-XML data usedas input or output. For example, JSON. It certainly is simpler and more direct than WSDL, so it does have that going for it.
All in all, good thoughts all around, but we have more work to do,more proof to provide. And by more work, I don't mean getting a handful ofexperts in a smoky back room mandating what the formats are going to be.In fact, I'm not so sure we need a single 'format'. If you've creating somekind of machine-readable schema to describe your data and your services,you're way ahead of the game.
In any case, don't wait for WADL to be finshed before starting to build outschema for your services. Use WADL if you can, use something else (hopefully simpler)if it's more appropriate for you.Read More]
Joe Gregorio answered some questions about WADL in his post"Do we need WADL?".Also note that Leonard Richardson has chimed inrecently on the WADL issue.And I of course have some different thoughts. :-)
Quotes from Joe inbold italic, mine in plain.
If I describe an Atom Syndication Feed in WADL, how close will the generated code be to a feed reader like Bloglines? Or even to a library like Abdera? If I write a really good WADL for (X)HTML how close will the generated code be to a web browser? The point is that generated code stubs are so far from a completed consumer of a web service that the utility seems questionable.
I don't think anyone is expecting a magic machine to appear that eats WADL and generates applications out theother side. At best you're going to get some code stubs. Which is whatWSDL does today. And functionally works. I'm not saying it's nice, or pretty; just that it functionally works. It is easier than writing allthe SOAP glorp yourself, so I would say such a scheme has a lot of value.
Code generation of data bindings from XML Schema, though, seems fraught with problems.You can either design a nice document, in which case the resulting codewill be 'ugly', or you can design nice objects and your XML schema will be be 'ugly'. That's why I'm interested in JSON; perhaps we can have niceobjects AND
Yes, people want to describe interfaces, and those descriptions are brittle. If I download a WADL and compile my client today it will break tomorrow when you change the service. If, instead, you use hypertext, link following and request construction based on the hypertext and client state, then the interface won't break when you change servers, or URI structures.
I think of interfaces described here in the same way as Java interfaces.Namely, it's an external description of the system that people interact with.The guts can change, the interface can remain the same. One of the nicefeatures of Java, if you're in the 'binding contracts' business (ie, youuse Java). So, no, just because the service changes, does not imply thatthe client breaks.
But even beyond that, there's no reason someone can't deploy a new interfacefor a service and leave older interfaces still working. Some peoplecreate new versions of their service interfaces every few weeksand support each version for about a year.
Code gen is brittle,and I generally dislike it. But some languages don't requireany code-gen, like PHP's SOAP support. Just give it a WSDL,it provides an interface to make calls against, as long as youcan figure out what the methods and data are. Even for Java,there a minimizations that can be made; for instance, usingdynamic Proxy's against generated Java Interfaces could leaveyou with a story of just having to generate Java Interface'stubs', the rest being all handled dynamically.
And of course it would be useful to mention non code-gen uses; even if WADL were totally uselessas a code gen device, it might still be handy to have as adocumentation format for someone's services. It could alsobe used to provide validation for the client and server.
... you can't expect me to believe that if you had a carefully crafted WADL you could hand it to WADL2Java and out would pop flickrfs.
Again, of course we're not expecting fully formed apps or filesystems to pop out of a schema-2-language grinder (thoughI'm curious about what it would mean to plug APP into FUSE).But perhaps something like the "API Kits" listed here?Absolutely! That's what I'm talking about!
Q: You don't expect everything to be built with APP, do you?
Paraphrasing Joe's answer: Not everything, but a lot.
I'm leery of the blog-based legacy of APP. For instance, thesecond-class nature of binary resources. Also, attributes oncollections and entries such as categories,title, etc. A lot of human readable, textual attributes.The kind of stuff you see in ... what is that word again ...oh yeah ...hypermedia.
I'm leery of APP, but I'm hopeful. It's especially niceto think of someone creating all the infrastructure for theCRUD-like interfaces, especially being able to handleHTTP cache validators (I hope). In the end though,you still need to describe the non-Atomdata you are transferring over APP.