Bob Balaban's Blog

     
    alt

    Bob Balaban

     

    Talking to the Cloud: Web Services, XML and APIs

    Bob Balaban  July 27 2008 11:21:52 AM
    Greetings, Geeks!

    Welcome to part 3 of what may well turn into a continuing series on "cloud computing" (the other posts are here, and here). Depends, I guess, on how much interest there is, and on how much there is to say about it.

    I've been spending a bit of time lately using the Google "APIs" to query and populate Gmail and Google Apps accounts, so a lot of what I'm going to say here references Google. That does not mean that I'm writing a "review" of Google's interfaces or capabilities, and it does not mean that there aren't other companies that do similar things. I'm focusing (some) on Google because I now know it fairly well, and because I think the technology behind what they do is interesting, with broad applicability. Okay? Yes, it's certainly true that Amazon.com and Microsoft (and others, I'm sure) have similar capabilities. Maybe sometime, when I know more about those than I do today, I'll write that article.

    So, how is "talking" to the "cloud" (meaning, performing authentication, querying for info, posting new/modified info, etc.) similar to, and different from, regular old Client/Server interaction? Isn't it the same, just with more servers? The answer is yes ... and no. (I'll refer to "communication with the cloud" -- regardless of whose cloud we're talking about -- as "cloud talking". Saves typing).

    The basic client/server distinction is still there, of course. You (the developer/user) get to (have to) write the "client" code, which communicates to some server, somewhere, by using the Internet and HTTP as the transport protocol. What you send over the wire is nothing new, either, it's the usual set of HTTP headers, plus a data (query, update, delete, read...) "payload" consisting of XML text (mainly). The "answer" to your action likewise comes back over HTTP as XML data.

    So, it's a lot like Domino URLs, then, right? Yes ... and no.
    So, it sounds a lot like Web Services using SOAP, right? Sort of ... but not exactly.
    So, it's like using JavaScript in a browser app to do AJAX, right? Hmm, again I'd say yes ... and no.
    So, it's a lot like Lotus Connections APIs, right? Yes ... and no.

    Let's look at each of these other technologies in a little more detail to see how they're the same and yet different.

    Domino URLs.
    The Domino HTTP server responds to certain URL queries by sending back data of various kinds. In that sense it is like cloud talking, with a couple of differences.

    Domino URLs (today, anyway) are always HTTP GET operations. You specify what it is you want to do via a combination of the specific URL command (the thing following the "?" in the URL, e.g., "OpenDocument", or "OpenAgent") plus a series of URL arguments, of the pattern &keyword=value. You get various things back, often in different formats, depending on what options you specify (HTML, JSON, XML...).

    By contrast, you talk to Google services by using HTTP in a very REST-oriented way: you can use any of the common HTTP verbs (GET, POST, PUT, DELETE...), depending on what you want to accomplish. The basic structure of the data you send is always XML, though within that "outer" structure you can transmit many kinds of things (any MIME type, basically, such as base-64 encoded file attachments on a message). The "structure" that the Google cloud wants to see is Atom. The URL you use is the location of a particular "service" implemented by the cloud you're talking to. The results you get back, are likewise an Atom feed, though the specific content will be particular to the query you executed. If you queried for a list of calendar entries between 2 dates for a certain user, the result might be a 'feed" of those entries, encoded in some specified way. If you uploaded a set of email messages to some user's GMail account, you might get back a feed where each entry in the list tells you what happened to the corresponding message you uploaded (success, parsing error, whatever).

    Here's an example of an actual Google mail-upload URI:  https://apps-apis.google.com/a/feeds/migration/2.0/default/mail/batch

    I can use either HTTP or HTTPS. The URI up to the word "default" is specified by the service I'm invoking. The Google service implements a feature where I can always say "default" for the account access designator, meaning, "use the account for which I have provided login credentials", or I can specify a particular user/domain (e.g., "gmail.com/bobzmail", for an (imaginary) account named bobzmail@gmail.com).

    In the case of an upload I would use an HTTP POST command, for a query I'd probably use a GET. There have to be various other HTTP headers there to form a valid command, to handle authentication, specify the caracter set I'm using (almost always utf-8), the MIME type of the payload  (almost always application/atom+xml), size of the payload, etc. My XML "payload" always begins with the standard "" header, followed by an Atom tag. The structure and formats (as well as what namespaces to use) are specified in the Google documentation (look
    here, then wend your way down. Some doc is only available once you've signed up as a partner).

    The documentation Google provides for their services (mail, calendar, YouTube, GoogleDocs, there are many) is a comprehensive explanation of the service options and fields you either get (query) or specify (upload) to make things happen. It's all XML. More on this later.

    Web Services/SOAP.
    If you're still reading (I hope you are), then you already know that Atom and SOAP are not the same, though they can be used to perform similar functions. Each uses URIs to direct an "operation" to a particular service, and even to a particular action in that service. SOAP says you always use POST, though, whereas Atom/REST say you use the different HTTP verbs for different actions. Furthermore, the format of the payload for a particular SOAP web service request is specified in the SOAP spec plus in the WSDL description of what that particular service/operation wants. For Google, the format is dictated by a combination of the Atom spec and the documentation of the service provider.

    JavaScript/AJAX.
    AJAX is a programming style (you could, I suppose, say "protocol") or technique that Web app developers use to fetch data from a server asynchronously (on a "background thread"), so that the app user can continue to manipulate the UI without interruption while data is being loaded. JavaScript, of course, is a programming language. Neither has any necessary relationship to Atom, XML, REST, or anything else, though of course, given the right tools, they can take advantage of those data structures and protocols. My point is that JavaScript and AJAX can be used to do cloud talking, but you have to set it up and code your app to do that. Google happens to have a JavaScript interface to most of their cloud services, more on this below.

    Lotus Connections API.
    True confession: I don't know very much about the Connections API. I have heard, though, from two reliable sources on the Connections team, that it's an XML/Atom based REST-style protocol using HTTP talking to Lotus Connections servers. In those respects, then, the Connections API is very much like the Google Apps API, though a Connections server is by itself very different from a Google cloud.

    Google "APIs".

    Why the quotes? It's because I'm an engineer, and therefore a person addicted to hair-splitting. To me (and of course reasonable hair-splitters can, and will, disagree about this), "API" is something provided as a programming language library (or several), which a developer invokes through procedural invocations. This "definition" (I made it up on the fly, clearly it's not really rigorous) does not include an Atom/XML protocol. Which is not to say that Google's (or Connections' or anyone else's) Atom/XML protocol isn't perfectly functional and wonderful, only that I don't really consider it an API.

    However,
    Google actually DOES have a ("real") API for many of their cloud services. This is only natural, logical, and convenient (for us developers who want to use it). Why? Well, because, no matter how great Atom, XML and HTTP are at what they do, they're tedious as heck to program. Make no mistake, I love XML, I use it ALL the time, but I would not want to have to constantly be writing code (C, C++, C#, LotusScript, whatever) that formats XML strings. Too much typing! Too easy to miss a single character and end up with mess.

    No. I want a nice, functional, preferably object-oriented API that I can call things on to format the XML for me. That way I can focus on what it is I want to happen, rather than on the details of where the attribute, or the tag element, or the ">" escape syntax has to go. And, evidently, I am not the only one who feels this way. That's why Google has provided what they call "client libraries" -- each library is a single language binding which provides real programming APIs that handle things like formatting Atom feeds, connecting to the Google cloud services, handling the HTTP posts and replies, and so on. Google have language bindings for their Atom protocols for C#, Java, JavaScript, Ruby and a few other languages. Not only that, they give you the source code for their API code, so you can actually debug down into it to see exactly how it's serializing your XML, and so on. Pretty cool, this is a huge convenience, because otherwise I probably would have had to write my own version of it, in order to get anything done.

    That's not the end of the story, of course. You still have to deal with the API. Because it represents the service functionality of the cloud, you have to give it data in the formats it wants (MIME for file attachments, RFC822 stuff for mail messages, ICAL for calendar entries....). Sometimes when you send a feed "up" to "the cloud", it just works -- you get back a "201 - created" result code for a new mail message that you're inserting into someone's GMail account, for example. Other times you get back the strangest stuff: "500 - internal server error", "404 - Not found", "302 - Temporarily Moved", and my favorite, "503 - Server busy, try again later". Huh. Huh?

    We're out of time for today, kids, but maybe next time I'll write more about what happens when you talk to the cloud, but the cloud doesn't talk back to you. Or, if it does, you have no idea what it's saying.

    Meanwhile, post your thoughts and comments! Happy rest-of-weekend! (Yes, pun intended, though. like most of my puns, it wasn't a very good one....)


    (Need expert application development architecture/coding help? Contact me at: bbalaban, gmail.com)
    Follow me on Twitter @LooseleafLLC
    This article ┬ęCopyright 2009 by Looseleaf Software LLC, all rights reserved. You may link to this page, but may not copy without prior approval.
    Comments
    No Comments Found