Bob Balaban's Blog

     
    alt

    Bob Balaban

     

    What if....server events?

    Bob Balaban  May 11 2007 08:08:40 PM
    Greetings, Geeks!

    So, here's something that Domino has never done that I was thinking about -- sending event notifications to clients.

    I was pondering our Web Services future and wondering if we should try to create an implementation of "WS-Notification".

    The basic idea is that you send a SOAP message to a participating server, registering your interest in a "topic", and provide a "callback" to let the server tell you when there's something interesting there for you to look at. Basically pub/sub using HTTP protocols and SOAP wire formats.

    The big advantage, of course, is that the client no longer has to poll the server to see if anything has changed recently, which is the way most "feed" setups work (RSS, Atom, etc.), presumably (dare I say, "In theory"?) making the system less bandwidth intensive and therefore more scalable.

    I can see a couple of problems, though:

    1) This architecture requires the client to also act as a server in some fashion, waiting to be notified. Most browser apps do not do that, as it ultimately rests on the ability of the client to create a ServerSocket instance, or even a full-blown HTTP server, waiting for a URL (or just a Socket) notification. Too heavyweight?

    2) What if there's a firewall in front of the client, or even a router box/NAT? The "call" from the server will never get through, right? (Am I wrong about this?)

    So, fellow Geeks, please discuss. Would you use this for Domino apps if it were available? Is it implementable in our lifetimes? Should we look instead at sending the notification via other transports? UDP? (No, not going to go there, too weird, even for me). SMTP? (Well, we can already do that with agents triggered on the server; maybe we just need a richer event model?)

    Thumbs up? Thumbs down? What say you?
    Comments

    1  05/10/2007 10:12:20 PM  What if....server events?

    Client side http? Yes. It doesn't have to be a full blown http server though. See ch. 15 "Distributed HTTP" in "Practical Internet Groupware" by Jon Udell.

    { Link }

    Shorter term, network IO in Lotusscript would be very valuable to me...at least HTTP. I was hoping the SOAP client would sit on top of a LS HTTP object.

    2Peter Herrmann  05/10/2007 10:47:46 PM  Comet does this

    "Comet is a programming technique that enables web servers to send data to the client without having any need for the client to request it" - wikipedia

    Comet is like long lived HTTP connects with Content-Type "text/javascript". It's a bit like keep-alives in HTTP/1.1 but from the server's context rather than the client. It was 'invented' by Alex Russell (Dojo guru) here { Link } and has been implemented now by Apache (in Tomcat), IBM (apparently in WAS (early program) and Dojo (the comtetd project). Best to start here { Link } Ajaxian also had a post here { Link } that links to a better one.

    The server side guys in the Ext community have been keen on it too and that's where I picked up on it first.

    3Tim Tripcony  05/10/2007 11:24:53 PM  What if....server events?

    Some folks have taken to calling this concept "Reverse Ajax". I can see this coming in handy for auto-refresh of views, presence awareness, etc. But mostly I can imagine it extending the capabilities of dashboard applications.

    4Mikkel Heisterberg  05/11/2007 12:12:30 AM  What if....server events?

    When you said server side events I was thinking more like being able to more easily and from external programs link into some of the events that are available in the C API. Like being notified on document creation and update in a specific database. I think that would be great but I'm would use it for other server applications and not for client applications. In my experience most of this polling you describe is done from other (non-Domino) server applications looking into Domino. Being notified instead of polling would be great. Maybe even in batches for very active databases.

    In my opinion the implementation should be transport agnostic that is should be able to happen over a variety of protocols - HTTP, SMTP, native. Maybe one should even consider using a messaging middleware instead such as Websphere MQ or simply Java topics using Java MQ? Lets use the infrastructure already in place. Using existing messaging infrastructure would also make testing easier and would allow both topics and point-to-point subscriptions.

    Great idea.

    5Andrew Price  05/11/2007 12:29:15 AM  What if....server events?

    This sounds like a very valuable idea to me.

    I know that some software (I forget which -- could be VoIP (Skype?) or IM) does a clever little trick that works with most firewalls. When the outgoing connection to the server is made the firewall sets up a connection record that server can use to route packets back to the client even after the initial conversation has ended.

    I don't think the client needs anything very heavy to get the packets back.

    HTH

    Andy

    6Erik Brooks  05/11/2007 7:54:45 AM  What if....server events?

    Definitely an interesting idea. It would make client notification very easy (dashboard updates, status "pop-up" messages, etc.)

    Doesn't the Notes Client do some of this in a proprietary fashion already? E.g. when you use the server console command "broadcast"? And also when a view is set to auto-update/refresh?

    Just thinking off the top of my head about the limitations to overcome:

    1. If the browser/client initially opens the connection and leaves it idle then that should bypass firewall restrictions. If you leave it open a long time unused, though, many firewalls will forced the connection closed.

    2. As things stand today it would be very easy to have 1000+ users' connections "hanging out" on the server, waiting for event notifications. I believe that under Domino's current thread model those threads would be scattered across dozens of server threads, making it very CPU intensive to manage them (and even more so when a notification event occurs). Domino would need to have a special "event notification" thread that would process those special connections so that other worker threads could handle traditional "give me data" requests.

    3. There are *tons* of caching/proxy servers and "Internet security" programs out there that would muck this up big-time as things exist today.

    4. Ensuring that notification works properly with Domino's clustering could be tricky (e.g. client is continuously connected to server "A" for events, server "A" fails - how can "B" know to now notify the client when an event occurs? Perhaps the event management pool would need to be replicated across the entire cluster realtime...)

    Overall, if this was implemented it would (in theory) save bandwidth, save resources on the server versus continuous polling directly from the client, and possibly make it easier to programatically notify a client of events.

    I'd have to (for now) put it waaaaaay low on the totem poll in terms of the 2 other "What ifs" that you've posed. There's nothing here that you technically *can't* do already -- it just makes things more efficient and theoretically easier to program.

    7Brian Miller  05/11/2007 9:21:55 AM  What if....server events?

    Not interesting. Please give this the lowest possible priority. We need the resources for so many other important things!

    8Nathan T. Freeman  05/11/2007 10:04:28 AM  What if....server events?

    Isn't the WS-Notification model designed more for server-to-server interactions for portals and such?

    9Karl-Henry Martinsson  05/11/2007 11:07:03 AM  What if....server events?

    I think it is a great idea. I can see one use for it directly in my environment... We have a link between the Notes client and a backend FoxPro application/server. At certain times (at month-end), the FoxPro server is shut down. This will prevent centain actions being done by the users on teh clients. If we could send oout a notification inside the Notes client, we would be able to prevent some frustrations.

    I can see many more uses, as well.

    10Markus Koestner  05/14/2007 4:24:01 AM  What if....server events?

    I think the idea is good. It is another part to complete the infrastructur. I´m with #4 - to ensure the delivery of events the implementation should be MQ like. That may not be necessary for most applications, but for some it is critical.

    My first use would be the refresh of Eclipse views, when a document has been changed / added / deleted.

    But I see a low priority for this, too.

    11Richard Collette  05/14/2007 12:27:57 PM  Absolutely, but maybe in a different way

    I've been looking into SOA quite a bit lately for my company and to enable Notes to participate, we definitely need events. But, I am hoping that it is available in an easy programmatic way that also makes it flexible, rather than just exposing a soap based subscription mechanism (although SOAP based subscription would be useful in addition to..) I guess my goals are more system to system than server to client.

    Can we have QueryCRUD and PostCRUD event calls to LotusScript/Java in real time? These should be triggered by documents replicating in from the client as well with an indicator to say it is coming in via replication. The query functions should include the ability to halt the operation (continue=false)and the delete functions should still pass the document items since other systems may need key values on the document rather than just the UNID of the document.

    As you mentioned, the "CRUD" triggered agent should be able to specify some sort of criteria as to what it is interested in (databases, forms, CRUD, formula) so that the agent isn't triggered for every single event, having to handle the criteria evaluation in much slower LS or Java rather than by the server in C.

    I would like to be able to hook into these events from the current database as well as other databases. On the one hand I may have an event handler that is db specific, yet another (perhaps an auditing application) that needs to work with events across multiple databases.

    From there we should be able to make any necessary web service calls. I am not familiar with WS-Notification at this point but having objects which represent those notifications might be helpful.

    Personally I think events are critical to keeping Notes a member of enterprise architecture. Polling based data synchronization (LEI, etc.) is just not quick and flexible enough for some projects. We have a system composed of many integrated applications. The polling based design results in confusion as to "where" the current state document lives, why it hasn't gotten from system to A to B yet, etc. not to mention the compounded delay it adds to the work flow as the information moves from system to system. Sometimes an issue is not an issue at all merely because the synchronization has not happened yet. In an event driven system, the lack of wait times reduces some of the guess work when there is an issue, and the architecture acts more like one big computer rather than a messaging system.

    Can this all be done with a custom add-in task. Probably, but how many people are successful at creating well written add-in tasks in C/C++.

    12Thomas Schulte  05/15/2007 4:44:01 AM  What if....server events?

    I think what Richard said @11 is the crucial point. Having events to keep data in sync with other systems would deliver some real value for companies which have multiple systems running.

    13Alan Bell  05/15/2007 11:42:49 AM  What if....server events?

    personally I think this is a little too delicate. It is going to struggle with NAT and other obstacles. I don't get the value of server fired events aimed at a client which can't be assumed to be reliably online. Persistent web services would be much more useful so the client could poll a server based event queue or something like that.

    14Bruce Lill  05/21/2007 10:40:25 AM  What if....server events?

    Couldn't this be easily done by using the sametime interface? It currently monitor's for messages. It would be a client bot that could initiate an event locally. After all, we just talking about sending & receiving messages and the sametime infrastructure is in place. This would be easy with the eclipse sametime client and shouldn't take a lot of effort to add to the Notes sametime client.

    15Dave Sanders  06/08/2007 7:26:34 PM  What if....server events?

    You Sir, are a man ahead of your time! While you see abstract concepts, alas, I only understand them when I see the practical utility. [<b>]This does have tremendous utility![</b>]

    Without going into the details of my own technical issues, this would bring Notes back into the enterprise infrastructure by listening to all manner of topics routed from our ESB to Notes. By making the back-end data available to our employees in Timbuktu... well dang, just tell me who I make the check out to?