Bob Balaban's Blog

     
    alt

    Bob Balaban

     

    What if.....we made the remote Java API run over HTTP?

    Bob Balaban  November 8 2007 07:22:42 AM
    Greetings, Geeks!

    I don't know if many of you have ever experimented (or even really used) the "remote" Java APIs for Domino (also known as "the Domino CORBA classes", also known as "NCSO.jar"). They are the "same" (mostly) as the "local classes" (a/k/a "Notes.jar"), in that both sets of Java implementation classes implement the same lotus.domino package of interfaces. But they are different in a few ways:

    The "local" classes are a thin layer of Java code on top of native call-outs to C/C++ entry points in a Notes/Domino DLL (LSXBE)
    As such, the local classes require that a copy of Notes or Domino be installed on the machine where your Java code runs
    The "remote" classes do not require a copy of Notes or Domino be resident locally, NCSO.jar uses a special comms module to talk to any (enabled) Domino server via a wire protocol known as IIOP (Internet Inter-ORB Protocol, part of the CORBA standard).
    Because a special wire protocol is in use for rendezvous and communication, you have to run a special Domino server task (DIIOP) for it to work
    Again, because the CORBA classes use a special wire protocol (and server task), a special TCP/IP port must be cleared between the browser (or other) client and the server. Typically, this means that the configured port must be opened through any firewalls

    Therein lies the rub: we've been hearing for several years that adoption of the Remote classes for Domino are not being used as much as they might otherwise be, because of the two extra admin steps that must be performed (always start the DIIOP task on the server, always open another port in your firewall(s)).

    So, Question 1: Is this a real problem for you?

    We've been cogitating (fancy word for "thinking") about this, and we have a possible solution: re-implement NCSO.jar to use HTTP(s) to talk to Domino instead of IIOP. There are a couple of different ways this might happen, but the net result would be that NCSO.jar would still talk remotely to a Domino server, but differently. And it would use HTTP/HTTPS, so a) no new firewall ports need be opened (we assume you probably already have 80/443 open anyway), and b) assuming you already run the HTTP task on Domino, no new server tasks need to be configured.

    So, Question 2: If your answer to Question 1 was "yes", does this solution fix that problem for you?

    Question 3: Does this possible solution raise any new concerns or issues for you?

    And, before I forget: Once again, the IBM lawyers make me say -- no promises! no commitments! not saying when, or even if!

    Thanks!
    Comments

    1Lance Spellman  11/08/2007 2:25:22 PM  What if.....we made the remote Java API run over HTTP?

    I've seen it used very rarely, and where it has, making sure the additional port could be opened hasn't seemed to make a difference.

    Personally, with Web Services and REST-style interfaces becoming more commonplace in Domino, I'm not sure I see the need.

    2Charles Robinson  11/08/2007 2:38:28 PM  What if.....we made the remote Java API run over HTTP?

    Does a blank stare with my mouth gaped open qualify as a response? I swear as I read that I heard the voice of one of the adults in a Peanuts cartoon in my head. :-p

    3Michael Bourak  11/08/2007 2:47:40 PM  What if.....we made the remote Java API run over HTTP?

    We have used it quite a lot and port opening has never been a problem when explained correctly.

    For ex, if you want to connect to a remote RDBMS from a J2EE app, you also have to ports to open...and that's not a pb. So why would a special port for Remote Access to Notes Java API would be one ? To the contrary, some customers WANT to have protocol "break" between tiers

    By the way, the "only" pb we face is the performance of the remote API in some types of operations (view iterations for ex).

    4Steve Smillie  11/08/2007 3:09:34 PM  What if.....we made the remote Java API run over HTTP?

    I agree with Mr. Bourak #3. When we tried to use it performance was the biggest problem not the port setup.

    Moving to HTTP would be create more issues for us, much like things like RSS feeds and Quickr Connectors do today. We use Siteminder for SSO and it prevents us from using these features since the client isn't really a browser and doesn't have Siteminder cookies needed. All those calls return is the HTML login page for Siteminder rather than the XML that is expected for the Feed or Connector.

    5Nathan T. Freeman  11/08/2007 3:18:48 PM  What if.....we made the remote Java API run over HTTP?

    With ATOM and REST APIs in the pipeline, this seems non-essential to me.

    6Ian Connor  11/08/2007 4:05:07 PM  What if.....we made the remote Java API run over HTTP?

    The port is not the issue so much for my clients. I always wrap what calls I need on the server using local classes and having my own URL paramater or REST interface for the remote calls. This keeps the network traffic as lean and mean as possible. The idea of sending corba objects back and forth over the network never appealed to me as I couldn't imagine it being as efficient as a single HTTP request that is directly targeted at my needs.

    Efforts like caching the SOAP stuff server side is more exciting if this can make its way into other types of agents.

    7Bill McCuistion  11/08/2007 9:11:59 PM  What if.....we made the remote Java API run over HTTP?

    Well, why not do away with all 65K ports except 80/443 and reduce the complexity of the internet to just HTTP/S?

    The CORBA/IOOP/DIIOP protocol worked exceedingly well when using Eclipse with the Notes/Java API to develop and test code against the remote Domino server. (Except that the JVM was on the client and not on the server, whch caused issues when moving from dev to stand-alone-test to production).

    From a secuity standpoint, I liked having to "turn on" the DIIOP ports when needed, and turning them off when not needed.

    On the other-hand, if you make this "magic" just work, then it likey is OK.

    OH, Hi Ian.

    8Mikkel Heisterberg  11/09/2007 2:22:56 AM  What if.....we made the remote Java API run over HTTP?

    Well it really isn't that important to me as opening the ports haven't really been an issue. Either I have have used it internally behind a firewall or have had the port protected by IP blocking as well. But of cause being able to tunnel the traffic over HTTP would be a nice option. As Nathan would probably say - shouldn't this just be a configuration option?

    If I had to choose I would rather have IBM spend its energy on getting the Java API in general up to speed than reworking the remote connection scheme.

    @5 - Nathan how does a ATOM/REST API help me? How does it remove the need to solve the problems with the Java API? The remote Java API is something completely different in my mind.

    9  11/09/2007 3:08:20 AM  What if.....we made the remote Java API run over HTTP?

    Opening a port has never been a problem for my customers. But I often meet this problem :

    in Banks, they sometime impose the protocol "break" between tiers (as said in @3) : Firewall -> Presentation -> Firewall -> Business -> Firewall -> Data.

    If you want to make a critic app, the infrastructure guys would say : use WebSphere cause Domino apps are not breakable. this is true.

    If you try using IIOP, some infrastructure guys comes up and say: "give up, perf is not good with IIOP". They are scared for their servers And once again this is true.

    In fact, perf can be improved creating a pool (org.apache.commons.pool) but serious improvement are needed.

    10Markus Köstner  11/09/2007 6:30:46 AM  What if.....we made the remote Java API run over HTTP?

    Hello,

    I´ve worked with the remote JAVA API sometimes. I had always problems with performance, not with the ports. So I would agree that the ports are less a problem than performance.

    11Richard Schwartz  11/09/2007 6:56:32 AM  What if.....we made the remote Java API run over HTTP?

    If you do it, you still need to give us control over whether or not it is enabled. I.e., without turning off ports 80/443 altogether, give us the ability to turn off the handlers on the server for the remote Java calls.

    Why? Because of Application-Specific Security. This is the unfortunate tendency of some developers to say "Since there is no Notes client access, and the IIOP port is closed, I don't need no stinking readernames fields. I can make the application secure just by controlling navigation and visibility of fields on forms." This sort of thing is not security, of course. It is a security hole just waiting to happen. But, the sad fact is that there are plenty of applications that would have their Application-Specific Security Holes exposed if you enable remote Java over HTTP.

    12Gerald Mengisen  11/09/2007 7:14:17 AM  What if.....we made the remote Java API run over HTTP?

    In the projects I was working in, it was not really an issue because the portal was talking internally to the Domino server. Domino admins however were more concerned about the load the DIIOP task generated additionally (could of course also come from a poorly programmed servlet). In that regard, having an additional DIIOP task is actually a good thing, because it is more transparent than if everything was included in the HTTP task.

    13Nathan T. Freeman  11/09/2007 3:14:11 PM  What if.....we made the remote Java API run over HTTP?

    @8 - Because if I need a HTTP-based API, then I should be able to work with Domino data transactionally through the REST API, even from Java. That's just HTTP I/O at that point, so there's a bunch of ways to get databases, indexes, documents and items.

    I'm sure there's some demand for remote instantiations of richTextNavigator classes -- just seems pretty marginal to me. I'd wager the vast majority of remote Java API deployments in the real world do little more than get view entries and document contents. I'd advocate seeing that move to HTTP-based transactions via REST instead.

    14Lance Spellman  11/09/2007 5:23:15 PM  What if.....we made the remote Java API run over HTTP?

    Shockingly...I think I'm in agreement with Nathan here. Not that there's isn't a bona-fide need for remote java api, I'm sure there are lots of good use cases. With the increasing availability of other methods of working remotely with Domino (REST and Web Services) there are fewer cases where people feeled compelled to go the remote Java API route. So don't put new and additional resources into porting it to new use method. If effort's spent in that area, spend it on performance issues and what not.

    15Erik Brooks  11/10/2007 10:52:09 AM  What if.....we made the remote Java API run over HTTP?

    1. No. If I'm going to bother distributing NSCO.jar, I'm probably going to bother to open another firewall port. Or I'm going to just do a full Notes install. I'm also going to prefer to keep tasks and protocols segregated for security and performance monitoring as others have mentioned. Moving from IIOP to HTTP would be *less* desirable, as it blurs the two types of transactions.

    2. n/a

    3. Yes. My concerns:

    - It seems like it will create a whole lot of wasted effort. I'm sure you won't be completely replacing DIIOP (since IBM rarely drops support for anything), so it will be more work for Lotus development and support going forward. If I send in an NSD crash file showing HTTP crashing and the first question I get back is "Are you using CORBA?" then I'll shoot myself.

    - There were and still are going to be certain packet and content-filtering firewalls that will botch up the data anyway, since it doesn't look anything like the usual HTTP traffic that's out there. This means installing firewall updates, patches and other administration... It will likely be a whole lot easier to simply open a separate port in the first place.

    - It would add more complexity to the HTTP task, which is already enough of a 'black box' as it is. HTTP needs other basic architectural and administrative capabilities added first (e.g. drop a specific user, the ability to kill a rampant thread that is hogging resources without having to "tell http quit", seeing how long a job on a thread has existed, etc.).

    - I would still have to load HTTP, which some admins wouldn't be running anyway. So all you're really saving those people is having to deal with the firewall port -- there still may be an extra server task involved anwyay.

    The nail in the coffin is what Nathan said -- there is virtually zero need if ATOM/REST are on the way.

    16Scott  11/12/2007 3:09:35 AM  What if.....we made the remote Java API run over HTTP?

    Never bothered with it really - so I think thats a no to question 1. Although I don't think that opening ports should be a good reason for moving diiop to within the http task - if you have good reason for using DIIOP then why not just open the ports. I kind of ended thinking that with this reasoning we'd all be trying to get all services running over the same port!

    And I'd agree with most of the above .. Web services & REST is becoming way more popular, is lighter weight and cross-platform (VB or .Net has never natively supported CORBA as far as I'm aware - although you can buy 3rd party code).

    17Ralf M Petter  11/12/2007 4:09:31 AM  What if.....we made the remote Java API run over HTTP?

    We have tried DIIOP and had no problems with the configuration or with the opening of ports on the firewall, but it is way to slow compared to local access. So i will really be happy to use DIIOP if the performance will be much improved. Second problem is that there is no 100% compatibility between local access and DIIOP. So some code will not work with DIIOP, that works perfectly with local access.

    18Kerr  11/12/2007 6:31:17 AM  What if.....we made the remote Java API run over HTTP?

    Just reiterating what others have said. I’ve used DIIOP a few times, the last of which we backed out of and went with a home grown REST style services. This was down to intolerable performance when accessing views on a 6.5 server; the bigger the view the worse the performance. It was as if db.getView() was pulling the whole index across the wire. Urrgh!

    Never had a problem getting a port opened in this scenario. It takes time and you might get to bounced around different admin departments trying to find someone that can actually do it for you, but that’s a bureaucracy rather than policy issue.

    19Steve Tigher  11/13/2007 1:14:44 PM  What if.....we made the remote Java API run over HTTP?

    Having read your post I've been doing a bit of cogitating on the issue myself. Not sure who you've been talking to over the years but in my mind the major issue(s) with the Domino Java API classes has/have never been one of getting a hole punched in the firewall but a number of other issues, namely:

    1. The initial stability of the DIIOP task (memory leaks et al) left much to be desired and in turn lots of admins were reticent to fire the task up on ANY Domino server. Of course this has significantly improved over releases however I've seen one too many 'political battles' where admins have used the firewall issue to masquerade this reticence.

    2. From an admin perspective, configuration was not a simple effort especially when trying to setup a secure/encrypted channel. Too many config options in far too many places.

    3. From a dev perspective, there were/still are far too many issues which need workarounds/intimate knowledge of the API.

    a. Performance has been a primary issue. CORBA scales but not within Domino. Having discussed things with Steve (Nikopoulos) some time back I have always designed any solutions with a basic understanding of the limitations that were in place during the initial design of the API. Here I feel better documentation/understanding of constructs such as ViewNavigators would go some way to alleviate some of the performance issues.

    b. Integration and recovery issues also deter many devs from implementing the API. Although there were workarounds that could be put in place (your DominoPool toolset was one) but again this needed to be worked into the API having to consistently check for recycled sessions should not be a developer burden.

    c. The inability to access any real RichText content was another limiting factor as was the inability to pass any user context into the API to gather info such as unread marks, ACL access etc. Yes, we could always have used dynamic session creation and/or session per user but these patterns were never going to fly in an enterprise scenario that’s expected to scale beyond 2 users per second with sub-second response times. And yes it looks like Domino 8 does address the last point but maybe a little too late.

    d. Too many nuances when developing to the API. e.g. the ‘to recycle or not recycle’ question is never far from the mind leading to a recycle any and everything approach. Another was the persistent need to ‘sync’ with the ?ReadviewEntries format returned by the HTTP task. It would be useful if the Java API had simply plugged into the ReadView* code and offered a modular call.

    Based on your ‘god-like’ status in the Domino Java API world and your initial mission statement back in April - { Link } I know that you’re fully aware of the shortcomings and have expressed a commitment to resolve. However whether this manifests itself as a ‘blank-sheet’ API, REST or NCSO API improvements, of utmost importance it that the API put in place and promoted needs to address the above issues (and others).

    20bgreen  11/13/2007 10:01:12 PM  What if.....we made the remote Java API run over HTTP?

    We only use DIIOP in the DMZ. Port 63148 is just fine. We block this port elsewhere.

    We have stand-alone Apache Tomcat servers. They use NCSO.jar and DIIOP to interact with Notes/Domino and SQL applications. It works very well, unless it's sloppy code, in which case you can easily crash the Domino server. No news there.

    21Bob Balaban  11/18/2007 4:39:17 PM  What if.....we made the remote Java API run over HTTP?

    Greetings, Geeks!

    I'll make a general comment, and then go on to reply to specific posts.

    This thread has been a great example of the value (to me, anyway) of this whole blog project. I wanted to use it to validate opinions/assumptions that have, in some cases, been kicking around the Lotus lab for years, and also to test some new ideas on the people who work with the product day to day.

    In the case of this post, specifically, I thought I knew what the problems were with the CORBA classes/DIIOP, based on some rather consistent feedback I'd received from various (IBM) sources. But y'all have convinced me that the intel I've been getting on this topic was all wrong. Apparently the real problem is NOT with firewalls and running another server task, but with (firstly) performance and (somewhat less importantly) with consistency between the behavior of the local and the remote class implementations.

    So, clearly, the hypothetical solution we crafted to address a problem that doesn't really exist would be a total waste of time! COOL! Now we know!

    Not only have we just saved ourselves (and you) a bunch of time and effort, we can now focus on fixing performance, something which is likely to have a much more beneficial impact on the product and be a much better investment for ourselves and our customers. THANKS!

    Just proves that asking, even when you think you know the answer, is a really good idea.

    @2 - Peanuts, or Dilbert? :-)

    @6 - Ian, Can you explain a bit further what you mean by "caching the SOAP stuff server side"? I'm not clear.

    @9 - Yes, I discovered the magic of pooling Session instances for remote Domino several years ago (when I was Looseleaf Software), even wrote a prodcut for it. Unfortunately, pooling is not as easy a technique to deploy as it ought to be. Maybe we need to work on that as well.

    @11 - Nice one, Rich!

    @14 - Shockingly, I think I am also in agreement with Nathan. Somebody write down the date and time!! :-)

    @15 - All valid point, Erik. There will be no need to shoot yourself.

    @17 - Ralf, can you provide me additional, and more specific, information about what performance issues you encountered? This will be very useful in helping us fix it up. Also, some details on where you found behavioral differences between local and remote would be helpful.

    @18 - I'm pretty sure dg.getView() does not download the entire index, but I haven't actually looked at the code in a while (like, 10 years), so I could be wrong.

    @19 - Thanks Steve, very helpful. Clearly better doc/examples would help with guiding people to use the remote classes more appropriately (but then the next problem will be to get people to read it....). I hope to have some interesting announcements to make regarding recycle() by Lotusphere time....but I can say no more right now. I seriously question your sudden elevation of me to "god-like status", though I appreciate the thought. I'm sure my wife would disagree, fond though she is of me. :-)

    @20 - I hope you have been scrupulous at reporting the crashes. Otherwise, how will we know about them? And if we don't know about them, how can we fix them?

    22Nick Radov  12/04/2007 2:24:56 AM  What if.....we made the remote Java API run over HTTP?

    We have experimented with DIIOP in a few limited cases but poor performance makes it generally unusable for production applications. It's just too low level and requires too much back and forth on the network. When we need to write Java applications that communicate to remote Domino servers we make sure to have Notes or Domino installed locally just so we can use the DLLs and communicate through Notes RPC (much faster).

    For the longer term we are building higher-level web services where a single request accomplishes much more work. So I think DIIOP is pretty much a dead end and not worth further development effort.

    23Ranjeeta  02/13/2008 4:30:11 AM  Can we call a Servlet from a Dll loaded on domino server (Domino C API)?

    24Paresh Shah  02/04/2009 5:32:08 PM  What if.....we made the remote Java API run over HTTP?

    What I notice using the DIIOP api's is that inspite of "recycling" the Sessions the memory usage on the "ndiiop" process dowe not seem to go down. I verified that the sessions are getting closed on the server by "tell diiop log=3".

    I would like yo know if anyone else has observed the same. The server version is 8.0.1