Bob Balaban's Blog

     
    alt

    Bob Balaban

     

    Huzzah! New LSX Toolkit is posted (Part 1)!!

    Bob Balaban  March 31 2009 02:13:39 AM
    Greetings, Geeks!

    I am pleased to report that a very long, and sorry saga, lasting more than 6 years, has finally come to a close! I am, of course referring to the LSX Toolkit. I got my new version of it today, the previous release having come out in 2001. That's a long time to wait, but Friends! The waiting is Over!

    Few living now remember (or care) what the brouhaha (brew? ha ha!) as all about, but I will give you a radically shortened history.

    What is the LSX Toolkit anyway?


    LSX (originally "LotusScript Extention") is an architectrue that was invented in the course of developing the LotusScript "back-end classes" for Notes version 4.0, back in 1994 or so. We (primarily me) were developing the original set of back-end classes (meaning, they would run on either the client or the server) using what we'd now call "extension points" in the LotusScript interpreter code.

    English translation:
    A developer could create a bunch of C++ classes, put them in a DLL, and have the "host product" (Notes in the initial case) load the DLL. The host would, as part of the load-and-register protocol, would provide to the DLL code a specially crafted object instance of the LotusScript interpreter. This object has methods that allow "foreign" (or "extension") code to "register" the classes and methods and properties and errors that it implements with LotusScript. From the LotusScript developer's point of view, suddenly a whole bunch of new classes were available to be used in the LS editor and runtime environments.

    The Notes back-end classes were the first ever LSX. The LSX-i-ness of that DLL (nlsxbe.dll on windows, other name prefixes and suffixes on other OS platforms) resided in the fact that without that DLL, Notes had no "native" LotusScript classes at all (beyond the ones built to implement the editor and debugger). The DLL is hard-coded to load automatically when Notes (and now Domino) starts up. Other LSXs can be dynamically loaded when needed with a "USELSX " type statement in a LS program.

    This had two very interesting and immediate impacts on the thinking of how LotusScript could be used:
      1. Since, at that time (1995) Lotus had several products which embedded LotusScript (remember SmartSuite? 1-2-3? Freelance? Wordpro? Approach?), these "desktop products" could (easily, it turned out) be made to ALSO use the Notes LSX! You could run a script in the spreadsheet that could (behind the scenes, as these were the "back-end" classes) have 1-2-3 talk directly to the Notes objects. Cool!

      2. Anyone could write ANY DLL that did, well, ANYTHING, and make it a "plug-in" to either the client or the server, exposing entirely new (and not necessarily strctly Notes-related) functionality.

    And both of these things were done. SmartSuite products learned to be able to load the Notes LSX. People actually built apps using it. More people created new LSXs to be used with Notes/Domino to do different things: connectors to outside data sources (eventually rationalized and collected into the Lotus Connector LSX, a single LotusScript API to talk to a variety of outside (non-Notes) relationsl databases. Still in use today!!

    One major problem remained: it was HARD to configure an LSX just so such that your classes and methods and constants and errors would all get registered properly. And it was HARD to set up the code so that when the LS Interpreter needed to invoke a function whose code resided in an LSX, it would happen correctly, passing through the right data structures and so on. The first few LSXs that were built were all hand-crafted, with a lot of help from those few who'd done it before.

    So the cry went out across the Lotus Rogers St. building in Cambridge, Massachusetts: "We need a TOOLKIT for this bugger! It needs to be easier! And it CAN be made easier!"

    Fortunately, management at the tiem were very responsive and supportive. A team was created at Lotus, and they architected and built the first edition of the LSX Toolkit. It was AWESOME. A thing of beauty. You operated it by creating documents in a Notes database, describinb what you wanted (classes, methods, etc). Then you pushed a button, and it generated a crapload of C++ code that created the "skeleton" of your implementation (of course, the "Wizard" as it was called knew nothing about your business logic, but it COULD create all the code you needed to build, register with LotusScript and accept callbacks for evaluation at runtime. You would then go and "flesh out the skeleton" to add the business logic.

    I think that first version came out in 1998 or 1999, only 4 years after Notes v4.0 had shipped.

    I wrote an LSX to sell as a product at my then new company (I had departed Iris Associates in 1997 to create Looseleaf Software). I had decided that it might make me some money (and be fun) to write an LSX that expanded the reach of the Lotus-owned back-end classes across more of the CAPI functionality than was currently available to LotusScript. Is there a Notes CAPI function you want that you can't (easily) access from LotusScript? Tell me, and I'll "wrap" it in a class in my "ScriptExpander" LSX, and you can buy it from me.

    I found lots, and people suggested others. I sold a bunch of copies over a few years. In fact: Here's what I'm gonna do. EVERYONE reading this blog post can HAVE (for free!) my Notes database that contains the entire product, except for the DLL itself. You can read the documentation, take a look at all the sample agents that show you how each and every one of the 100 or so LSX functions worked. Go on, take it! I said it was free (about 1.5mb zip file containing 1 NSF):
    lsxdoc.zip

    Much cool stuff in there. You want to set the replicaid of a database from LS? It's in there. You want to create a relica stub on another server (not a full replica, the way Notes -- at the time -- made you do it? It's in there. Need text fields up to 2gb in size? It's there.

    Other people wrote LSXs too. The most famous is probably Ben Langhinrich's MIDAS rich text conversion classes (http://www.geniisoft.com). Bill Buchan (http://www.hadsl.com/) wrote one for his FIRM product.

    Life was good (or at least, fun). The toolkit supported all of the client and server platforms for Notes and Domino. An updated kit came out in 2001. Some of the code that glues your LSX stuff to the LotusScript stuff and brings them together is actually delivered in the kit in source code form. I found a couple of bugs there (Reported them, of course!), but I was able to just fix them locally, 'cause I had the code. Life went on.

    As a few of us used LSXs more and more frequently, we noticed some, er, glitches that made things, um, inconvenient. Some could be worked around by tweaking the skeleton code after it got generated, other stuff (crashes in the toolkit wizard, for example), we had no source code for. A list of complaints began to accumulate, but Lotus was, um, slllloooooowwww to acknowledge that fixing these were a priority. The group that had been maintaining the kit was disbanded (ALWAYS a bad sign).

    Yet, I (and oathers) had real products out in the market depending on this technology. What could we do? It was frustrating. So. I think it was 2003, or maybe it was 2004 (I admit to being hazy about this, heck it might have been 2002) that I showed up at the "Meet the Developers" Session at Lotusphere (I think now it's called "Ask the Developers", we just always called it "Beat the Developers"). I happened to be invited to ask the first question. I mentioned that I was having a bunch of problems with the lsx toolkit, and said that, if Lotus wasn't really staffed to fix the problems, I would be happy to donate some time for free to sit in the Westford lab (where I used to work) and do it for them.

    At first I thought something might actually happen. One Vice President asked for my bizcard, and said he'd call me to set it up. He never did. The following year, I was again able to position myself to be the first one called on at "Beat the Developers" at Lotusphere. I repeated my offer. And again the following year! This continued right up to Lotusphere 2006, the first oneat which I was again an IBM employee. This time, a few other people asked if, now that Bob was back at IBM, would the kit get fixed. Damn thing was turning into a joke, and all I wanted was for it to be fixed.

    I was given permission at Lotusphere 2007 to pull a project together to begin the work. Of course things had deteriorated. Whereas the 2001 version of the kit built fine (on Windows) with Visual Studio 2003, it did not work with the 2005 version. There was an extensive list of outright bugs and a fairly long list of important enhancements. To be honest, I didn't spend a lot of time on this project. But I did convince a couple of other developers to do most of the work. We engaged with the release management team to get trhough the paperwork and legal review. At Lotusphere 2008 "Beat the Developers", I "announced" that we were finally going to ship it within a few months.

    Then, in February of 2008, I left IBM to join a Lotus Business Partner company. LSX pretty much fell off my radar. But I did notice, around early January of 2009, that the updated kit had still not shipped. EIGHT YEARS, and they were still sitting on it. Once again, for the 5th or 6th (maybe) time, I complained at "Beat the Developers, Lotusphere 2009 Edition". Brent Peters made a joke at my expense, implying that the delay had been caused by the poor quality of my code. I guess he didn't realize that virtually none of the code in the LSX Toolkit was mine.

    But! Brent also promised that the kit would be released by Feb. 1. Whoo hoo! Light-at-end-of-tunnel! AND, light-is-not-train-coming-my-way! I forgave him for the joke.

    After Lotusphere, Feb. 1 came and went. Brent blogged that the date had slipped to Feb. 15, which also came and went. Buchan blogged about the delays in a way designed (successfully) to embarrass IBM. Brent stepped up in a posted reply and said he agreed, enough was enough, he was going to make sure it got out. Soon.

    Sure enough! I read on Brent's blog last week (http://www.LotusStaffNotes.com/LotusStaffNotes/LSNblog.nsf/) that it was REALLY READY!!! (And! he dedicated it to me. How sweet!).

    BUT! I couldn't find it on the old toolkit site (http://www.notes.net). Only the old one was there. But TODAY! I got the link to the page where it REALLY lives:
    http://www14.software.ibm.com/webapp/download/preconfig.jsp?id=2006-09-14+09%3A05%3A37.162620R&S_TACT=104CBW71&S_CMP=

    Note: This is the windows-only version, unix version to come....soon. You have to register to get the download, but it's free.

    Phew. I'm exhausted. Eight years! And now it's here. I'm thinking I may just begin a project to revive and extend ScriptExpander with this baby, see how that goes. You can assume I will be blogging about that experience.

    But for now, MANY thanks to:
       Brent Peters, George Langlais, Scott Morris, Willie Arbuckle, Roberto Olivares, John Beck
    and everyone else who helped get the darn thing FINALLY out the door.

    The saga continues! This has been but the end of the beginning!

    (Need expert application development architecture/coding help? Contact me at: bbalaban, gmail.com)
    Follow me on Twitter @LooseleafLLC
    This article ©Copyright 2009 by Looseleaf Software LLC, all rights reserved. You may link to this page, but may not copy without prior approval.
    Comments

    1Jesper Kiaer  3/31/2009 4:23:14 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    A long journey has finally come to an end ! :-)

    Congrats Bob!

    brgds Jesper Kiaer

    { Link }

    2Ben Langhinrichs  3/31/2009 7:39:50 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    An excellent history and summation, Bob. Only one thing that I am sure you got wrong, and that is when the first toolkit was released. I released our Midas Rich Text LSX 1.00 on August 6, 1997 (see { Link } ) and my vague recollection is that it took at least nine months to create (more if you include all the code I had already created for my @YourCommand Toolkit which manipulated rich text, but that was before the LSX Toolkit). My best guess for the very first release is Nov-Dec 1996, which is borne out by the source copyrights I have all over which list 1996 as the start of the code. I will try to dig up an old SDK CD to see if I can pin it down more exactly.

    3Bob Balaban  3/31/2009 1:41:07 PM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    @2 - Thanks Ben. I'm pretty sure you're right about the dates. I'm sure I remember convincing Jeff Papows to fund the toolkit project before v4.0 shipped, so it's entirely possible that the first release of the LSX Toolkit came out in '96.

    Also, about the naming: Originally, LSX stood for "LotusScript Extensions". At some later date, a Java layer was added (though never widely used), so the acronym was changed to "Lotus Software Extensions".

    Now it's back to "IBM LotusScript Extensions", closer to the original, if a bit awkward. I'm not complaining.

    4Scott Morris  3/31/2009 2:59:56 PM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    One more major contributor to this toolkit is Tom Carriker of IBM. Thanks Tom!

    5Paul Gaudion  4/1/2009 8:20:28 AM  VERY generous giveaway Bob

    Bob

    You are a very generous person to give away something that you have obviously poured your heart and soul into.

    I am one very GRATEFUL recipient developer.

    Paul - Melbourne, Australia

    6Bob Balaban  4/1/2009 9:39:04 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    @4 - Thanks for the correction Scott. And now that you've "outed" Tom, Thanks Tom!

    @5 - You are very welcome, Paul. Of course, you have realized by now that the software itself it not part of that package, right? :-)

    7JP Liggett  4/1/2009 12:48:46 PM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    Thanks for sharing the background of LSX and your code nsf.

    8Rob  4/1/2009 2:45:30 PM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    Thanks for your persistence.

    Forgive me for my ignorance but can Java make use of the any LSX additions I might make?

    Would it be possible to implement an interface so web users could log in using the OpenID protocol? (Or is IBM got any plans to support OpenID? That would be better.)

    9Bob Balaban  4/1/2009 8:51:11 PM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    @8 - Hey Rob! LSX is pretty much a LotusScript thing. If what you want is a Java package that can be exposed in Notes, just write Java code, put it in a JAR, and load it! (add it to the JavaUserClasses Notes.INI variable, if you want it loaded from disk, or load it into a Java Scriptlib, or add it to an Agent). I think that's a much easier way to go. Of course, if your Java API is tied to a C library, you can use "native" methods in Java to call out to C code, but you usually need an intermediate layer (check the web for doc on how to create JNI -- Java Native Interface -- code).

    I'm not familiar with OpenID, but web authentication is a completely separate issue from LSX and APIs. If you find that OpenID is not supported (I don't know if it is or not), then you can look into writing a DSAPI filter to do custom authentication in the Domino web server. There are both doc and samples available for that.

    Good luck!

    10Scott Leis  4/6/2009 8:46:57 PM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    Nice to see this, but I hope someone can answer one question I've always had about the toolkit (but never asked before):

    Is it possible to use the toolkit (or create an LSX at all) with anything other than C++ and Visual Studio?

    Ideally, I'd like to be able to write code for an LSX in the Delphi language. Failing that, I'd rather use C++ Builder than Visual Studio.

    11Bob Balaban  4/7/2009 2:59:38 PM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    @10 - Interesting question. VC++ is prescribed (for Windows; other platforms would, of course, use compilers native to those systems) because that's the one that Lotus uses to build the kit (and Notes, too), and that's the one they test the kit with.

    That does not automatically mean that other c++ compilers won't work. But it does mean that you will have to take extra care. DLLs are finicky things, and exported entry points in C++ are senstive to many variations: struct alignment padding, calling convention (cdecl, pascal, stdcall, etc.), default int size and name mangling.

    Try it out and let us know if it works!

    12Giulio  4/12/2009 2:39:42 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    Wow bob! That blog entry is almost as long as the LSX debacle itself..LOL! Good to see your persistence has finally paid off, but this is one of the classic things that IBM do. Which is completely neglect areas of their products to the point that it's embarrassing. Would like to know why they find it hard to put a couple of resources onto it. Geez, you even offered your time for free to fix stuff, and it still didn't get traction.

    I know of some colleagues who are trying to build a complex lotus notes product (no LSX's or anything like that) but it's pushing notes to it's limits and, (no surprise), they are running into a menagerie of issues between versions of release 7 and also in release 8 of Notes.

    Mind you they are relying on the implicit UI functionality which is always trouble in my books. But they're expecting IBM to fix these consistency issues in subsequent releases..... bwahahaaha!! So, I am forwarding your story to them so they can take a reality check and perhaps re-design it before they dig themselves any deeper....

    13anon  6/15/2009 1:07:41 PM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    very cool.

    This running ok on vs2008? I am getting an LStatus definition problem with winreg.h...

    14Bob Balaban  6/18/2009 8:14:34 PM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    I have not tried the new kit with VS2008, and I believe the dev work was all done with VS2005, so it would not be a shock to learn that there are incompatibilities....

    15Ulrich Krause  12/21/2009 3:01:44 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    @13: Did you find a solution. I ran into the same problem ...

    16Ulrich Krause  12/22/2009 2:42:07 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    Found a (possible) solution

    { Link }

    17Bob Balaban  12/22/2009 4:48:03 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    @16 - Thanks for posting, Ulrich! Unfortunate that one would have to go to so much trouble.

    I don't have vs2008 installed yet. Do you know what the platform sdk's definition of LStatus is? Sometimes it's possible to work around an incompatibility like this with some definition-manipulation...

    18Ulrich Krause  12/23/2009 12:22:53 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    @Bob: The platform SDK definition of LSTATUS in line 175 of WinReg.h is:

    typedef __success(return=ERROR_SUCCESS) LONG LSTATUS

    It seems to be new to v6.0a of the SDK ( SDK WIndows 2003 SP2, because the SDK for Windows 2003 SP1 does not have this line in WinReg.h.

    I have already created a PMR with IBM.

    There are also some other issues like registering the environment, because the routine does not recognize VC2005 and VC2008 EE as installed on the computer. The vsvars file is not found and the environment setup stops.

    The sources compile and bind fine now, but the Customer.dll is always indicated as not available when I edit the LS samples.

    Hope I get this all working until may session at EntwicklerCamp2010 ;-)

    19anon (#13)  12/29/2009 10:16:23 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    @Ulrich - That machine was replaced, so I don't have the code now. I think i just re-tweaked the definition in the lsx toolkit, everything seemed to work ok.

    20Ulrich Krause  1/7/2010 2:48:21 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    Enhancement request # ADEE7ZF852 has been created for this issue

    21Bob Balaban  1/7/2010 3:01:28 PM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    @Ulrich - thanks for posting!

    22Will Barrows  9/25/2010 4:24:48 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    G'day Bob,

    Since you wrote this, the LSX Toolkit seems to have disappeared again!

    Neither link works, and I can't find it anywhere! Any clues?

    23Bob Balaban  9/25/2010 7:23:53 AM  Huzzah! New LSX Toolkit is posted (Part 1)!!

    Here's what I found out from the ever-helpful Pete Janzen (Lotus product manager for Notes/Domino appdev):

    "The LSX toolkit was put into a kit named IBM Lotus Toolkit 8.x Multilplatform English eAssembly.  You can find it by going to the download - toolkits pagehttp://www.ibm.com/developerworks/lotus/downloads/toolkits.html on developerWorks/lotus. Then click on the LSX toolkit which takes you to this page:https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?lang=en_US&source=ESD-SUPPORT where you will find the  IBM Lotus Toolkit 8.x Multilplatform English eAssembly. Select the eAssembly and click continue.  After you enter your Lotus ID, you will be taken to a page where you can dowload the LSX Toolkit.   

    We are working to revert back to the old process of not having an eAssembly for the toolkits.  "

    Thanks Pete!