 Bob Balaban | Bob Balaban September 29 2008 09:15:07 AMGreetings, Geeks! Have you heard? Lotusphere09 is coming! (in 4 months...). If you wanted to submit an abstract or 3 so you could get the free ticket (value of almost $1900) or so you could work on your campaign to become a Lotusphere rock star, it's too late, the deadline was last Friday. Will I be going? Freakin' A! So far I've been to all 15, and I'm not breaking my streak now. That's not the only reason I'm going, of course. However, there was a time several years ago (2001) when I was seriously thinking I wouldn't attend. Really seriously thinking. I went and dug up the article I wrote about that at the time, and I'm posting it here for your reading pleasure (the magazine I wrote it for no longer exists, so there's no copyright issue). I have edited out a bunch of complaints about the registration and hotel reservation process, mainly because that's gotten a whole lot better since '01. I also added some [editorial comments, like this]. While I wouldn't write the same article today (lots of things have changed since 2001), it's fun to look back and remember where things were at back then.
"To Go, Or Not To Go: That Was the Question" Bob Balaban, January, 2001 To be completely honest about it, I was really on the fence about whether to attend Lotusphere 2001 for a long time. I had previously attended all seven Lotuspheres, the first four as a Lotus/Iris employee and speaker, and the succeeding three as a business partner. I wasn't especially looking forward to the eighth one this year, and my registration experience back in October had, to put it mildly, sucked. I began to question the certainty of my attendance, and wondered if I were alone in those feelings of doubt. So, in anticipation of writing this article, I made some notes on why I did and didn't want to go, and polled some of my friends and colleagues to see whether they were going, and if so why, if not, why not. Much of what I heard was expected, some of it was surprising. Now that Lotusphere 2001 is over (and, to steal my own thunder, I did go), I've polled some other friends and colleagues who also attended, to see how they felt about the conference: were they glad they went, or sorry? This article is a summary of my feelings, both pro and con, about attending Lotusphere, and some thoughts and observations from a selection of other people, some of whom attended and some of whom chose not to go this year. "It's Insanity"
There were certainly many reasons to stay home this year. Attending Lotusphere is expensive. The conference registration and the hotel bills alone are around $3000 [WAY more than that now, in 2008; bob], and while the plane fare is relatively cheap (at least, from Boston it is), the revenue lost from spending 4 or 5 days in Orlando instead of spending that time doing billable work for my customers is substantial.
And then, there's the hassle and aggravation. It started for me this year with the conference registration. I knew in advance that registration would open at 9am Eastern time on Tuesday, October 3. But I was teaching a course all day that day (ironically, at Lotus), and didn't get to log in to the registration site until 5pm, by which time the conference was sold out, as was every listed conference hotel. I added myself to every possible wait list, and put off making plane reservations. [Remember when Lotusphere sold out within hours? I'm not sorry that's not happening anymore...; bob]
I was bummed, but as I began contemplating January without Lotusphere, I actually started looking forward to not going. Let's face it: the thrill of being at Disney World wore off for me after the third trip. While it would still be fun for me to spend that extra couple of days there with my kids, they're now too old to miss school for a week. Myself, I'm thoroughly sick of the Disney/Lotusphere experience: big crowds, long walks between hotels several times a day, boring food in big, noisy tents, long lines, overflow rooms, and spending 15 minutes just trying to get from one side of the showcase to the other.
If I'd been given a session to do, I wouldn't have given it a second thought: the free registration and exposure for me and my business would have made the decision a no-brainer. But that didn't happen. I didn't have any customers going that I couldn't see somewhere else more conveniently. I wasn't desperate to garner any particular technical info or business coaching, or to rub shoulders and press the flesh with Lotus and Iris people.
And there's the hassle.
A week later I was notified that I could have a conference ticket, and because I'd provided my credit card number at wait-list time, they went ahead and charged me. My anticipation of staying home in January was short lived: to cancel now would mean paying the $100 penalty. Sigh.
Karen Hobert, president of Top Dog Training and Development and a six-time Lotusphere attendee said, "It's insanity. It should have been moved to a bigger location many years ago. Why do I have to get up at 6am or else get locked out of sessions? I'm going on vacation. The downtime has already been built in to my schedule, and if I'm going to spend that kind of money, I'd rather relax." Karen also made the point that for the technically oriented, smaller business partner, the show has become less relevant. I agree with that assessment: much of the really technical content is now reserved for the DevCon (where the crowds are a lot smaller), and the business oriented sessions on raising venture capital, hiring employees and developing sales techniques are not particularly relevant to small businesses who focus primarily on consulting and services. [Note: Lotus stopped doing DevCons a long time ago, and the technical content at Lotusphere is definitely back up there; bob]
There did seem to be a general sense of gloom among many of the people I discussed things with. There were rumors of layoffs at Lotus, further IBM-ification of the company and the product, and dejection over a perceived lack of commitment by Lotus to the Notes Client.
It wasn't looking good for Lotusphere. "It's the People, Stupid"
So I went anyway. I did all the usual stuff: picked the sessions I thought I'd get the most out of, cruised the showcase, went to some parties and other get-togethers (skipped the Wednesday night bash for the fourth year in a row, and was not a bit sorry), hung out with people I don't get to see very often.
I asked around among my friends and colleagues to see if they were glad they had attended. The results of my informal poll (no, don't even think of asking for a recount) were a little surprising: most of the people I asked were happy they'd made the trip, despite the attendant hassle and aggravation.
Gabriella Davis, of The Turtle Partnership, said, "It was my best Lotusphere in years." Wayne Scarano, of SGA Business Systems, was in general agreement, though he would have liked RNext to be "more in the forefront". [In 2001, "RNext" was the code name for what later became Version 6; bob]
Most said that the main reason they attend Lotusphere is not for the technical information, but to get an idea of Lotus' product direction and strategy, and to network with other attendees. And that assessment is not exclusive to people from large companies. Richard Schwartz, president of RHS Consulting and an 8-time Lotusphere attendee, said "I go back every year to renew relationships and form new ones, and this is the key value for me. Access to Iris developers and Lotus product managers is invaluable to me. The networking opportunities with customers and other business partners are just as important, too. There is no single event during the year that gives me even half the networking opportunities that Lotusphere does."
Some (like me) also went to try to sign new customers. The expense is usually more than repaid if you can engage even a single new client. Of course that's a hit or miss proposition, and you can't count on it. Others who had been morose about perceived negative trends at Lotus/IBM were reassured by the Business Development Day and other sessions. Dick Gill, of Gill & Piette, Inc., said "Al Zollar is a steady hand on the tiller. The IBM presence was thankfully subtle, maybe they finally understand that the audience is Lotus people." Several people expressed relief that most of the sessions would be available (at no extra cost) via webcast following the conference, thus relieving some of the strain of having to choose between competing sessions. Conclusion: Worth it, or not?
I'd say that most of the people I discussed it with who actually attended were happy they did. None of the people I discussed it with who did not attend were sorry that they didn't.
For myself, I'm still on the fence, neither glad nor sorry that I went. The expense and aggravation was mainly balanced by the networking, by a few good technical sessions and by the labs. For me and those like me (small business partner, focused on consulting rather than seat-selling and primarily technical rather than sales/marketing), the value of Lotusphere has declined relative to the geekier DevCon. On balance, though, I'd have to admit that there is still gold to be mined down there in Lake Buena Vista. Just don't make me go on the "It's A Small World" ride, ok?
[Summary: I've been to 7 Lotusphere's since 2001, 2 of them as an IBM employee. I still haven't been to the Wednesday party, but I'd still count the trip as an overall plus. It's still the only event where you get a chance to see absolutely everybody. And that's worth a lot. See you there, in 4 months!]
Visit the Binary Tree website here.
Bob Balaban September 13 2008 07:40:07 AMGreetings, Geeks! I don't often do soapbox-type posts, but this one's been building for a while. I've heard it said that most people who use Notes use it only for email (EdB: if you're reading, would you care to comment on the accuracy of that statement?). Assuming that's still true, then clearly IBM's competitive focus has to be on fighting the email wars with Microsoft and Outlook/Exchange (which is most definitely only an email system). That's where most of the revenue is coming from, after all. BUT, here's the thing: while nobody in her right mind would try to use Exchange for anything other than email, Notes/Domino is a great application platform for ALL kinds of things besides mail. Look at the list of " core strengths" people came up with over on Vowe's blog (take a look at the " core weaknesses" too, very interesting stuff). Why does this bug me? Aside from the fact that (evidently) relatively few users of the product realize what they could be doing with it, but aren't, it bugs me that the product doesn't seem to get the respect it deserves within IBM either. Sure, if most people pay you for an email system, you're going to focus sales and marketing effort around selling email systems, and you're going to focus a certain amount of development effort on making the product a better email system. Normal, heads-down, capitalistic behavior. But really, the product is a floor wax AND a dessert topping. Really. And, I claim, this really, really matters, and SHOULD matter, even to heads-down, capitalist bean-counters over at IBM. Why? Because there are lots and lots of accounts where applications (not email) are the only thing keeping Notes and Domino around. Where, if email were all it did, it would have been tossed out completely. Notice that I am not calling the email wars over. I don't have access to the numbers, and nobody (nobody reliable, anyway) is publishing any numbers that mean anything. IBM win some, Microsoft win some, I don't really care who's ahead. OK, that's not entirely true, because I have a 20-year history with Lotus and Notes, and I emotionally want it to be successful (that's why I'm writing this post, after all....) But I'm seeing it more and more -- organizations which have had Notes/Domino for a while, and (for whatever reason) who decide to make the change for email, are keeping a few Notes desktops and Domino servers around, even after they stop using the product for email. Why? Because they have applications that they either can't get rid of, or which are too expensive to re-create on another platform. Applications! NOT email. These are often messaging-enabled apps, often workflow-oriented. They use email infrastructure, but only as a service to make the application better able to do its thing. And, lest you think I'm making this up, go on over to Wild Bill's blog and see where he says:
... we've found more and more customers who use Lotus Notes for applications, and Exchange for email. Or companies (through mergers and acquisitions) that end up with mixed environments... Yeah, I'm seeing that too. Does a situation where a given customer dumps Notes email seats, but keeps a few application seats (and servers) show up on IBM's radar as a competitive "loss"? I'm sure it would if that customer went completely over to the other side, but if IBM can still count them as a customer, does it watch the actual revenue decline (as long as it doesn't go to zero) and take notice? I really don't know. Sure, I worked there for quite a while, but I was never plugged into the sales and marketing side of things (Ed? Care to comment??). One has to wonder, then how product development investments are made in this environment. Naturally, there are way more cool features and ideas and widgets and languages and @functions that people within the Lotus Notes/Domino dev teams want to do than the company can afford to fund. That's true of every software company. But, like any software company, IBM has to "score" the relative worth of these ideas and measure them against the cost of doing them and the expected benefit. And the decisions get made based on the results of the scoring exercise, however that gets done. "Yes, we'll fund putting the Notes Client on Eclipse. No, we won't fund creation of a Ruby on Rails interface to the back-end classes, at least not right now." (I'm making up the Ruby on Rails thing as an example. To my knowledge, nobody ever seriously proposed doing that, at least not while I was working there. Of course, Sam Ruby works for IBM, but not for the Lotus division....). So, one wonders, in the meetings where these decisions are made, how does the value of an investment into Notes AppDev capabilities get scored against, say, a more efficient router? Furthermore, one wonders if the decisions might not go another way if the scoring algorithm used (and believe me when I say I truly have no idea how this happens "on the ground" in the conference rooms and offices at Lotus) took more account of the value to customers (and therefore to IBM) of Notes/Domino's application-not-email capabilities. Yes, they're putting Designer onto Eclipse in v8.5 (yay!!). Yes, they're doing a bunch of other great things to beef up "appdev" in the product. But, if the scoring algorithm were a bit more cognizant of the fact that "appdev" is the ONLY thing keeping the product alive in a growing number of customer accounts, would we have maybe seen all of these things (and more!) sooner? Better? Discuss! Visit the Binary Tree website here Bob Balaban September 7 2008 06:52:48 AMGreetings, Geeks! As many of you already know, I had something to do with getting LotusScript (the language) and the Notes object model interface for LotusScript (the "back-end classes" and the LSX architecture) into Notes version 4.0, released in January, 1996. I've been a big supporter of that technology ever since, and I still am. I like it that there's a relatively simple, "scripting" type interface to the power of the product. I like that it's usable for server-only programs (primarily via agents and Web Serivices), and it's also available for "front-end" programming. I also had something to do with adding the Java version of the back-end classes to Notes/Domino release 4.6 (September, 1997). Java as a language offers some capabilities that LotusScript doesn't have (e.g., multi-threading, network access...), although Java has never been quite a tightly integrated into the product as LotusScript is (no Java front-end classes, for example, plus the need to worry about "recycling" objects). I've been thinking about the future of programmability in Notes and Domino for a while. I wrote this post on how JavaScript was likely to play a bigger role. Of course, moving the Notes Client onto an Eclipse foundation is likely to have a big impact -- Java interfaces to all sorts of Eclipse-based functionality becomes easy to add to the product. And, since it's easy to make JavaScript talk to Java, and since JavaScript is a real "scripting" language (whereas Java, I claim, is not), the role of both will inevitably become much more important to Notes application developers (assuming IBM continue to develop and support the functionality, and I have no reason to think they won't). So, where does that leave LotusScript? Clearly, they can't take it away; by now it's too ingrained, and too many people have working code in production. Removing it would be a disaster. There was talk at one time of cross-compiling LotusScript byte code (what you get when you compile LS source code) to Java byte code. Some research was done on that, but it didn't pan out, there were too many mis-matches. So, LotusScript is not dead yet. But it seems equally obvious that Java-only capabilities will continue to be added to the product (XPages is one new example). And, while interoperability between Java and LotusScript will not be completely absent, I predict that it won't be a major feature of N/D programmability in the future. LS2J never really took off, and aside from running LS Agents, XPages has very little in the way of integration with LotusScript. So, dear Geeks, I want to know what you think! Please comment on/discuss the following (or whatever else seems relevant to you): - Do you agree with my analysis? Are Java and JavaScript the wave of the future for N/D? - If you are currently a LotusScript developer, how do you feel about learning Java? - Does the Eclipse plug-in model open up new business opportunities for ISVs? If not, why not? Feedback welcome! Update Monday 8 Sept: Hey Geekdom, I've never done this before, but thought y'all ought to take a look over at Nathan Freeman's blog. He did some interesting code archeology on the new Notes 8.5 Beta 2. Very relevant to this topic! Bob Balaban August 30 2008 Greetings, Geeks! I think I am the one that invented that phrase, and I think it was back in 1999. My colleage Karen Hobert and I were touring around, doing 3-day workshops on advanced Notes/Domino development. One of the topics I presented a couple of hours worth of material on was J2EE (Java 2, Enterprise Edition, the stuff of which WebSphere Application Server and other platforms are made). Or, it might not have been until 2002, when I started speaking at Lotusphere again, and did some presentations on "Java for LotusScript Developers" and "Integrating Domino and WebSphere". In any case, the term emerged spontaneously after reciting a long list of technology features, as in: "It's got platform-portability, multi-threading, scalability and code reusability. It's object-oriented, powerful AND easy to use, network-aware, MIME compliant and stream oriented. It won't drive you to work, but it might drive you to drink. It's obviously fully buzzword enabled." Since then, I've used it many times. I've even started to see it (or one of its common variants, such as "fully buzzword compliant") used by others out in the blog-o-sphere. What does it mean? In essence, it's really an ironic comment on the tendency of technology marketers to hype product using buzzwords. I've always been a bit annoyed by the (now rather common) practice of reducing the capabilities of a complex technology down to a handful of technical terms that someone thinks has currency, or that someone thinks adds "cool" to a product, so that it will fit on the top half of a 1-page marketing brochure. I understand the desire to describe in simple terms what a product *IS* (if you can, it tends to sell better), but the buzzwords don't really tell you anything about what something *does*. It seems to be all about the hype, not about the thing being hyped. Maybe that's why I'm an engineer, not a sales guy. Does anyone want to refute my claim of invention? If so, let's hear about it! Here's another one I'm pretty sure I invented: It was only about 15 minutes after Iris first added Java language capabilities (primarily for Agents) in Notes 4.6 (Autumn, 1997) that people started saying, "LotusScript is dead". Well, HAH! (that was me, having the last laugh, because it's more than 10 years later now, and LotusScript is still alive and well. At least for a little while longer...). Java, at the time, also suffered from being over-hyped. It was going to take over the world, it was the stuff that the UI of the future would be made of, etc. etc. ad nauseum. All the hype actually got in the way of talking about Java as a bit of useful technology -- lots of developers were afraid of it (some still are). So, in order to attempt to defuse that, I started referring to Java this way: "Java is a tool, not a religion" Followed by: "If it solves more problems than it creates, use it. Otherwise, don't" Powerful, AND easy to use!  (My friend Amy Blumenfield first gave me this animated GIF, several years ago. It makes me smile) Bob Balaban August 18 2008 02:47:42 AMGreetings, Geeks! I promised you a follow up to my "Spawn of the Devil" post of a couple of days ago, and this is it. I sincerely hope this will be my last-ever discourse on the GetNth topic. I wanted to be fair, and mention that there was one time, recently, in fact, when GetNth was the right solution to a problem I was having, and I really needed it. But, to tell you that story, I have to tell you this: Back when the NotesDocumentCollection class was first created (for Notes 4.0, mid-1990's), it was essentially a LotusScript wrapper for a CAPI data structure called an IDTable. The IDTable "object" is a compressed list of NOTEIDs, which is optimized for two things: efficient storage of a potentially large number of IDs, very fast lookup to determine if a given ID is in the list or not, and ver fast traversal (if you have an ID and want to know the next one, NOT if you have to count N entries from the beginning every time....). Ok, that was three things. IDTable allowed such operations as getting a list of all the documents in a database to be very, very fast (expressed in LotusScript as NotesDatabase.AllDocuments, for example). The thing about db.AllDocuments (and the CAPI that underlies that function) is that normally it returns an IDTable that contains both actual document IDs, AND the NOTEIDs of any deletion stubs which are also in the database. Originally, the code underlying .AllDocuments stripped out the deletion stubs, leaving only actual data documents IDs, so you could be pretty sure that when iterating over a DocumentCollection, that you would always get a valid NotesDocument (assuming someone else didn't go and delete something on a shared copy of the database between the time when you executed AllDocuments and when you accessed everything in the resulting DocumentCollection). That all changed sometime in between Notes 5 and Notes 7 (I don't know exactly when, I was off doing other things...), and deletion stubs are no longer stripped out. Now when you want to iterate over all the documents in a database using NotesDatabase.AllDocuments, you have to check each NotesDocument instance with either (both is probably a better choice) the NotesDocument.IsDeleted or .IsValid properties before you try to access the document contents. Do I know the difference between the two? No, I do not. Ok, back to the story. So, there I was, needing to check all documents in a database that happened to have a lot of deleted documents in it. As always, I got a DocumentCollection using Database.AllDocuments, and wrote my NotesDocumentCollection.GetFirst/GetNext loop (I was actually using the COM classes from C#, but they're the same as the LotusScript and Java classes anyway, so that difference is meaningless for the purpsoes of this sad story). Inside the loop I coded a test so that if .IsDeleted was true or .IsValid was false, I could skip the current document and go on to the next one. HOWEVER (this is the sad part), it turned out that DocumentCollection.GetNextDocument() would fail if the current document was not "valid". (Gasp!) What to do? I thought maybe someone was messing with the database on the server while I was iterating, and that the document had been deleted out from under me. But no! The same thing happened when I ran a test on a local copy of the database. I was stuck, couldn't get the next document, ever. AHA! But then GetNth came to my rescue. Why? Because, GetNth doesn't NEED the previous document to get the next one, it just counts IDs in the IDTable. Problem solved, worked like a charm. Sure, slower. But how do you measure speed when the "correct" way of doing something runs you into a brick wall? I'll take correct behavior that's a bit slower over a "faster" approach that doesn't work, any day. (Visit the Binary Tree web site here.) Bob Balaban August 16 2008 04:22:09 AMGreetings, Geeks!! I was browsing my feed reader this morning, and came across a couple of interesting posts on the TeamStudio site (see the posts titled "Show Time" and "Time Differences"). Sigh. More discussion about NotesDocumentCollection.GetNth..... This topic has been a pimple on the butt of Notes developers for, let's see, EVER (ok, well, actually since Notes 4.0 shipped in January, 1996, so really only for 12 years....). I was moved to blog about it, one more (and I sincerely hope, LAST) time. I've written about it, talked about it at conferences and among friends, and it STILL comes up every now and then. So, ok. ONE MORE TIME! IF (you have a DocumentCollection object -- doesn't matter whether it's LotusScript, Java, COM, whatever language binding), AND (your DocumentCollection object is an UNsorted list -- i.e., the result of a Database.AllDocuments, or Database.Search operation, NOT an FTSearch) THEN iterating that collection using the DocumentCollection.GetFirst/GetNext pattern is MUCH (i.e., A LOT) faster than iterating using DocumentCollection.GetNth(index). CAVEATS: - For small collections the difference doesn't matter - I'm talking about navigation time, regardless of what happens to the documents themselves inside your iteration loop. - I'm ONLY talking about DocumentCollection navigation here, VIEW navigation is completely different (GetNth in a View is just as fast as GetFirst/GetNext) - Your results may well be version-dependent! (more on this later...) How do I know? How do i know for sure? I wrote an experiment, which I will now share with you, gentle geek readers. Here's what I did: - Using Notes/Designer v7.02, - Created a blank database - Wrote a LS agent to populate the database with 50,000 documents (see code, below) - Wrote another LS agent to iterate all the documents in the database (see code, below), first using GetFirst/Next, then using GetNth - The agent times each full iteration, and reports the elapsed time of each.
Results? The envelope, please....
- GetFirst/Next: 1 second
- GetNth: 446 seconds You can't tell me that's not a significant difference. I wondered whether someone at Lotus had managed to fix this forever problem recently, so I asked my colleague Steve Mullen to run the test on Notes 8. Steve reported results almost identical to the ones I receved ("painfully slow"). Twelve years and counting, DocumentCollection.GetNth is still.....painful.
So, What's Going on Here? Why so slow with GetNth? It's because unsorted DocumentCollections, internally, represent a data structure called an "IDTable", a compressed list of Note IDs (see the Lotus CAPI toolkit for documentation, this stuff has been around a long time). There's no way to find the Nth entry in the table other than by starting at the beginning and counting to N. So, to iterate the entire list of X entries will take (x*(x-1))/2 CAPI calls (which is Order(n-squared). In other words, for 50,000 documents, you're going to call the IDTable interface something like 1,249,975,000 times. As Kurt Vonnegut might have said, "That's a lot of times!" The shocker (to me) is that it ONLY takes 7 or 8 minutes (on my laptop). So, there you go. I think I'll write one more blog entry on this topic, as I recently came across a situation where I had no choice -- I HAD to use GetNth to iterate a document collection (first time in 12 years!). Stay tuned! Here's the code for my "PopulateDb" agent: Sub Initialize Dim s As New notessession Dim db As NotesDatabase Dim i As Long Dim doc As NotesDocument Set db = s.CurrentDatabase For i = 0 To 49999 Set doc = db.CreateDocument() doc.somefield = i doc.Save True, False Next End Sub And here's the code for my test agent: Sub Initialize Dim s As New notessession Dim db As notesdatabase Dim dc As NotesDocumentCollection Dim doc As notesdocument Dim dtStart As NotesDateTime Dim dtEnd As NotesDateTime Dim elapsed As Long Dim i As Long Set db = s.CurrentDatabase Set dc = db.AllDocuments Set dtEnd = s.CreateDateTime("") Set dtStart = s.CreateDateTime("") dtStart.SetNow Set doc = dc.GetFirstDocument While Not (doc Is Nothing) Set doc = dc.GetNextDocument(doc) Wend dtEnd.SetNow elapsed = dtEnd.TimeDifference(dtStart) Msgbox "GetFirst/Next elapsed time = " & elapsed dtStart.SetNow For i = 0 To dc.Count Set doc = dc.GetNthDocument(i) Next dtEnd.SetNow elapsed = dtEnd.TimeDifference(dtStart) Msgbox "GetNth elapsed time = " & elapsed End Sub Try it yourself! Bob Balaban August 14 2008 06:28:31 PMGreetings, Geeks! Good news (for me, anyway)! I get to attend UKLUG! The people who pay me, Binary Tree, are a sponsor, and will have a ped (or table, or booth, or however they set it up) at the show. Unfortunately they ran out of speaker slots before I could get one, so I won't be presenting a session. However, the kind/ generous/ organizers were nice enough to offer me the unique (to me, anyway) chance to host the "Gurupalooza"panel, which,of course, I immediately accepted. This should be fun.
I had another idea that is still in the brainstorming stages -- I'd love to get some feedback on it. Can't promise that it will happen, of course, but what about this:
I'd like to organize a "Speakers Roundtable" kind of get-together. The basic idea is along the lines of a Lotusphere BOF, except more collective. We'd get a bunch of the conference speakers together in a big room, each with a table to sit at. Attendees could come and sit, chat, ask questions, etc., move around, maybe talk to some people they wouldn't normally talk to.
And, since the "official" conference space is both limited and expensive, it was suggested that maybe the venue could be one of the local pubs..........
So, discuss! Good idea? Bad idea? Anyone who lives or works in the area willing to help organize it? (We'll come up with a suitable reward, I assure you).
Hope to see you there!
Bob Balaban July 31 2008 04:49:52 PMGreetings, Geeks! I am very pleased to announce that DNUG (Deutsche Notes User Group) have invited me to present 2 sessions at their upcoming admin/developer conference (November, in Dortmund). Here are the (not yet finalized) abstracts: - "Cloud computing". The Google (and Amazon, and other companies') computing services model is very different from the traditional IBM/Lotus packaged software, client/server one to which we have all become accustomed over the last 2 or 3 decades. Come hear about the mega-differences (business model, architecture, even the social and political impact) as well as the micro-differences (how application APIs are different, how data handling and compliance are VERY different...), and even about the nano-differences, such as tech support, debugging, and error handling. How will "cloud" business partners and ISVs make money? Will you need to transform your software development processes? Is "cloud computing" still "vaporware"? - Achieving "Peaceful" Coexistence Between Microsoft Exchange and Lotus Notes While many organizations strive to standardize their messaging infrastructures on a single platform, many others do not. Whether through merger, acquisition or simple user preference, many organizations face a need to achieve peaceful coexistence in the messaging wars, perhaps for only a short time (while they transition from one to the other), but perhaps (if they can pull it off) for a very long time. The technical challenges involved in maintaining a well-performing dual messaging infrastructure are not trivial, but they can be met successfully with proper planning and the right technology. The big issues are all around directory synchronization, message transport, rich text, scheduling logic differences and data formats. Come to this session to learn how to overcome these technical challenges, how to keep both Notes and Exchange up and running, and how to avoid mutually assured destruction! I always enjoy DNUG events, the audiences are great, the conversations are always interesting, and the beer is always good. Hope to see you there! Visit the Binary Tree website here Bob Balaban July 27 2008 11:21:52 AMGreetings, Geeks! Welcome to part 3 of what may well turn into a continuing series on "cloud computing" (the other posts are here, and here). Depends, I guess, on how much interest there is, and on how much there is to say about it. I've been spending a bit of time lately using the Google "APIs" to query and populate Gmail and Google Apps accounts, so a lot of what I'm going to say here references Google. That does not mean that I'm writing a "review" of Google's interfaces or capabilities, and it does not mean that there aren't other companies that do similar things. I'm focusing (some) on Google because I now know it fairly well, and because I think the technology behind what they do is interesting, with broad applicability. Okay? Yes, it's certainly true that Amazon.com and Microsoft (and others, I'm sure) have similar capabilities. Maybe sometime, when I know more about those than I do today, I'll write that article. So, how is "talking" to the "cloud" (meaning, performing authentication, querying for info, posting new/modified info, etc.) similar to, and different from, regular old Client/Server interaction? Isn't it the same, just with more servers? The answer is yes ... and no. (I'll refer to "communication with the cloud" -- regardless of whose cloud we're talking about -- as "cloud talking". Saves typing). The basic client/server distinction is still there, of course. You (the developer/user) get to (have to) write the "client" code, which communicates to some server, somewhere, by using the Internet and HTTP as the transport protocol. What you send over the wire is nothing new, either, it's the usual set of HTTP headers, plus a data (query, update, delete, read...) "payload" consisting of XML text (mainly). The "answer" to your action likewise comes back over HTTP as XML data. So, it's a lot like Domino URLs, then, right? Yes ... and no. So, it sounds a lot like Web Services using SOAP, right? Sort of ... but not exactly. So, it's like using JavaScript in a browser app to do AJAX, right? Hmm, again I'd say yes ... and no. So, it's a lot like Lotus Connections APIs, right? Yes ... and no. Let's look at each of these other technologies in a little more detail to see how they're the same and yet different. Domino URLs. The Domino HTTP server responds to certain URL queries by sending back data of various kinds. In that sense it is like cloud talking, with a couple of differences. Domino URLs (today, anyway) are always HTTP GET operations. You specify what it is you want to do via a combination of the specific URL command (the thing following the "?" in the URL, e.g., "OpenDocument", or "OpenAgent") plus a series of URL arguments, of the pattern &keyword=value. You get various things back, often in different formats, depending on what options you specify (HTML, JSON, XML...). By contrast, you talk to Google services by using HTTP in a very REST-oriented way: you can use any of the common HTTP verbs (GET, POST, PUT, DELETE...), depending on what you want to accomplish. The basic structure of the data you send is always XML, though within that "outer" structure you can transmit many kinds of things (any MIME type, basically, such as base-64 encoded file attachments on a message). The "structure" that the Google cloud wants to see is Atom. The URL you use is the location of a particular "service" implemented by the cloud you're talking to. The results you get back, are likewise an Atom feed, though the specific content will be particular to the query you executed. If you queried for a list of calendar entries between 2 dates for a certain user, the result might be a 'feed" of those entries, encoded in some specified way. If you uploaded a set of email messages to some user's GMail account, you might get back a feed where each entry in the list tells you what happened to the corresponding message you uploaded (success, parsing error, whatever). Here's an example of an actual Google mail-upload URI: https://apps-apis.google.com/a/feeds/migration/2.0/default/mail/batch I can use either HTTP or HTTPS. The URI up to the word "default" is specified by the service I'm invoking. The Google service implements a feature where I can always say "default" for the account access designator, meaning, "use the account for which I have provided login credentials", or I can specify a particular user/domain (e.g., "gmail.com/bobzmail", for an (imaginary) account named bobzmail@gmail.com). In the case of an upload I would use an HTTP POST command, for a query I'd probably use a GET. There have to be various other HTTP headers there to form a valid command, to handle authentication, specify the caracter set I'm using (almost always utf-8), the MIME type of the payload (almost always application/atom+xml), size of the payload, etc. My XML "payload" always begins with the standard "" header, followed by an Atom tag. The structure and formats (as well as what namespaces to use) are specified in the Google documentation (look here, then wend your way down. Some doc is only available once you've signed up as a partner).
The documentation Google provides for their services (mail, calendar, YouTube, GoogleDocs, there are many) is a comprehensive explanation of the service options and fields you either get (query) or specify (upload) to make things happen. It's all XML. More on this later.
Web Services/SOAP. If you're still reading (I hope you are), then you already know that Atom and SOAP are not the same, though they can be used to perform similar functions. Each uses URIs to direct an "operation" to a particular service, and even to a particular action in that service. SOAP says you always use POST, though, whereas Atom/REST say you use the different HTTP verbs for different actions. Furthermore, the format of the payload for a particular SOAP web service request is specified in the SOAP spec plus in the WSDL description of what that particular service/operation wants. For Google, the format is dictated by a combination of the Atom spec and the documentation of the service provider.
JavaScript/AJAX. AJAX is a programming style (you could, I suppose, say "protocol") or technique that Web app developers use to fetch data from a server asynchronously (on a "background thread"), so that the app user can continue to manipulate the UI without interruption while data is being loaded. JavaScript, of course, is a programming language. Neither has any necessary relationship to Atom, XML, REST, or anything else, though of course, given the right tools, they can take advantage of those data structures and protocols. My point is that JavaScript and AJAX can be used to do cloud talking, but you have to set it up and code your app to do that. Google happens to have a JavaScript interface to most of their cloud services, more on this below.
Lotus Connections API. True confession: I don't know very much about the Connections API. I have heard, though, from two reliable sources on the Connections team, that it's an XML/Atom based REST-style protocol using HTTP talking to Lotus Connections servers. In those respects, then, the Connections API is very much like the Google Apps API, though a Connections server is by itself very different from a Google cloud.
Google "APIs".
Why the quotes? It's because I'm an engineer, and therefore a person addicted to hair-splitting. To me (and of course reasonable hair-splitters can, and will, disagree about this), "API" is something provided as a programming language library (or several), which a developer invokes through procedural invocations. This "definition" (I made it up on the fly, clearly it's not really rigorous) does not include an Atom/XML protocol. Which is not to say that Google's (or Connections' or anyone else's) Atom/XML protocol isn't perfectly functional and wonderful, only that I don't really consider it an API.
However, Google actually DOES have a ("real") API for many of their cloud services. This is only natural, logical, and convenient (for us developers who want to use it). Why? Well, because, no matter how great Atom, XML and HTTP are at what they do, they're tedious as heck to program. Make no mistake, I love XML, I use it ALL the time, but I would not want to have to constantly be writing code (C, C++, C#, LotusScript, whatever) that formats XML strings. Too much typing! Too easy to miss a single character and end up with mess.
No. I want a nice, functional, preferably object-oriented API that I can call things on to format the XML for me. That way I can focus on what it is I want to happen, rather than on the details of where the attribute, or the tag element, or the ">" escape syntax has to go. And, evidently, I am not the only one who feels this way. That's why Google has provided what they call "client libraries" -- each library is a single language binding which provides real programming APIs that handle things like formatting Atom feeds, connecting to the Google cloud services, handling the HTTP posts and replies, and so on. Google have language bindings for their Atom protocols for C#, Java, JavaScript, Ruby and a few other languages. Not only that, they give you the source code for their API code, so you can actually debug down into it to see exactly how it's serializing your XML, and so on. Pretty cool, this is a huge convenience, because otherwise I probably would have had to write my own version of it, in order to get anything done.
That's not the end of the story, of course. You still have to deal with the API. Because it represents the service functionality of the cloud, you have to give it data in the formats it wants (MIME for file attachments, RFC822 stuff for mail messages, ICAL for calendar entries....). Sometimes when you send a feed "up" to "the cloud", it just works -- you get back a "201 - created" result code for a new mail message that you're inserting into someone's GMail account, for example. Other times you get back the strangest stuff: "500 - internal server error", "404 - Not found", "302 - Temporarily Moved", and my favorite, "503 - Server busy, try again later". Huh. Huh?
We're out of time for today, kids, but maybe next time I'll write more about what happens when you talk to the cloud, but the cloud doesn't talk back to you. Or, if it does, you have no idea what it's saying.
Meanwhile, post your thoughts and comments! Happy rest-of-weekend! (Yes, pun intended, though. like most of my puns, it wasn't a very good one....)
Please visit the Binary Tree website here.
Bob Balaban July 24 2008 09:19:06 AMI am most pleased to welcome Steve Mullen to Binary Tree as Software Development Manager, beginning Monday the 28th. I have known and worked with Steve since my days at Iris in the 90s, and he was most recently the 2nd-line manager in charge of the Notes Client development team at IBM/Westford. In fact, Steve successfully survived having both me and Rocky Oliver report to him directly for nearly a year, when we all worked at IBM together. I am very excited about having Steve join us, because I know we can benefit from his tremendous experience and from his extensive skills, both as a developer and as a manager. Visit the Binary Tree web site here Bob Balaban July 15 2008 08:47:29 PMMaybe you've had days like this (but I hope not)? http://youtube.com/watch?v=_BSEwH3za1w Bob Balaban June 24 2008 07:15:27 PMGreetings, Geeks! I was going to write a follow-up post to my "Clouds" article of about 2 weeks ago, and the focus was going to be some interesting aspects of "cloud" architecture: how it's different from "regular" client/server, how you use it differently, and what benefits you derive thereby. I'm still going to write that article, but not today. Today is still about cloud computing, but after looking at several of the replies to my earlier post, it's clear that many people see real problems with buying into the cloud model of computing, particularly with the fact that you're handing all your data over to someone else, and you may very well not even know where that data is being stored. Commentator Kerr points out (rightly) that companies give their data to 3rd parties all the time, so what's the big deal? (I'm paraphrasing) Commentator (and former Burton Group analyst) Karen Hobert usefully distinguishes between "web centric/saas" clouds and "data center centric" clouds. Google is, I think, mostly in the former category, whereas Microsoft Online, or IBM Strategic Outsourcing, say, are in the 2nd category. Yes, Google runs from data centers, but you don't know where they are (the locations are secret), there are many of them (the number is secret), and you have no idea in most cases which one (or ones) have your data. Kerr's point that others already have their hands on your data is true enough, but I would claim that in order to be successful as a business, companies like Google who want to "host" not only your apps, but your data and your email/application infrastructure as well need to convince their customers that they have certain things under control: - Reliability. If I give you my stuff, I need to be confident that I can get it back in good shape, whenever I want it. - Security. I need to be confident that nobody else can hack into my stuff - Privacy. I need to know that the holder of my stuff (Google, Amazon, whoever) is not reading it for their own purposes. It's ok to create a search index for authorized users to access, but you better not be mining MY stuff for aggregations, usage patterns, or anything else. - Locality. Many users of "cloud" services don't care where their data are stored. Others do care, some are required by law to care. Financially regulated companies, for example corporate auditors, are often required by law to keep their data stored within certain political boundaries (country, state, whatever). This is true in some states within the United States, and also for many countries within the European Union. Google offers to guarantee (for an extra cost, one imagines) that if a customer has this requirement, and if Google has a data center that meets the locality requirement imposed on the customer, then that customer's data will not be stored outside that data center. As an aside to that last point on Locality, I heard a Google executive remark during a presentation that Google is receiving a lot of requrests from non-U.S. customers for guarantees that their data will be stored only OUTSIDE the United States. The reason for this, of course, is due to the rather fascistic (my opinion) provisions of the USA Patriot Act, which allows certain government agencies (FBI, CIA, NSA, probably others) to snoop your data by simply declaring to the holder of that data (your ISP, your phone company, a public library you visit to use their books or computers) that they need to. AND, the provider of access to your stuff is PROHIBITED from notifying you that this intrusion has happened. [sarcasm]Real "land of the free, home of the brave" stuff, huh?[/sarcasm]. As usual, those who can will vote with their feet, the rest of us have to hope that a new regime will overturn this nasty law. One other legalistic implication of cloud computing I wanted to point out: today it is perfectly legal for companies to snoop employees email (traffic and/or repositories) and other files, so long as the company is the owner of the computer on which the data resides. This is true in most states within the U.S., and (I am told) throughout the E.U., and probably other places as well. Of course, it's much more of a speculation to try to guess how many companies actually take advantage of this legally permissable behavior. I'm sure some do. Others may only snoop when they have a real reason to: they're worried about a specific case of industrial espionage, perhaps, or maybe a crime is being investigated. Naturally, the normal reaction by the "sensitive" among us to this situation is to start (or continue) encrypting everything. But there are limits on this. In Notes, you can encrypt databases stored locally on your Client machine (laptop, say), but you can't encrypt the server replica of your mail database, because if you did, the server would not be able to deliver mail to it. You can encrypt all of your outgoing mail traffic in Notes pretty easily, but you're still reliant on your Notes-generated key to do so. Are you completely certain that your employer hasn't sequestered a pre-determined set of bits which are used in all keys? Really? You're sure? Well, it was well known in the 90's that Lotus did a deal with the U.S. Government to provide them with some of the bits used in all non-NorthAmerican keys, reducing the effectiveness of any keys used outside NAmerica, and thereby making it easy for U.S. government agencies to crack encrypted messages. In return, Lotus won the right to not have to have 2 separate security systems in Notes (one for the US one for everywhere else), and also got Notes removed from the list of "weapons" maintained by the Defense Department. Being on that list was awful, it (technically/legally, though it was rarely enforced) prevented people from taking computers containing the US version of Notes out of the country. Anyway, the real point I'm trying to work towards is this: do employers lose the legal right to snoop employees' data when they've outsourced data storage and applications like email to Google? The employer no longer owns the computers that the stuff sits on, Google does. So your employer may no longer have the right (or the ability) to read your email. Maybe that's a good thing (depends on whose side you're viewing the issue from, I guess). But consider the flip side: if all of your stuff is on computers owned by Google, does that give Google the right to snoop it, if they choose to? I'm not a lawyer, so I don't know. I suspect part of the answer lies in the terms of whatever agreement the employer and Google contract to. Perhaps Google will readily agree not to peek at your stuff in their data centers. But if they do anyway, are they obligated to tell you about it? What is the enforcement mechanism? Yet another wrinkle: within some regulated industries, the law (such as Sarbanes/Oxley in the U.S., as one example) imposes so-called "Chinese walls" between certain groups of employees. This means that for some organizations, they must institute email tracking and archiving, AND they must prevent email from flowing between certain groups. Nobody in the trading department (for example) is allowed to send email to or receive email from anyone in the analysis department. I've worked on such security features for a couple of customers, generally with Notes/Domino you would write a special purpose Extension Manager plug-in to monitor the traffic and kick back "invalid" messages (there are other ways of doing it, too). In this situation the "security" code is not reading the message (though a copy is probably being stored away somewhere, in case it's needed later), it's only scanning the From field and the recipient lists, and looking for invalid pairings. Still, I don't think (as of now) that Google offers a service like this (they do offer archiving and other security/spam filtering features through their Postini acquisition), I could be wrong. If someone uses a company computer to commit a crime (download kiddie porn, perpetrate embezzlement or fraud, or whatever), the company may have to accept some liability. If you set up a scam site on Google Apps and cheat people by stealing credit card numbers, or something, does that make Google liable too, because you used their computers? The point is that when you shove everything into "the cloud" you give up a measure of control over your content and over your operations. The implications are sometimes subtle, but nonetheless important. Should be great for the lawyers! (ok, NEXT post on cloud computing will, I promise, get into the architecture/geeky stuff....) Bob Balaban June 16 2008 01:50:08 PMI've been working with Google Apps APIs and trying out various features of the mail, calendar and contacts services (products?) for a few months now. I think I'm getting a handle on the whole "cloud computing" thing. So I thought I'd collect a few observations in a blog post and see what y'all think as well. This is neither a sales job nor a trash job on Google, nor is it meant to be a comprehensive feature comparison of Google Apps vs. Notes. I'm really using Google Apps as a prominent example of "cloud computing", and starting to think about the implications for the collaborative software world.
What is 'cloud computing'? Here's a Business Week article that attempts to define it. I think it does a reasonable job, and it points to IBM as a technology leader in that space. But while I"m sure that IBM has pockets of advanced technology and even offerings in the "cloud", it seems fairly obvious to me that Google is far and away both the technology and the market leader in cloud computing (Microsoft has stuff too, naturally). One could argue with that conclusion, I suppose, and point out that while Google has more offerings in the consumer/mass-market space (gadgets, youtube, etc), they are not necessarily the leader in "enterprise" markets. Maybe. I don't know, and that's not the point I'm trying to make, anyway: this isn't a vendor comparison. To me, the basic point (or maybe there's more than one point here) of cloud computing appears to be this: cloud providers (if we can call them that) will not only host apps for you, they will supply you with virtually unlimited storage space, and virtually unlimited scalability of access, as well. Sign up for a free GMail account with Google, and you get 6 GIG of storage. Sign up for a $50 per seat per year (+ or -) enterprise account, and each GMail user gets 25 GIG of storage, plus more features. That's a lot of email. On the scalability front, Google will happily host your enterprise email regardless of whether you have 3 seats, or 300,000 seats. They offer good searching (duh), security, spam filtering, archiving and so on. And you hardly ever have to delete anything, with that much space available.
Major Benefits of 'cloud computing' The biggest argument I see cloud vendors making for people (and companies) to switch over is the economic one: when you go to Google Apps you're not just going to a "Software As A Service" (SAAS) model, you're outsourcing your entire email infrastructure. No more in-house maintenance of hardware and networking, no more paying pesky admins to wake up in the middle of the night and fix stuff that broke, no more having to re-evaluate and do software/hardware upgrades every year. Google handles ALL of that for you, and it's pretty cheap. You just hired another 100 employees? Go online and create accounts for them on your hosted site (or write a program to do it for you via the Google provisioning APIs), and you're done. Google never goes down, they handle all the backups, authentication, access control, routing, everything. And it's all accessible with a browser.
What are you giving up for 'cloud computing'? So what's the downside? Well, you do give up a few things: - Functionality. Let's face it, GMail is not the best email system out there, feature-wise or UI-wise. It's still in "beta" 4 years after its launch, it does WAY less than, say, Notes. Same for the Google Apps calendar (one annoying example: they don't allow attachments on meeting invitations...) - "Ownership". By that I mean that when all your email is in GMail, you don't ever know where it really is. Same with all the other "cloud" apps -- your data could be anywhere. For a lot of people this isn't a big problem (once they get over that initial slightly queasy feeling, especially when they remember how much money they're saving). But, for some people, it's a show-stopper. Accounting firms, for example. In some countries (and in some states within the U.S.), regulated firms are prohibited from storing sensitive data outside the boundaries of their political unit. Other firms worry that data stored on Google servers that happen to reside in the United States might be vulnerable to arbitrary U.S. Patriot Act subpoena. - Application deployment. Email is one thing, enterprise applications (especially custom-built ones) are quite another. You can't go to Google and ask them to give you a Domino server to run your mission critical apps. You can deploy Web apps on Google now, but only if they're written in Python (or something like that). Maybe that will change, but in the meantime, I think most companies who may want to use Google for email will hang on to their app servers for quite a while. But that leads to what I call the "coexistence" question: Since quite a lot of Notes/Domino based applications use email messaging as a transport (many workflow apps, for example, send email notifications to users), how does that work when my email moves to the cloud? Do I have to rewrite all those apps? Yikes. (Quick plug here - no, you don't. Binary Tree (and some other vendors) have coexistence solutions you can, uh, buy).
Bottom line? Are the negatives show-stoppers? Clearly not, Google (and Microsoft, and maybe even IBM someday) seem to be very happy with the rate of growth of their cloud offerings. On the first point (Functionality) -- yeah, ok, maybe it's not so great now, but it'll get better over time. And there are evidently a lot of CIOs who look at the cost savings and (perhaps, I'm just speculating here) say to themselves, "Wow! My bonus is going to be HUGE this year. I guess my users can live without a few bells and whistles on their email. No more whining about increasing the mail quotas! Yee-HAH!" On the "Ownership" issue, obviously this is a nonstarter for some types of organizations (I don't see the CIA or the military going for it anytime soon). For others, it's just another cost issue: Google will (for a fee) offer certain guarantees about where your organization's data will reside. If you're in (I'm just making up this example) say, the Cook Islands (yes, it's a real place, go look it up), and you really, really, need all your bytes to stay in the Cook Islands, and you can show Google that there's enough like-minded organizations in the Cook Islands to make some money, then maybe Google would simply build a data center there. Or, maybe not, and you find another solution. As far as coexistence goes, there are technology solutions, and if you have really important apps that you want to keep on Notes, then the cost is still probably worth it. Careful planning is required, and YMMV. I could go on (and on), about APIs, RESTful interfaces, message interoperability architectures, you name it. BUT, I won't, this post is already long enough. Maybe I'll do a Part 2, if there's interest. What I REALLY want is for YOU to tell me what you think -- impressions, experiences, razzing, you know, the usual. And, in the end, "It's cloud's illusions I recall. I really don't know clouds at all!" :-) Bob Balaban May 16 2008 12:20:20 PMGreetings, Geeks! Just wanted to let you know that I will be presenting a session (and attending many others) at the Irish Lotus User Group (ILUG) get-together in Dublin next month (June 4-6). My session title and abstract are: Achieving "Peaceful" Coexistence Between Microsoft Exchange and Lotus Notes While many organizations strive to standardize their messaging infrastructures on a single platform, many others do not. Whether through merger, acquisition or simple user preference, many organizations face a need to achieve peaceful coexistence in the messaging wars, perhaps for only a short time (while they transition from one to the other), but perhaps (if they can pull it off) for a very long time. The technical challenges involved in maintaining a well-performing dual messaging infrastructure are not trivial, but they can be met successfully with proper planning and the right technology. The big issues are all around directory synchronization, message transport, rich text, scheduling logic differences and data formats. Come to this session to learn how to overcome these technical challenges how to keep both Notes and Exchange up and running, and how to avoid mutually assured destruction! Hope to see you there/then! Bob Balaban May 13 2008 07:13:39 AMGreetings, Geeks! I'm hoping that many of you out there are fans (or recovering fans) of The Simpsons, in which case you'll have no trouble recognizing the names in this week's post as the characters in the always-fighting cat and mouse TV show beloved by all the children of fictional Springfield. If you're closer to my age (Boomer) than to Gen-X, and don't have any kids that you watch TV with, the equivalent characters of our time were probably Ben and Jerry (the cat and mouse cartoons, NOT the ice cream makers!) So, the thing I want to get at thoday is, why can't we all just get along? Nah, that's not it. Let me try again: Why is it so darn hard to release quality software? Nope, that's not really it either (but it's a good question). Ok, I think I have it now: If you were working in a relatively small software-making company (let's say, between 20 and 200 employees, and by "software maker" I mean, creates and sells software as a primary business), and you had to figure out how to organize the people doing the actual software creation and (we hope) testing, how would you set that up? Of course we need to constrain the question a little bit more, so as to be answerable in our lifetimes. Here are the (arbitrarily ranked) goals I would want to maximize in such a situation:
Minimize the nuber of defects (bugs, glitches, doc errors, user errors whatever) reported by customers (defined as, the people you pay you for your software). Notice that I lump "user errors" in with the more common kinds of problems. I do that because, IMHO, a "user error" (the user did something "wrong", resulting in a bad outcome, but the software is "working as designed") is rather more likely to result from a problem with the User Interface (UI) or perhaps with the documentation, leading the user to expect a result other the one s/he got when they did whatever it was. We could argue whether these are "real" bugs, or "equal" in some way to "real" software problems, but I don't care. By my definition, this kind of error is a quality problem in the product.
Maximize the amount of automated testing a product can undergo before release. I put this item in here because experience (and arithmetic) has shown me that release cycles are dramatically reduced when you apply automated testing to a product. I won't belabor the point, and yes, it's true that you can't generally achieve 100% automation (at reasonable cost), but test automation is a good thing.
Keep as many employees happy in their jobs as possible. Yeah, yeah, I know. There are a LOT of factors that affect employee satisfaction, but here's one thing I have noticed myself, and at more than one company where I have worked: implementing caste systems leads to unhappiness among employees. Creating sub-divisions within the broader "development organization" where one group of people (let's call them the "Lords") is responsible for the fun, creative, and more highly paid work of writing (inventing, architecting, designing, coding...) software, and another group (the "Serfs") are responsible for taking the work product of the Lords and testing it (looking for defects, broadly defined, which might also include things like unacceptable performance, poor UI, etc.). Most companies at which I have been an employee implement (whether blindly, or on purpose) this kind of 2-tiered system. The Lords (developers) get more money, and more status than the Serfs (QA/QE *, pick your terminology). Some people feel that this system is "natural", after all, the developer's job is harder and more creative. The "QE" role is to receive and test, a rather more "mundane", yet unfortunately necessary step in the release cycle. Anyhow, the point here is, your job (as the hypothetical keeper/maker of the org chart in this example) is to try to not have a caste system.
Minimize the cost of producing quality software. Since we're talking about the software business, there has to be a business constraint on all of this. Headcount is expensive. Back in the days when Lotus was mostly a spreadsheet company, the normal ratio of QA to Developer headcount on any major project was 3:1 or so. For every developer creating code, there were 3 people testing it. Today that would be inconceivable. So? Now what? We can probably all agree that there's a difficult problem here. What's the answer? Speaking for myself, I don't have an "answer". There probably is no one single "answer". What I have, though, is a hypothesis. Which is: [hypothesis] If you view the problem of "software product quality" from a broad perspective (as I described it above), then your approach to building the software cannot treat "QA" (or "QE", or whatever you call it) as a distinct operation from "development". ** [/hypothesis] One of the things this implies for software companies is that "Development" should probably not be a separate organization, or department from "Testing" (QA, etc.). I would almost advocate that within a "Software Development" organization (let's say, a team dedicated to creating and evolving a single product, or suite of products), there is indeed reason to declare and nurture a "specialization" of skill that is distinct (i.e., not all architects/designers/coders need to be expert in testing, and not all testers need to be expert in software design/implementation), but really, isn't there an awful lot of overlap? Don't testers benefit from understanding the product's construction? Don't developers benefit from an understanding of a testing process, so that they can (as one possible example) instrument their code appropriately? For sure, once you get to thinking about test automation, someonehas to build the automation harnesses, script the test runs, etc. Right? Is that not "development"? So, is there a benefit to organizing around a "QD" (Quality Development?) job function, and organize a primarily "development" team where perhaps there is some internal specialization (but not necessarily a clear distinction) between creating software and making sure it works properly? What do YOU think? How would YOU organize for quality, if you could? ------------------------------------------------------------------------------------------------- (Footnotes)
* In fact, when I first started working at Lotus Development in the late '80s, the "testing" department was called "QA" (Quality Assurance). Around the time that we shipped Notes V4 (mid-'90s, before IBM bought Lotus), the Lotus QA organization working on Notes was merged with the Iris organization (developers of Notes), which had been composed almost entirely of developers. Somewhere in there, the term "QA" was explicitly switched to "QE", or "Quality Engineering". When I asked why, I was told that people felt it was a better name, because it more accurately reflected the engineering basis of the task, and that it might improve the level of respect given to the job title and to the people. To this day, the terminology (at the Lotus division of IBM, anyway) remains "QE", though I, personally, haven't detected any significant reduction in the caste differential (though of course these things are always in the eye of the beholder, YMMV).
**Again, the underlying assumption here is that you're trying to optimize for quality. Naturally, many, many companies (businesses) do not inherently optimize for quality, they try to optimize for profits, or maybe for market share, or maybe for personal career growth, or maybe for something else. I'm not primarily a business person (maybe if I were, I would still be running my own company!), so I can't say whether quality is the thing for which a business should globally optimize, and certainly, from an employee perspective, working on a "quality" product doesn't make me happy if I can't get paid for it. But it does seem to me that quality should be an important objective of product development, a "differentiator", if you like. :-) But that's another discussion. | |