Planning for 2003

It has gotten supernaturally quiet. Everyone but me is on vacation, for some reason. Here’s a look ahead at some of the things that Fog Creek Software, and Joel on Software, will be doing next year.

Programmer’s Paradise is one of the largest mail-order retailers of computer software, and they’re trying to make their catalog more interesting by including feature articles of interest to software developers. So over the next year, I’ll be writing a monthly column for them: the first printed version of Joel on Software. The best way to make sure you don’t miss an article is to subscribe to the printed catalog, it’s free (unfortunately, they only sell to North America, so the catalog is only available if you have a mailing address in the US or Canada).

Of course, I’ll publish articles here too. I think I’ve already told you everything I know, so most of what I’ll write about during 2003 will be rampant speculation, disguised as hard facts! The best way to avoid missing any new articles is to sign up for the mailing list (at the bottom of any page on the site). This results in one or two emails a month when there’s significant new content on the site, and zero spam.

I’ve been working on a second book. The first book I wrote was an expansion on all the user interface design stuff on this site; it consisted of about 50% more material than the site itself. The second book, now under construction, was intended to include all the management stuff (basically, everything else on this site). As I cut and paste articles I’m discovering that I’ve learned a lot over time and some of the older articles need some revision. So it’s taking longer to get this book out than I expected. But I hope it happens in 2003.

Early next year, I’m going to move the Joel on Software web server to a new home. I’ve ordered a new server from Dell (it will be a 2U rackmount Xeon with dual RAID-1 drives), and the Canadian networking company Peer 1 Networks has graciously offered to provide colocation space and bandwidth for free. If you need colocation or Internet connectivity, consider Peer 1 (and tell them I sent you!). I’ll write more about this when I install everything.

The townhouse where Fog Creek has its offices will be undergoing renovation, so we’re looking for new office space. Our goals: easy commute to the Upper West Side (of New York City), windowed private offices (not easy to find in this city), and cool. I don’t have a technical description of cool. “I know it when I see it.” Real estate people keep showing me these interior offices that are completely closed off, without even a view of windows in the distance. I ask them how they would feel about living in an apartment where the bedroom didn’t have a window. “You would freak out, right?” And that’s for a room you’re going to sleep in, at night. For a room where you spend the day it’s obviously non-negotiable.

The most important goal for us as a software company next year is to ship a series of new releases of CityDesk. The primary goal is to provide answers to the people who have said to us: “I like CityDesk, but I can’t use it for my application because x.” For example: “I like CityDesk, but my site appears in Japanese and English, and there are a few places where CityDesk doesn’t preserve Unicode properly.” Or, “I like CityDesk, but I need more flexibility in scripting so that I can do x, y, and z with my site automatically.” Overcoming these x’s is the easiest way to improve sales. The great thing about having customers and listening to them is that we get this kind of information, and we don’t have to speculate about what features would be neat,
and we don’t do things just because some programmer thinks they would be fun. Read Strategy Letter III to see what I mean. (By the way, we’ve already got a Unicode version of CityDesk “in the lab” which is how I publish all the translations, but it’s not quite ready for prime time.)

It’s no secret that the direction FogBUGZ is going is to become a better tool for communicating with customers. It’s already the best way for a small team or company to handle incoming email, and we’re getting more and more non-programming customers who simply use it to make sure customer email gets handled by the right person. But the other thing I’d like to see happen with FogBUGZ is to commoditize our complements. We can do this by making sure it works with more databases, especially open source ones, instead of requiring Microsoft SQL Server, which costs an arm and a leg. (Actually the current version also works with Microsoft Jet, which is free, but can wheeze with large bug databases.) In the other direction, FogBUGZ needs to handle more of the development process: schedules, specs, etc.

Anyway, thanks for reading Joel on Software, and have a great new year.

Lord Palmerston on Programming

There was a time when if you read one book by Peter Norton, you literally knew everything there was to know about programming the IBM-PC. Over the last 20 years, programmers around the world have been hard at work building abstraction upon abstraction on top of the IBM-PC to make it easier to program and more powerful.

But the law of leaky abstractions means that even as they built the abstractions that are supposed to make programming easier, the sheer amount of stuff you have to know to be a great programmer is expanding all the time.

Becoming proficient, really proficient, in just one programming world takes years. Sure, lots of bright teenagers learn Delphi one week and Python the next week and Perl the next week and think they are proficient. Yet they don’t have the foggiest clue how much they’re missing.

I’ve been working with ASP and VBScript since it first came out. VBScript is the dinkiest language on earth and ASP programming consists of learning about 5 classes, only two of which you use very often. And only now do I finally feel like I know the best way to architect an ASP/VBScript application. I finally think I know where the best place to put database access code is, the best way to use ADO to get recordsets, the best way to separate HTML and code, etc. And I finally use regexps instead of one-off string manipulation functions. Only last week, I learned how to get COM objects out of memory so you can recompile them (without restarting the whole web server).

Fog Creek is too small to have specialists, so when I needed to write a really good installer for FogBUGZ, our ASP/VBScript based product, I drew on several years of C++/MFC experience, and years of experience with Windows APIs, and good Corel PhotoPaint skills to create a neat picture in the corner of the wizard. Then to get FogBUGZ to work perfectly with Unicode, I had to write a little ActiveX control using C++ and ATL, which drew upon years of C++ and COM experience and a week or so learning about character encodings when I implemented that code in CityDesk.

So when we had a weird NT 4.0-only bug, it took me 3 minutes to debug, because I knew how to use VMWare, and I had a clean NT 4.0 machine set up in VMWare, and I knew how to do remote debugging with Visual C++, and I knew to look in the EAX register to get the return value from a function. Someone who was new to this all might have taken an hour or more to debug the same problem, but I already knew a tremendous amount of “stuff” that I’ve been learning, basically, since 1982 when I got my first IBM-PC and that Norton book.

Leaky abstractions mean that we live with a hockey stick learning curve: you can learn 90% of what you use day by day with a week of learning. But the other 10% might take you a couple of years catching up. That’s where the really experienced programmers will shine over the people who say “whatever you want me to do, I can just pick up the book and learn how to do it.” If you’re building a team, it’s OK to have a lot of less experienced programmers cranking out big blocks of code using the abstract tools, but the team is not going to work if you don’t have some really experienced members to do the really hard stuff.

There are a lot of programming worlds, each of which requires a tremendous amount of knowledge for real proficiency. Here are the three I personally know best:

  • MFC/C++/Windows
  • VBScript/ASP
  • Visual Basic

All, basically, what you would call Windows programming. Yes, I’ve written Unix code and Java code, but not very much. My proficiency in Windows programming comes from knowing not just the basic technologies but also the whole supporting infrastructure. So, I claim, I’m really good at Windows programming because I also know COM, ATL, C++, 80×86 Assembler, Windows APIs, IDispatch (OLE Automation), HTML, the DOM, the Internet Explorer object model, Windows NT and Windows 95 internals, LAN Manager and NT networking, including security (ACEs, ACLs, and all that stuff), SQL and SQL Server, Jet and Access, JavaScript, XML, and a few other cheerful facts about the square of the hypotenuse. When I can’t get the StrConv function in VB to do what I want, I bang out an COM control so I can drop into C++ with ATL and call the MLang functions without dropping a beat. It took me years to get to this point.

There are lots of other programming worlds. There’s the world of people developing for BEA Weblogic who know J2EE, Oracle, and all kinds of Java things that I don’t even know enough about to enumerate. There are hard core Macintosh developers who know CodeWarrior, MPW, Toolbox programming in System 6 through X, Cocoa, Carbon, and even nice obsolete things like OpenDoc that don’t help any more.

Very few people, though, know more than one or two worlds, because there’s just so much to learn that unless you have to work in one of these worlds for more than a couple of years, you don’t really grok it all.

But learn you must.

People get kind of miffed when they go on job interviews and get rejected because, for example, they don’t have Win32 (or J2EE, or Mac programming, or whatever) experience. Or they get annoyed because idiot recruiters, who would not know an MSMQ if it bit them in the tailbone, call them up and ask if they “have 5 years MSMQ.”

Until you’ve done Windows programming for a while, you may think that Win32 is just a library, like any other library, you’ll read the book and learn it and call it when you need to. You might think that basic programming, say, your expert C++ skills, are the 90% and all the APIs are the 10% fluff you can catch up on in a few weeks. To these people I humbly suggest: times have changed. The ratio has reversed.

Very few people get to work on low level C algorithms that just move bytes around any more. Most of us spend all our time these days calling APIs, not moving bytes. Someone who is a fantastic C++ coder with no API experience only knows about 10% of what you use every day writing code that runs on an API. When the economy is doing well, this doesn’t matter. You still get jobs, and employers pay the cost of your getting up to speed on the platform. But when the economy is a mess and 600 people apply for every job opening, employers have the luxury of choosing programmers who are already experts at the platform in question. Like programmers who can name four ways to FTP a file from Visual Basic code and the pros and cons of each.

The huge surface area of all these worlds of programming leads to pointless flame wars over whose world is better. Here’s a smug comment somebody anonymously made on my discussion board:

“Just one more reason why I’m glad to be living in the ‘free world.’ Free as in speech (almost) and freedom from pandering to things like setup programs and the registry – just to name a few.”

I think this person was trying to say that in the Linux world they don’t write setup programs. Well, I hate to disappoint you, but you have something just as complicated: imake, make, config files, and all that stuff, and when you’re done, you still distribute applications with a 20KB INSTALL file full of witty instructions like “You’re going to need zlib” (what’s that?) or “This may take a while. Go get some runts.” (Runts are a kind of candy, I think.) And the registry — instead of one big organized hive of name/value pairs, you have a thousand different file formats, one per application, with .whateverrc and foo.conf files living all over the place. And emacs wants you to learn how to program lisp if you’re going to change settings, and each shell wants you to learn its personal dialect of shell script programming if you want to change settings, and on and on.

People who only know one world get really smarmy, and every time they hear about the complications in the other world, it makes them think that their world doesn’t have complications. But they do. You’ve just moved beyond them because you are proficient in them. These worlds are just too big and complicated to compare any more. Lord Palmerston: “The Schleswig-Holstein question is so complicated, only three men in Europe have ever understood it. One was Prince Albert, who is dead. The second was a German professor who became mad. I am the third and I have forgotten all about it.” The software worlds are so huge and complicated and multifaceted that when I see otherwise intelligent people writing blog entries saying something vacuous like “Microsoft is bad at operating systems,” frankly, they just look dumb. Imagine trying to summarize millions of lines of code with hundreds of major feature areas created by thousands of programmers over a decade or two, where no one person can begin to understand even a large portion of it. I’m not even defending Microsoft, I’m just saying that big handwavy generalizations made from a position of deep ignorance is one of the biggest wastes of time on the net today.

Frequent readers, by now, have noticed that I’ve been thinking of the problem of how one might deliver an application on Linux, Macintosh, and Windows without paying disproportionately for the Linux and Macintosh versions. For this you need some kind of cross-platform library.

Java attempted this but Sun didn’t grok GUIs well enough to deliver really slick native-feeling applications. Like the space alien in Star Trek watching Earth through a telescope, they knew exactly what human food was supposed to look like but they didn’t realize it was supposed to taste like something. Java apps have menus in the right places but there are all these keyboard things that don’t work the same way as every other Windows app and their tabbed dialogs look a little scary. And there is no way, no matter how hard you try, to make their menubars look exactly like Excel’s menubars. Why? Because Java doesn’t give you a very good way to drop down to the native facilities whenever the abstraction fails. When you’re programming in AWT, you can’t figure out the HWND of a window, you can’t call the Microsoft APIs, and you certainly can’t intercept WM_PAINT and do it differently. And Sun made it plenty clear that if you tried to do that, you weren’t Pure. You were Polluted, and to hell with you.

After a number of highly publicized failures to build GUIs with Java (e.g. Corel’s Java Office suite and Netscape’s Javagator), enough people know to stay away from this world. Eclipse built their own windowing library from the ground up using native widgets just so they could write Java code that had a reasonably native look and feel.

The Mozilla engineers decided to address the cross platform problem with their own invention called XUL. So far, I’m impressed. Mozilla finally got to the point where it tastes like real food. Even my favorite bugaboo, Alt+Space N to minimize a window, works in Mozilla; it took them long enough but they did it.

Mitch Kapor, who founded Lotus and created 123, decided for his next application to go with something called wxWindows and wxPython for cross platform support.

Which is better, XUL, Eclipse’s SWT, or wxWindows? I don’t know. They are all such huge worlds that I couldn’t really evaluate them and tell. It’s not enough to read the tutorials. You have to sweat and bleed with the thing for a year or two before you really know it’s good enough or realize that no matter how hard you try you can’t make your UI taste like real food. Unfortunately, for most projects, you have to decide on which world to use before you can write the first line of code, which is precisely the moment when you have the least information. At a previous job we had to live with some pretty bad architecture because the first programmers used the project to teach themselves C++ and Windows programming at the same time. Some of the oldest code was written without any comprehension of event-driven programming. The core string class (of course, we had our own string class) was a textbook example of all the mistakes you could make in designing a C++ class. Eventually we cleaned up and refactored a lot of that old code but it haunted us for a while.

So for now, my advice is this: don’t start a new project without at least one architect with several years of solid experience in the language, classes, APIs, and platforms you’re building on. If you have a choice of platforms, use the one your team has the most skills with, even if it’s not the trendiest or nominally the most productive. And when you’re designing abstractions or programming tools, go the extra mile to make them leak proof.


Lord Palmerston: “The Schleswig-Holstein question is so complicated, only three men in Europe have ever understood it. One was Prince Albert, who is dead. The second was a German professor who became mad. I am the third and I have forgotten all about it.” Programming has gotten too hard.