Up the tata without a tutu

Until yesterday, the FogBUGZ license said that you couldn’t reverse engineer the program, attempt to look at the source code, or modify it in any way. Various honest people have asked how much we charge for a source license, so that they could customize a few things.

Hmmm. Why does the license say you can’t change the source code? I couldn’t think of a single reason. In fact I thought of a lot of counter-reasons, and immediately changed the license agreement. So now you’re going to have to sit through one of those old-fuddy-duddy stories from my past.

Way back in 1995, I was working at Viacom, where a small group of us hardy pioneers were building web sites for various Viacom properties.

In those days, there were no application servers. Sybase was so clueless that if you wanted to use their database on the Internet they told you that you needed to buy a $150 client license for every user that connects to your web site. Netscape’s web server was up to version 1.0.

A brave company called Illustra started telling people that their database management system was perfect for the web. You see, Illustra was designed to make it easy to add new data types by writing some C code and linking it in with their DBMS. (Any programmer who’s used a DBMS will tell you that this is already sounding a bit too dangerous. C code? linked in? Oy.) This was originally intended for exciting data types like Latitude/Longitude, time series, and so on. But then the web happened. Illustra wrote something they called a “Web Blade” and linked it in. The Web Blade was a sort of half-baked system that allegedly made it possible to extract data from the database and create dynamic web pages on the fly, which was the biggest problem everybody had in 1995.

A colleague of mine at Viacom was put in charge of building an ecommerce site so that Blockbuster could sell, I kid you not, CDs on the web. (Because that’s what people think of when they think of Blockbuster, right?) Anyway, he thought that Illustra would be perfect for the job. Now, Illustra cost something like $125,000, and shaking that much money loose from the Viacom Tree is like herding cats, so it took a while. My colleague taped a paper cup to his cube labeled “Illustra Fund” and collected a few dollars that way. The CTO negotiated hard and long hours with Illustra and eventually a deal was struck. We installed Illustra and got to work.

Unfortunately, disaster struck. Illustra’s Web Blade was barely half baked and completely not up to the task. It crashed every few minutes. When it did run, it proved to have the only programming language I’ve ever seen that wasn’t Turing-equivalent, if you can imagine that. The license manager kept deciding to shut you off and your site would die. Building a site with it was terrible, my colleague’s annus horribilis. So when they came to me and said, “Joel, you’re making a site for MTV,” I said, “uh oh.”

“Please can I not use Illustra?” I begged.

“Well, OK, but what are you going to use instead?” There really weren’t any other app servers in those days. There was no PHP, no AOLServer with TCL stuff, perl had to fork, we didn’t have penicillin, life was terrible.

And my reputation was on the line. And I decided that the scariest thing about Illustra was that when it crashed, you couldn’t do anything about it. At least, if you had the source code, I thought, if Illustra crashes, well, it falls into your lap in the debugger and you can try to fix the bug. You may have to stay up all night for a week debugging someone else’s code, but at least you have a chance. Whereas, without the source code, you are up the proverbial tata without a tutu.

And that’s where I learned a key lesson in software architecture: for your most important, mission critical stuff, you have to use a tool that is one level lower in abstraction than ideal. For example, if you’re writing a cool 3D shoot-em-up game (like Quake, around the same time period) and your key number 1 differentiator is to have the coolest 3D graphics, you do not use whatever 3D library you can find. You write your own, because it’s fundamental to what you do. The people who use 3D libraries like DirectX are using them because they are trying to differentiate their games on something other than 3D performance. (Maybe the story line.)

That’s when I decided not to trust anyone else’s poxy application server, and decided to just write my own, in C++, using Netscape Server’s low level API. Because I knew that at least, if anything went wrong, it was in my code and I could eventually fix it.

And this is one of the greatest virtues of open source / free software, even if you could afford Illustra’s $125,000 piece of tata: at least if anything goes wrong, you are going to be able to fix it, somehow, and you won’t get fired, and the nice-if-hyperactive people at MTV won’t be all pissed off at you.

When I sit down to architect a system, I have to decide which tools to use. And a good architect only uses tools that can either be trusted, or that can be fixed. “Trusted” doesn’t mean that they were made by some big company that you’re supposed to trust like IBM, it means that you know in your heart that it’s going to work right. I think today most Windows programmers trust Visual C++, for example. They may not trust MFC, but MFC comes with source, and so even though it can’t be trusted, it can be fixed when you discover how truly atrocious the async socket library is. So it’s OK to bet your career on MFC, too.

You can bet your career on the Oracle DBMS, because it just works and everybody knows it. And you can bet your career on Berkeley DB, because if it screws up, you go into the source code and fix it. But you probably don’t want to bet your career on a non-open-source, not-well-known tool. You can use that for experiments, but it’s not a bet-your-career kind of tool.

So I got to thinking about how to make FogBUGZ a safe bet for smart engineers. Almost by accident, it ships in source code form — because that’s how ASP pages work these days. Which doesn’t bother me. There are no magical, trade-secret algorithms in bug tracking software. This stuff is not rocket science. (In fact, there are very few magical, trade-secret algorithms in any software. The fact that it’s fairly easy to disassemble an executable and figure out how it works just doesn’t matter as much as intellectual property lawyers think it should.) It doesn’t matter to me that people look at the code or modify the code for their own use.

There’s another risk when you modify source code that you bought from a vendor: when the vendor upgrades the code, you are going to have a heck of a time migrating your changes to their new version. There’s something I can do to ameliorate that, too: if you find a bug in FogBUGZ and fix it, and send me the fix, I’ll incorporate it into the next version. This is intended to make people feel a little bit more comfortable that (a) FogBUGZ works, and (b) if it doesn’t work, in some mission-critical way, they can fix it rather than get fired, and (c) if they do have to fix it, and their fix makes sense, it will get back into the source tree so that the next version will incorporate their fixes and life will be less brutish.

By now I can hear the open-source and free software advocates practically screaming, “you silly goose! just make it open source and be done with it! open source doesn’t have any of these problems!” And that’s nice. But my wee company with three programmers costs $40,000 a month to operate. So we just charge for our software, and we don’t apologize, because it’s worth the money. We don’t claim to be open source but we can make sure that FogBUGZ is a safe decision to make, by adopting two or three nice features from the open source world.


About the author.

I'm Joel Spolsky, co-founder of Trello and Fog Creek Software, and CEO of Stack Overflow. More about me.